It sounds great… but what does it actually do?

I read an article this morning on TechCrunch by the co-founder of a mobile app company (they make “Bump”, which shares contacts or other data through phones’ NFC chips). He was talking about the difference in understanding of technology between the creators of that technology verses their target market. It’s not a bad article, although the points are reasonably well rehearsed. But I think he’s misunderstood the problem slightly.

According to David Lieb, the article's author, for the mass market, things should be very, very simple, at the expense of features. Bump’s creators apparently discovered this idea and helpfully named it for us, “cognitive simplicity”, although I doubt the notion will be a surprise to many. The other side of the coin is “cognitive overhead”, a more widely used term for the brain power necessary for the uninitiated to understand a point or operate a tool. With delightful irony, a number of the comments below the article (as well as complaining about the rehashing of well-known ideas) pointed out the “cognitive overhead” of giving jargon names to straightforward concepts.

The article implies that technology producers often simply do not understand that non-techy users could struggle to comprehend how something they’ve designed operates. This is almost certainly correct in some cases: it’s easy to overlook the learning curve of an item you’ve been intimately familiar with since its inception. By and large though, I think it’s a problem of scale rather than intent – it’s actually quite hard to make something simple without losing the essence of the product or what differentiates it from its competition. And this is where people fall down – it’s a much bigger job than people often think and it’s compounded by the fact that you can stop at any point and still have a perfectly functional product.

And one of the difficulties is going too far. Albert Einstein said that things should be made as simple as possible, but no simpler. Several article commentators who use Bump complained about vanishing features in the latest version, presumably removed to increase simplicity. It’s obviously going to annoy a certain proportion of your users if you take functions away and perhaps the trade-off here is the right one, but it makes it clear that it is a balancing act.

In my opinion, a neglected counterpart to having an easily understandable product is having an easily describable product, or at least making an effort to describe it. Distressingly common is the inability of software companies (and to some extent providers of other forms of technology) to properly explain something they want to sell to me (or sometimes give to me, for which I can’t be quite so belligerent). This goes all the way from small open source projects to massive product suites from big companies. In the latter case, it’s usually because they have a marketing department who feel it’s their duty to talk up the product without worrying too much about whether it will actually fit the needs of potential customers. Examples can be found in the whole gamut between.

I don’t want to know how I’ll feel using your gadget. I don’t care (initially, at least) whether it’s been designed with solid engineering principles or by throwing a bag of C++ keywords in the air and seeing how they land. I especially don’t want to be greeted with a list of the minor functionality tweeks between versions 2.4.2.17 and 2.4.2.18, which seems particularly common in open source product pages. I want to know what it does. I think that should be more obvious than it would appear to be.

So I’d like to make a plea for simplicity myself, in the sales pitch: if the first page an interested visitor will see on your product’s website or information brochure does not contain a brief, clear description of what the program / gadget / vegetable can be used for (and I don’t mean, “Use the Big Bright Green Pleasure Machine and it will make your TCO lower / your life better / your partner sexier”, unless one of these purposes is its sole function) then your marketing is rubbish.

Excel: Conditional Formatting of Formulae

Yesterday, I was working with an Excel sheet and wanted to calculate a table of values, but be able to override some of the values. To make it obvious which values were calculated and which typed in, I decided to make the calculated values grey and leave the overridden values as the default black text.

Excel is massively flexible, so I didn't think I'd have much trouble – something in the Conditional Formatting arena would do what I needed. To my surprise, although I could do it, the only technique I found (on the j-walk website) relies on an obscure part of Excel that has almost been lost to memory: the XLM macro language, which was superseded by VBA in 1993.

I'm not entirely sure which version of Excel is in the picture on that page (97?), but it's old enough so that some of the technique has changed. For Excel 2007 (and probably later versions):

1. On the "Formulas" ribbon, click "Define Name"
2. Name your name, "CellHasFormula"
3. In the "Refers to:" box, type, "=GET.CELL(48,INDIRECT("rc",FALSE))"
4. Click "OK"
5. Select the cells for which you want the conditional formatting rule to apply
6. On the "Home" ribbon, click "Conditional Formatting" and select "New Rule…" from the menu.
7. Select rule type "Use a formula…"
8. In the rule description formula box, type, "=CellHasFormula"
9. Change your formatting to the desired style using the "Format…" button
10. Press "OK"

In all cases, inverted commas are not included in the values to be used. Be careful of typos: if you make a mistake you probably won't get any error messages (it just won't work).

For the curious, there is a description of the mechanics on the source site. The important point is that "48" is a magic number which instructs the GET.CELL() function to identify cells which contain formulae. Unfortunately, "=GET.CELL(…)" is not a valid argument to the Conditional Formatting rules engine but is to the Define Name engine. It's worth noting that Conditional Formatting rules also don't allow searching for '=' at the start of a cell (anywhere but the start is fine), which would have made this a lot easier.

A complication with using Excel 2007 upwards is that the standard file type (with the .xlsx extension) does not support macros because of potential security issues with passing around files in which they're used. You'll need to use the macro-enabled .xslm format instead. Template files in the .xltm format are fine. When reloading, you may also be told that macros have been disabled. If so, click "Options" on the message to remedy the situation and hit F9 to refresh the formatting once you've done so.

Does anyone know of a better way? Please let me know!

Squeezebox: Longer Timeout for Alarms

At home, we use Logitech Squeezeboxes for all our listening pleasures, including our bedroom where we use the alarm function to wake us up in the morning. The alarm has never been outstanding (a few years ago, a separate plugin was necessary to get reasonable functionality) and although it’s improved in fits and starts over the years, it’s still not a paragon of outstanding design.

For us, the worst problem is that the length of time the squeezebox plays for after the alarm has been triggered is restricted to 90 minutes or less (you can have it play indefinitely, but if you want it to switch itself off at all, 90 minutes is the cut-off). As the Today programme is on for two hours after we wish to be woken up, two hours is the time I want the alarm to last for.

It turns out the restriction is purely down to the user interface and not a limitation of the system itself. I seem to recall posting a feature request for the slider to become logarithmic, which would allow fine control at the 5 / 10 minute end of the scale and longer periods at the other, but it’s never been implemented. I can’t really complain: it’s open source so I should have done it myself. One reason I haven’t is that it’s relatively easy to get around just by editing the relevant prefs file.

On my Ubuntu system, this is /var/lib/squeezeboxserver/prefs/server.prefs; Windows users will probably find it in C:\Documents and Settings\All Users\Application Data\Squeezebox\prefs\server.prefs (according to Logitech's documentation, although newer Windows versions may have a slightly different path). It seems wise to stop the server while you make your edits: I found that changes can occasionally be overwritten.

The section for each player is headed by its MAC address (look in the "Player" tab of the settings pages of the web interface: it's listed in the "Basic Settings" section). Incidentally, the all-zeros MAC address at the start of the file is used as a template when new players are added. Skip through all the _ts stuff and you’ll find several entries starting “alarm”. As you might guess, alarmTimeoutSeconds is the value of interest and this needs to be set accordingly. There are 3600 seconds in an hour, so I have 7200 set. Save, restart the server and you’re done.

A couple of notes. Firstly, this is obviously only applicable to those running Logitech Media Server* at home: if your alarm is set using Logitech’s online system (MySqueezebox.com*) you’re out of luck. I have no idea how the online system works and it may not be subject to the same restrictions anyway. Secondly, be careful about changing the settings using the web interface afterwards – the UI pages will obviously overwrite what’s in the prefs file and you may need to reset the timeout value if you change anything else on the alarm page (other settings changes are fine). As a result, if you edit your alarm times frequently, you may find this process is not worth the trouble.

You can set the alarm time here too (“alarms: <number>: _time”), which makes things easier if you don’t want to disturb the timeout you’ve set, but you will need to calculate the time you’re after in seconds form midnight if you do.

* Correct this week, but nomenclature changes frequently.

Apache mod_rewrite & CodeIgniter

This article isn’t really about CodeIgniter. I’m getting to grips with that at the moment, so I might write some more about it in the future. It is about Apache’s mod_rewrite module and trying to get it to work in a way that’s useful on a dev server for the way CodeIgniter (and other PHP) projects are set out.

What I wanted was to have a single server (i.e. one virtual host) with space for several different projects, or branches of a project. In my opinion, the easiest way to access each project is just to use http://server/project/ in the browser (there are other ways – notably virtual hosts – but they usually require configuration for each new project and / or each new dev machine). With simple websites, it’s fine to put each project in a sub-folder and access them as suggested. However, that does ignores a recommendation for CI projects and one that I think should be followed on any web project and that is to move code that does not need to be publicly accessible outside of the browsable section of your file system (in this case, CI’s “system” and “application” folders should be outside “webroot”, or whatever you want to call it).

My goal was to have the dev server set up so projects could be moved between it and a production server without modification and to have each project wholly contained in its own folder, which means that each project needs its own “webroot” and its own space outside “webroot”. I therefore want every request to //server/project/index.php to be rewritten to //server/project/webroot/index.php (and of course similar for other files in other folders below webroot): in essence, “webroot” needs to be injected after every project name. This means that files and folders other than "webroot" in the project folder become inaccessible to the browser, which isn’t just a matter of convenience for the developer, it means that no resources can be accidentally accessed outside the correct area of the web server’s file system and all relative links (stylesheets, images, etc.) must be properly located.

The first thing I learned is to put the rules directly in the <virtual host> section and not in a <directory> tag wherever possible. There are two reasons for this. Firstly, it’s more efficient – Apache deals with rewriting much faster if it’s not done on a per-folder basis which is because of (at least in part) the second reason, which is that <directory> entries (and .htaccess files in particular directories, which are equivalent) can be parsed multiple times as the request is processed. This can cause major headaches for the unwary because there’s nothing to stop Apache deciding it needs to run through the rules again (in fact, it always seems to do so if the URL has been rewritten) and rather than starting with the original URL, you get the modified one. This means that you can get into an infinite loop if you, say, simply add something on to the end of whatever URL comes in.

The difference between behaviour for rules located in different sections of the config file is not limited to multiple passes, unfortunately. The other thing that changes is the content of some of the variables that you can make use of in the rules. For this reason it is important to check (and potentially modify) any rules you see suggested unless you’re sure that the rules were designed to go in the same place that you want to put them.

I ended up with the following, the second part of which adds index.php after project names (if not present), whilst retaining the rest of the URL as parameters. It’s based on examples in the CodeIgniter documentation:

# Inject 'webroot/' if request starts with a valid folder
# and '/webroot' is not already 2nd folder
RewriteCond %{DOCUMENT_ROOT}$1 -d
RewriteCond $2 !/webroot
RewriteRule ^(/[^/]+)(/?[^/]*)(.*) $1/webroot$2$3

# Rewrite any */webroot/* file request to index.php
# Don't rewrite if file exists OR it's already
# index.php (even if 404)
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !/index\.php$
RewriteRule ^(/[^/]+/webroot)/?(.*)$ $1/index.php/$2

I've used %{REQUEST_FILENAME} in conditions for the second rule. Although there are several other variables with similar content, be careful which you choose to use in situations like that above: not only do the values of some of them change depending on the location of the rules within the Apache config files, but I found that some of them had their contents rewritten by earlier rules and some did not (and I found no reference to this in the mod_rewrite documentation).

Linux Fileserver and ClamFS

I recently needed to provide a file server for a client that would work with Windows and OS X clients. For reasons of cost and maintenance we decided to use Ubuntu LTS Server. We also wanted anti-virus scanning as customer files are introduced to this server regularly. I decided to use the popular, open source ClamAV engine, with ClamFS providing the on-access scanning. I want to talk briefly about ClamFS in general, because there isn't much comment on it that I can find and then about a specific problem I had, because the solution is not necessarily obvious and uses an interesting feature of samba.

ClamFS seems to be most straightforward way to provide on-access scanning with ClamAV. It's a FUSE based daemon that mirrors one part of the file system to a mount point elsewhere, providing on-access protection for reads and writes to the mirrored version. I discovered the following about it:

1. The version I installed from the Ubuntu repository doesn't include an init.d script – adding a line to rc.local seems to be the preferred method of boot time initiation. You can, of course, write your own init.d script
   
2. The config file is written in XML, rather than the more readable and more easily editable (certainly on a GUIless server) familiar format that pretty much every other Unix-based config file uses. You need to include the config filename when starting ClamFS
   
3. There is apparently no way to stop the process other than using kill and then manually umounting the FUSE mount associated with it
   
4. Lack of permissions caused a bit of difficulty – the ClamAV user might need some additional permissions before your users can read and write protected files
   
5. There is little documentation; a tutorial taking new users through the steps of installation and configuration would make its use clearer
   
6. Once set up, it seems to work fine: I've had no problems with it.
   

My configuration is as follows: Truecrypt volumes (which are normal files, stored at a point we'll call location A) are mounted at another point in the filesystem (location B) and ClamFS mounts a copy of B to a third point (location C). Location C is then used for the samba share path.

I wondered if having ClamFS start at boot time and mounting a copy of B elsewhere would prevent TC (which doesn't start at boot time) mounting a volume to B later on, but it turns out mounting volumes "underneath" an existing ClamFS mount works fine.

I had another problem though. Because I have more than one share and more than one encrypted volume, I configured ClamFS to protect the directory above the one in which all the TC drives were mounted. Because of this (or maybe because of some other aspect of the redirection), the free space reported by samba was not that of the individual drives mounted within the ClamFS protected directory, but the space on the drive that contained those mount points (or the point which the ClamFS was mounting to, I'm not sure which as they are on the same partition).

This can be more than an annoyance because Windows systems from Vista onwards actually check this free space before attempting to write a file. If there isn't room, you can't write. In my case, reported size was on a partition that was almost full of TC volumes, so the reported free space (and therefore the maximum file size that could be written by Windows 7 clients) was severely curtailed.

There are two possible ways round this. The most obvious is to only allow ClamFS to mount to and from points inside any TC volumes you want to share. This will cause you headaches if either you have many shares and only want to have ClamFS configured to protect one directory or ClamFS needs to be started before TC mounts its volumes (common, because manual intervention is usually needed on TC mounts for security reasons).

The second solution is to use a feature of samba which allows you to override the internal free space code with a method of your design. The smb.conf man page explains the details – essentially you need to provide a command (writing a script seems to be the most common solution) that will return two numbers. These give the total number of 1K blocks in the filesystem and the number that are free, respectively. The man page makes a suggestion which I tailored slightly:

#!/bin/sh
df -P $1 | tail -1 | awk '{print $2,$4}'

The "-P" switch (added to the df command) forces the results for each drive onto a single line. If you don't do this and the path reported for the partition is longer than 20 characters, a line break is inserted and the positional parameters to awk will be incorrect.

You then need to make sure the definition in smb.conf for each affected share contains the following:

[Sharename]
   …
   path = /path/to/share  # loc C
   dfree command = /path/to/script.sh /path/to/TC/mount  # loc B

A quick side note: samba calls the script with the location it is trying to ascertain the size of as a first parameter. We've included a first parameter here, which simply pushes the samba-appended one into second position (which is then ignored). I have read that samba may call the script with the parameter "/", having chrooted to the share point before executing the script. I haven't investigated exactly what is happening in my test or production installations, but both work with the procedure I have outlined and this would not be the case if any chrooting were going on. I can only conclude that this is not the behaviour of current versions of samba (I'm using 3.4.7, courtesy of Ubuntu 10.04 LTS) or something else about my environments is altering that behaviour. I'd be interested to hear about different experiences.

Installing Linux VMWare Tools on Ubuntu

Infrequently, I create a new VMWare Linux VM. I do this just infrequently enough that I can't quite remember the procedure for installing VMWare Tools to the VM. This is documented in lots of places, I'm sure, and I normally try to stay away from repetition of readily available material… but I can never find it when I want it. So, as an aide memoire to myself and (hopefully) a handy reference for anyone else who needs to go through the procedure, here are the necessary steps on Ubuntu. I'm using Ubuntu Server 11.04 (with no GUI) but the steps should work on other versions (including desktop versions if you open a terminal: the instructions assume you have a shell open already).

Many of the steps here will be obvious to most users, but I've detailed everything so you can (if you wish) just copy and paste the lot (almost - see the notes) into shell scripts which will get the job done quicker. And those just starting out will also have a reference they can use.

1. [Optional] Change the kernel. Even with the server install I did to write this article, the generic kernel was installed by default even though a kernel optimised for server operations is available. Not only that, but there is a version of the server kernel trimmed down to have only what is necessary for use in common virtualised platforms, including VMWare

# Install latest kernel version
sudo aptitude update
sudo aptitude install linux-virtual

# Reboot, so the new kernel is running when the tools
# package is built and the correct headers will be
# selected in step 3
sudo shutdown -r now

2. Attach the Tools ISO to the VM. In vSphere Client, you can right-click the VM in the inventory and select Guest -> Install / Upgrade VMWare Tools
3. Install tools, with necessary packages (I'm assuming you are starting in your home folder or somewhere equally appropriate for putting the tools installation directory)

   # Most commands need root access. You can use 'sudo'
   # where necessary instead
   sudo su
   
   # Update apt package database (if you didn't earlier)
   aptitude update
   
   # Install packages necessary to build tools
   aptitude install build-essential linux-headers-`uname -r`
   # note backticks around uname command, not ordinary
   # inverted commas
   
   # No suitable mount point existed in my default install:
   # create one
   mkdir /media/dvd
   
   # Mount tools image and extract tarball
   mount /dev/dvd /media/dvd
   tar -xzf /media/dvd/VMwareTools-*.tar.gz
   # You can use auto-complete above: it's just one file
   
   # Run install script
   cd vmware-tools-distrib/
   ./vmware-install.pl -d  # -d auto-accepts all defaults
   
   # Tidy up and exit root shell
   cd ..
   rm -rf vmware-tools-distrib/
   umount /media/dvd  # the script usually does this for you
   exit
   

Notes

1. The kernel headers are installed by default on Ubuntu, so the linux-headers-* package is only necessary if the kernel has been changed since installation.
2. The "uname" command in the install list ensures that the package for the running kernel is selected. If you've just installed a kernel using one of the metapackages listed above, it will be the latest one and headers can be installed simply with "linux-headers-virtual" (for example).
3. To initialise the tools, the "/etc/bin/vmware-config-tools.pl" script needs to be run. If you used '-d' or allowed the script to run this itself (it prompts for this in interactive mode), this will already have been done, but it can be useful to know about this separate step in case of problems.
4. If you put the second set of commands into a script, you'll need to remove "sudo su" from the start and run the script as root. "su" opens a new shell and the commands from the rest of the script will not be passed into it if you run as-is.

Once the tools are installed, updates can be performed automatically from the host so there is rarely a need to refer back to this process for an existing machine

Temporary PATH Additions: Modifying the standard CMD Here Extension

A relatively common shell extension for Windows systems is to have right-click for folders in Explorer offer the option to open a cmd prompt window with that folder as the current working directory (CWD). I seem to have added this to any Windows installation that I've used for any period of time. This is often called "cmd here" or "command here" and Microsoft provide an installable for this function in their PowerToys collection.

In fact, this feature is so useful that it's built-in to OSes from Vista onwards, but to access it you need to hold "shift" while you right-click the item and it only works for folders (some versions of the extension let you right-click a file and have the command prompt open in the folder containing that file). 

Something I find useful from time to time (and which I've never seen elsewhere) is to have file and folder context menus open cmd windows with the folder concerned added to the PATH environment variable, just for that session. This is great for uncommon or temporary use of a folder containing one or more exes without permanently bloating your PATH variable. Some programs with command line interfaces (such as Visual Studio) provide Start Menu shortcuts that open a cmd window with PATH modified for that window only and what I'm suggesting here is similar (but more dynamic).

All that happens when installing the "cmd here" extension is the addition of a few registry entries, and so I made a typical version of this add-on (based on the entries in the Win 7 registry and this web page) and adapted it. Save the following lines as a .reg file and you can add this to your file / folder context menus too:

Windows Registry Editor Version 5.00

    [HKEY_CLASSES_ROOT\*\shell\pathhere]
    @="Cmd with &Path here"
    ;"Extended"=""
     
    [HKEY_CLASSES_ROOT\*\shell\pathhere\command]
    @="cmd /k path %W;%%PATH%% && pushd %%USERPROFILE%%"

    [HKEY_CLASSES_ROOT\Directory\shell\pathhere]
    @="Cmd with &Path here"
    ;"Extended"=""

    [HKEY_CLASSES_ROOT\Directory\shell\pathhere\command]
    @="cmd /k path %L;%%PATH%% && pushd %%USERPROFILE%%"

    [HKEY_CLASSES_ROOT\Directory\Background\shell\pathhere]
    @="Cmd with &Path here"
    ;"Extended"=""

    [HKEY_CLASSES_ROOT\Directory\Background\shell\pathhere\command]
    @="cmd /k path %V;%%PATH%% && pushd %%USERPROFILE%%"

The entries for \Directory\Background enable the same effect by clicking in the empty space in an Explorer window: you just get the current folder the window is displaying.

As an aside, Raymond Chen explains the difference between the \Folder and \Directory classes in this blog post. Note that what the registry (and Raymond) are referring to here as "directories" are called "file folders" in parts of the Windows UI. We have used the "Directory" branch because it makes no sense to have virtual folders as targets for this sort of extension.

You'll have noticed that each entry's root has a commented-out, empty string called, "Extended". If these are un-commented (and you're using Vista onwards), commands will be added only to the extended context menu, available with a Shift-right-click.

You could copy the \Directory\shell entry to the \drive\shell branch if you wanted to provide the same facility for drive roots. You may also want to specify a different CWD, and this is controlled by the appended "pushd" command. If you delete this (everything from the first ampersand onwards, but don't forget to retain the closing quotation mark), the CWD will be the folder in which the context-menu was opened.

One thing I haven't sorted out completely is the variable expansion. When context menu entries for files are activated, %D, %L and %V all hold the filename with its path and %W just holds the path; some other letters hold other cryptic values. The details for directories seem similar, but my tests for \directory\background consistently crashed Explorer. The MS implementation of "cmd here" in Win 7 uses %L for \directory and %V for \directory\background. I can't find any documentation listing all these variables and I'd be interested to know if anyone's come across any.

Usual disclaimers apply (although it's highly unlikely to do anything you don't want) – specifically, I haven't tested this on anything other than one Win 7 Pro 32-bit installation. However, I'd expect it to work pretty much across the board, although older OSes (XP / 2003 and previous) will probably ignore the "Extended" key.