Thursday, 2 February 2017

VBox shared folders: making them work in a LAMP stack

The Setup

This is one of those “tutorial” posts that is really written for me to remember how I did something. It’s for a specific set of circumstances, but I won’t be the only person to be in these circumstances and there is generally applicable info too.

My problem was that I wanted to edit a PHP project on my Windows Desktop, but have it served from a LAMP environment similar to the one in which it would run. In the past, I’ve used a VM on a remote server for this. It works fine, but you need a technique to synchronise files on the dev machine with the server and you need the server available. This time, to keep everything local, I decided to use a VM on my dev machine, using VirtualBox. As well as the lower admin overhead of keeping everything on my dev machine, it seemed a major advantage would be using “shared folders”: the LAMP machine could see my local dev folder and serve from it directly. This means I can tweak a setting in the VBox settings panel if I want to work on a different project, which is very easy and I don’t have to worry about the availability of a VMWare instance somewhere else.

I started with an Ubuntu 16.04 LTS server that had OpenSSH and LAMP added as part of the initial install process, through Ubuntu’s handy task packages, installed Guest Additions and set networking to “bridged” so the server would be available across the local network with it’s own routable IP address. I also shared my project folder on the host using a permanent “shared folder” addition to the settings for my LAMP VM. All this was fine.

Then I hit a snag…

The Problem

By default, apache looks at /var/www/html for web pages. I assumed I could just mount the shared folder device over the top of this and everything would work - but it didn’t:

  • I started by adding a traditional fstab entry, but the VBox shared folder kernel module isn’t loaded by the time fstab is parsed.
  • I read a suggestion that vboxsf should be added to /etc/modules, which should then load the module early enough in the boot process for fstab parsing. My mount point wasn’t established.
  • I put noauto in the options for the fstab entry and then added a command to mount it in /etc/rc.local. Nothing was mounted.
  • I Removed the fstab entry altogether and put a complete mount command in /etc/rc.local. Once again, nothing happened.

It’s worth noting that the mount commands I was using worked fine if used from a terminal after logging in; both a complete mount command and one that referenced and activated an entry in fstab. But I couldn’t get them to work at any point in the boot process and there were no particularly useful log messages.

The Fix

Now, there is an option in the “shared folder” creation process in VBox, “Auto-mount”. Unless I had this checked, I was receiving a “Protocol Error” (no further explanation) wherever and whenever I tried to mount the share. So I had this ticked and was going to worry about removing it later. What it does is cause a mount point to be created automatically (/media/sf_<share name>). I wanted control of the mount point so I was trying to avoid this, but with nothing else working, the obvious solution was to simply use this point. I created a symlink:

ln -s /media/sf_hosthtml /var/www/html

and this solved the mount problem immediately. I cursed myself for not taking the obvious, expedient route much sooner. I have since read that using the auto-generated mount point is the “preferred” option for mounting shared folders and if this is the case, I can see why.

The last problem was simply permissions. The original /var/www/html/ folder’s contents are world readable, but the VBox replcemant is not:

-rw-r--r-- 1 root root 12K Feb 1 12:55 index.html
vs.
-rwxrwx--- 1 root vboxsf 1.4K Feb 1 17:00 index.php

As there is no obvious way to alter the permissions that windows files are given by VBox’s shared folders module, I solved this by adding the apache user, www-data to the vboxsf group:

usermod -a -G vboxsf www-data

This may not be ideal because unlike the original permissions on the default index.html file, group permissions now allow apache to write to these files, but it’s a closed test system and it’s good enough for me.

Friday, 16 September 2016

Bash on Ubuntu on Windows

I ran some Linux commands today. I do that a lot, but it’s the first time I’ve run most of them from a command prompt I opened in windows, and had them act on that Windows machine.

This isn’t unheard of, of course. The Cygwin project has provided Win32 builds of Linux tools for years and virtual machines provide another route to these things but they have their problems. Cygwin requires that binaries are recompiled for Windows and while they provide a dll containing much of the standard POSIX API, there’s a substantial amount of friction in getting anything new to work. Sure, that’s been done for you in a lot of cases, but even then, the results aren’t always perfect.

VMs have different problems: you have to allocate memory and disk space in large chunks and then put up with the resource overhead that running a whole OS within an OS creates. After that, you have to access them – typically you’re looking at a new IP address for the VM and ssh, VNC or some kind of proprietary access method specific to the hosting software is necessary to overcome the rigid separation between Windows and the VM. Operating on or moving data from one to the other is hard work.

This is new.

With the first anniversary update (build 1607) of Windows 10 (x64 only), comes Ubuntu, which is not something I’d have put much money on. More specifically, the user space root filesystem (including all the binaries) from Ubuntu 14.04 LTS “Trusty” can be installed inside Windows. It is quite simply a command-line only (no X) Ubuntu installation, bit for bit, with just the Linux kernel and the boot code taken out.

To make it useful, MS (with Canonical’s help) have created a new Windows subsystem that allows the NT kernel to service all the calls that would normally be made to the Linux kernel. And it’s that simple. All the Ubuntu software can behave as it normally does, making the syscalls it normally makes and getting the responses it expects to get, so nothing needs to be recompiled even though there is no Linux kernel running.

You get apt, so you can install stuff from the Ubuntu software repository, anything you like. Software (including servers) get access to Windows’ own ports and the Windows file system. You can write C in Visual Studio Code and compile it with the Ubuntu version of gcc. You get bash, but if that’s not your thing, apt-get install zsh or something more esoteric. In short, this is the perfect way to run grep and sed on your windows files.

Now, it’s not all rosy. It is currently a beta release and not everything works as it should yet. There are apparently about 350 syscalls in the API being emulated and not all of them are used, so the devs have implemented a subset based on what they believe to be useful and what has been possible, given the inevitable restrictions. There is no hardware access and no graphical implementation beyond what you get in a normal terminal. Also, the talk back in March was that 16.04 was just around the corner and, now in September, it hasn’t arrived yet.

Furthermore, everything Linuxy is done in the bash shell and you can’t run Win32 exes from there, nor can you run Linux ELFs from a normal Windows command prompt, unless you invoke the bash shell as a wrapper (and if you can pipe output from one to the other, I’ve yet to work out how). For me though, there’s so much you can do that the restrictions don’t feel like restrictions and this is still a pre-release version.

Apart from the technical details, the other aspect of this that intrigued me was the political side. MS and Canonical have never been obvious bedfellows and although the stance MS has taken on Linux and open source in general has significantly mellowed under Satya Nadella (cf Steve Ballmer making SMB changes to deliberately stop Samba working), collaboration like this is still a surprising step. In essence, The Windows Way is being side-lined for a rival. I came to the conclusion that this does make sense for both parties, but it’s really Canonical who are riding the tiger.

From Microsoft’s perspective, they’ve come to realise that Linux isn’t going to go away. Whether antagonism towards it helps or hinders their own cause is an interesting and probably nuanced question but given that people are using it, devs especially, it makes some sense for them to offer tools to those devs inside the Windows marquee that they used to have to go elsewhere for – people using Linux (and at risk of moving to Linux exclusively) now have less of a reason to do so. Is there a downside? Well, yes: devs who haven’t been exposed to Linux previously may feel more comfortable with it when server OS options are being considered. Big deal? Probably not massive.

Canonical’s position is more interesting and it may take a while to see how it’s going to play out. Their stuff is freely available, of course, and MS could have just come and taken it, but they’ve been actively engaged in this project (even showing up on panels at BUILD) and as leaders in a world where MS is frequently seen as the enemy, “selling out” is a charge that might come their way.

What’s the upside? Well, Windows devs have often dabbled in Linux for various reasons – usually server related, but also for tools that work really well at solving problems that have never been fully solved on Windows (or where the tools exist but are less at home). Those devs are now going to be doing that with Ubuntu rather than Fedora or SUSE. So what are they going to pick, familiar as they will be with Ubuntu, when they need a server for something (or even choose to use a Linux desktop)? It’s not rocket science to see this as a really, really effective advertising campaign. But what are they actually helping Microsoft do?

In the past, MS was known for its strategy of taking things like “standards” and building on them, bending them to their own way of doing things and forcing the de facto standard that resulted onto everyone else, regardless of whether it benefited the rest of the world or not. They can’t alter the Ubuntu code directly (although they could start offering pull requests for modifications), especially since Ubuntu aren’t themselves responsible for most of the most commonly used utilities.

What they could do, ultimately, is threaten a different freedom: the freedom from Windows in the server market. This "freedom" is forced (welcome or not) on projects who choose (or need) Linux tools. Soon, perhaps, there will be a server OS that can run software from both camps and at that point, it’s Linux that is being side-lined as Windows is suddenly there, wrapped around it, begging to be given some jobs to do.

So the balancing act is this: Canonical are stealing a march on their competitors and more selflessly, pushing Linux into the Windows world in an astoundingly direct way, but are they also greasing the way for Windows in spaces which would have been solely Linux enclaves previously?

Tuesday, 23 April 2013

It sounds great… but what does it actually do?

I read an article this morning on TechCrunch by the co-founder of a mobile app company (they make “Bump”, which shares contacts or other data through phones’ NFC chips). He was talking about the difference in understanding of technology between the creators of that technology verses their target market. It’s not a bad article, although the points are reasonably well rehearsed. But I think he’s misunderstood the problem slightly.

According to David Lieb, the article's author, for the mass market, things should be very, very simple, at the expense of features. Bump’s creators apparently discovered this idea and helpfully named it for us, “cognitive simplicity”, although I doubt the notion will be a surprise to many. The other side of the coin is “cognitive overhead”, a more widely used term for the brain power necessary for the uninitiated to understand a point or operate a tool. With delightful irony, a number of the comments below the article (as well as complaining about the rehashing of well-known ideas) pointed out the “cognitive overhead” of giving jargon names to straightforward concepts.

The article implies that technology producers often simply do not understand that non-techy users could struggle to comprehend how something they’ve designed operates. This is almost certainly correct in some cases: it’s easy to overlook the learning curve of an item you’ve been intimately familiar with since its inception. By and large though, I think it’s a problem of scale rather than intent – it’s actually quite hard to make something simple without losing the essence of the product or what differentiates it from its competition. And this is where people fall down – it’s a much bigger job than people often think and it’s compounded by the fact that you can stop at any point and still have a perfectly functional product.

And one of the difficulties is going too far. Albert Einstein said that things should be made as simple as possible, but no simpler. Several article commentators who use Bump complained about vanishing features in the latest version, presumably removed to increase simplicity. It’s obviously going to annoy a certain proportion of your users if you take functions away and perhaps the trade-off here is the right one, but it makes it clear that it is a balancing act.

In my opinion, a neglected counterpart to having an easily understandable product is having an easily describable product, or at least making an effort to describe it. Distressingly common is the inability of software companies (and to some extent providers of other forms of technology) to properly explain something they want to sell to me (or sometimes give to me, for which I can’t be quite so belligerent). This goes all the way from small open source projects to massive product suites from big companies. In the latter case, it’s usually because they have a marketing department who feel it’s their duty to talk up the product without worrying too much about whether it will actually fit the needs of potential customers. Examples can be found in the whole gamut between.

I don’t want to know how I’ll feel using your gadget. I don’t care (initially, at least) whether it’s been designed with solid engineering principles or by throwing a bag of C++ keywords in the air and seeing how they land. I especially don’t want to be greeted with a list of the minor functionality tweeks between versions 2.4.2.17 and 2.4.2.18, which seems particularly common in open source product pages. I want to know what it does. I think that should be more obvious than it would appear to be.

So I’d like to make a plea for simplicity myself, in the sales pitch: if the first page an interested visitor will see on your product’s website or information brochure does not contain a brief, clear description of what the program / gadget / vegetable can be used for (and I don’t mean, “Use the Big Bright Green Pleasure Machine and it will make your TCO lower / your life better / your partner sexier”, unless one of these purposes is its sole function) then your marketing is rubbish.

Wednesday, 9 May 2012

Excel: Conditional Formatting of Formulae


Yesterday, I was working with an Excel sheet and wanted to calculate a table of values, but be able to override some of the values. To make it obvious which values were calculated and which typed in, I decided to make the calculated values grey and leave the overridden values as the default black text.

Excel is massively flexible, so I didn't think I'd have much trouble – something in the Conditional Formatting arena would do what I needed. To my surprise, although I could do it, the only technique I found (on the j-walk website) relies on an obscure part of Excel that has almost been lost to memory: the XLM macro language, which was superseded by VBA in 1993.

I'm not entirely sure which version of Excel is in the picture on that page (97?), but it's old enough so that some of the technique has changed. For Excel 2007 (and probably later versions):
  1. On the "Formulas" ribbon, click "Define Name"
  2. Name your name, "CellHasFormula"
  3. In the "Refers to:" box, type, "=GET.CELL(48,INDIRECT("rc",FALSE))"
  4. Click "OK"
  5. Select the cells for which you want the conditional formatting rule to apply
  6. On the "Home" ribbon, click "Conditional Formatting" and select "New Rule…" from the menu.
  7. Select rule type "Use a formula…"
  8. In the rule description formula box, type, "=CellHasFormula"
  9. Change your formatting to the desired style using the "Format…" button
  10. Press "OK"
In all cases, inverted commas are not included in the values to be used. Be careful of typos: if you make a mistake you probably won't get any error messages (it just won't work).

For the curious, there is a description of the mechanics on the source site. The important point is that "48" is a magic number which instructs the GET.CELL() function to identify cells which contain formulae. Unfortunately, "=GET.CELL(…)" is not a valid argument to the Conditional Formatting rules engine but is to the Define Name engine. It's worth noting that Conditional Formatting rules also don't allow searching for '=' at the start of a cell (anywhere but the start is fine), which would have made this a lot easier.

A complication with using Excel 2007 upwards is that the standard file type (with the .xlsx extension) does not support macros because of potential security issues with passing around files in which they're used. You'll need to use the macro-enabled .xslm format instead. Template files in the .xltm format are fine. When reloading, you may also be told that macros have been disabled. If so, click "Options" on the message to remedy the situation and hit F9 to refresh the formatting once you've done so.

Does anyone know of a better way? Please let me know!

Friday, 9 March 2012

Squeezebox: Longer Timeout for Alarms

At home, we use Logitech Squeezeboxes for all our listening pleasures, including our bedroom where we use the alarm function to wake us up in the morning. The alarm has never been outstanding (a few years ago, a separate plugin was necessary to get reasonable functionality) and although it’s improved in fits and starts over the years, it’s still not a paragon of outstanding design.

For us, the worst problem is that the length of time the squeezebox plays for after the alarm has been triggered is restricted to 90 minutes or less (you can have it play indefinitely, but if you want it to switch itself off at all, 90 minutes is the cut-off). As the Today programme is on for two hours after we wish to be woken up, two hours is the time I want the alarm to last for.

It turns out the restriction is purely down to the user interface and not a limitation of the system itself. I seem to recall posting a feature request for the slider to become logarithmic, which would allow fine control at the 5 / 10 minute end of the scale and longer periods at the other, but it’s never been implemented. I can’t really complain: it’s open source so I should have done it myself. One reason I haven’t is that it’s relatively easy to get around just by editing the relevant prefs file.

On my Ubuntu system, this is /var/lib/squeezeboxserver/prefs/server.prefs; Windows users will probably find it in C:\Documents and Settings\All Users\Application Data\Squeezebox\prefs\server.prefs (according to Logitech's documentation, although newer Windows versions may have a slightly different path). It seems wise to stop the server while you make your edits: I found that changes can occasionally be overwritten.

The section for each player is headed by its MAC address (look in the "Player" tab of the settings pages of the web interface: it's listed in the "Basic Settings" section). Incidentally, the all-zeros MAC address at the start of the file is used as a template when new players are added. Skip through all the _ts stuff and you’ll find several entries starting “alarm”. As you might guess, alarmTimeoutSeconds is the value of interest and this needs to be set accordingly. There are 3600 seconds in an hour, so I have 7200 set. Save, restart the server and you’re done.

A couple of notes. Firstly, this is obviously only applicable to those running Logitech Media Server* at home: if your alarm is set using Logitech’s online system (MySqueezebox.com*) you’re out of luck. I have no idea how the online system works and it may not be subject to the same restrictions anyway. Secondly, be careful about changing the settings using the web interface afterwards – the UI pages will obviously overwrite what’s in the prefs file and you may need to reset the timeout value if you change anything else on the alarm page (other settings changes are fine). As a result, if you edit your alarm times frequently, you may find this process is not worth the trouble.

You can set the alarm time here too (“alarms: <number>: _time”), which makes things easier if you don’t want to disturb the timeout you’ve set, but you will need to calculate the time you’re after in seconds form midnight if you do.

* Correct this week, but nomenclature changes frequently.

Friday, 24 February 2012

Apache mod_rewrite & CodeIgniter


This article isn’t really about CodeIgniter. I’m getting to grips with that at the moment, so I might write some more about it in the future. It is about Apache’s mod_rewrite module and trying to get it to work in a way that’s useful on a dev server for the way CodeIgniter (and other PHP) projects are set out.

What I wanted was to have a single server (i.e. one virtual host) with space for several different projects, or branches of a project. In my opinion, the easiest way to access each project is just to use http://server/project/ in the browser (there are other ways – notably virtual hosts – but they usually require configuration for each new project and / or each new dev machine). With simple websites, it’s fine to put each project in a sub-folder and access them as suggested. However, that does ignores a recommendation for CI projects and one that I think should be followed on any web project and that is to move code that does not need to be publicly accessible outside of the browsable section of your file system (in this case, CI’s “system” and “application” folders should be outside “webroot”, or whatever you want to call it).

My goal was to have the dev server set up so projects could be moved between it and a production server without modification and to have each project wholly contained in its own folder, which means that each project needs its own “webroot” and its own space outside “webroot”. I therefore want every request to //server/project/index.php to be rewritten to //server/project/webroot/index.php (and of course similar for other files in other folders below webroot): in essence, “webroot” needs to be injected after every project name. This means that files and folders other than "webroot" in the project folder become inaccessible to the browser, which isn’t just a matter of convenience for the developer, it means that no resources can be accidentally accessed outside the correct area of the web server’s file system and all relative links (stylesheets, images, etc.) must be properly located.

The first thing I learned is to put the rules directly in the <virtual host> section and not in a <directory> tag wherever possible. There are two reasons for this. Firstly, it’s more efficient – Apache deals with rewriting much faster if it’s not done on a per-folder basis which is because of (at least in part) the second reason, which is that <directory> entries (and .htaccess files in particular directories, which are equivalent) can be parsed multiple times as the request is processed. This can cause major headaches for the unwary because there’s nothing to stop Apache deciding it needs to run through the rules again (in fact, it always seems to do so if the URL has been rewritten) and rather than starting with the original URL, you get the modified one. This means that you can get into an infinite loop if you, say, simply add something on to the end of whatever URL comes in.

The difference between behaviour for rules located in different sections of the config file is not limited to multiple passes, unfortunately. The other thing that changes is the content of some of the variables that you can make use of in the rules. For this reason it is important to check (and potentially modify) any rules you see suggested unless you’re sure that the rules were designed to go in the same place that you want to put them.

I ended up with the following, the second part of which adds index.php after project names (if not present), whilst retaining the rest of the URL as parameters. It’s based on examples in the CodeIgniter documentation:
# Inject 'webroot/' if request starts with a valid folder
# and '/webroot' is not already 2nd folder
RewriteCond %{DOCUMENT_ROOT}$1 -d
RewriteCond $2 !/webroot
RewriteRule ^(/[^/]+)(/?[^/]*)(.*) $1/webroot$2$3

# Rewrite any */webroot/* file request to index.php
# Don't rewrite if file exists OR it's already
# index.php (even if 404)
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !/index\.php$
RewriteRule ^(/[^/]+/webroot)/?(.*)$ $1/index.php/$2
I've used %{REQUEST_FILENAME} in conditions for the second rule. Although there are several other variables with similar content, be careful which you choose to use in situations like that above: not only do the values of some of them change depending on the location of the rules within the Apache config files, but I found that some of them had their contents rewritten by earlier rules and some did not (and I found no reference to this in the mod_rewrite documentation).

Tuesday, 10 January 2012

Linux Fileserver and ClamFS

I recently needed to provide a file server for a client that would work with Windows and OS X clients. For reasons of cost and maintenance we decided to use Ubuntu LTS Server. We also wanted anti-virus scanning as customer files are introduced to this server regularly. I decided to use the popular, open source ClamAV engine, with ClamFS providing the on-access scanning. I want to talk briefly about ClamFS in general, because there isn't much comment on it that I can find and then about a specific problem I had, because the solution is not necessarily obvious and uses an interesting feature of samba.

ClamFS seems to be most straightforward way to provide on-access scanning with ClamAV. It's a FUSE based daemon that mirrors one part of the file system to a mount point elsewhere, providing on-access protection for reads and writes to the mirrored version. I discovered the following about it:

  1. The version I installed from the Ubuntu repository doesn't include an init.d script – adding a line to rc.local seems to be the preferred method of boot time initiation. You can, of course, write your own init.d script
  2. The config file is written in XML, rather than the more readable and more easily editable (certainly on a GUIless server) familiar format that pretty much every other Unix-based config file uses. You need to include the config filename when starting ClamFS
  3. There is apparently no way to stop the process other than using kill and then manually umounting the FUSE mount associated with it
  4. Lack of permissions caused a bit of difficulty – the ClamAV user might need some additional permissions before your users can read and write protected files
  5. There is little documentation; a tutorial taking new users through the steps of installation and configuration would make its use clearer
  6. Once set up, it seems to work fine: I've had no problems with it.

My configuration is as follows: Truecrypt volumes (which are normal files, stored at a point we'll call location A) are mounted at another point in the filesystem (location B) and ClamFS mounts a copy of B to a third point (location C). Location C is then used for the samba share path.

I wondered if having ClamFS start at boot time and mounting a copy of B elsewhere would prevent TC (which doesn't start at boot time) mounting a volume to B later on, but it turns out mounting volumes "underneath" an existing ClamFS mount works fine.

I had another problem though. Because I have more than one share and more than one encrypted volume, I configured ClamFS to protect the directory above the one in which all the TC drives were mounted. Because of this (or maybe because of some other aspect of the redirection), the free space reported by samba was not that of the individual drives mounted within the ClamFS protected directory, but the space on the drive that contained those mount points (or the point which the ClamFS was mounting to, I'm not sure which as they are on the same partition).

This can be more than an annoyance because Windows systems from Vista onwards actually check this free space before attempting to write a file. If there isn't room, you can't write. In my case, reported size was on a partition that was almost full of TC volumes, so the reported free space (and therefore the maximum file size that could be written by Windows 7 clients) was severely curtailed.

There are two possible ways round this. The most obvious is to only allow ClamFS to mount to and from points inside any TC volumes you want to share. This will cause you headaches if either you have many shares and only want to have ClamFS configured to protect one directory or ClamFS needs to be started before TC mounts its volumes (common, because manual intervention is usually needed on TC mounts for security reasons).

The second solution is to use a feature of samba which allows you to override the internal free space code with a method of your design. The smb.conf man page explains the details – essentially you need to provide a command (writing a script seems to be the most common solution) that will return two numbers. These give the total number of 1K blocks in the filesystem and the number that are free, respectively. The man page makes a suggestion which I tailored slightly:

#!/bin/sh
df -P $1 | tail -1 | awk '{print $2,$4}'

The "-P" switch (added to the df command) forces the results for each drive onto a single line. If you don't do this and the path reported for the partition is longer than 20 characters, a line break is inserted and the positional parameters to awk will be incorrect.

You then need to make sure the definition in smb.conf for each affected share contains the following:

[Sharename]
   …
   path = /path/to/share  # loc C
   dfree command = /path/to/script.sh /path/to/TC/mount  # loc B

A quick side note: samba calls the script with the location it is trying to ascertain the size of as a first parameter. We've included a first parameter here, which simply pushes the samba-appended one into second position (which is then ignored). I have read that samba may call the script with the parameter "/", having chrooted to the share point before executing the script. I haven't investigated exactly what is happening in my test or production installations, but both work with the procedure I have outlined and this would not be the case if any chrooting were going on. I can only conclude that this is not the behaviour of current versions of samba (I'm using 3.4.7, courtesy of Ubuntu 10.04 LTS) or something else about my environments is altering that behaviour. I'd be interested to hear about different experiences.

Wednesday, 15 June 2011

Installing Linux VMWare Tools on Ubuntu

Infrequently, I create a new VMWare Linux VM. I do this just infrequently enough that I can't quite remember the procedure for installing VMWare Tools to the VM. This is documented in lots of places, I'm sure, and I normally try to stay away from repetition of readily available material… but I can never find it when I want it. So, as an aide memoire to myself and (hopefully) a handy reference for anyone else who needs to go through the procedure, here are the necessary steps on Ubuntu. I'm using Ubuntu Server 11.04 (with no GUI) but the steps should work on other versions (including desktop versions if you open a terminal: the instructions assume you have a shell open already).

Many of the steps here will be obvious to most users, but I've detailed everything so you can (if you wish) just copy and paste the lot (almost - see the notes) into shell scripts which will get the job done quicker. And those just starting out will also have a reference they can use.
  1. [Optional] Change the kernel. Even with the server install I did to write this article, the generic kernel was installed by default even though a kernel optimised for server operations is available. Not only that, but there is a version of the server kernel trimmed down to have only what is necessary for use in common virtualised platforms, including VMWare
  2. # Install latest kernel version
    sudo aptitude update
    sudo aptitude install linux-virtual
    
    # Reboot, so the new kernel is running when the tools
    # package is built and the correct headers will be
    # selected in step 3
    sudo shutdown -r now
    
  3. Attach the Tools ISO to the VM. In vSphere Client, you can right-click the VM in the inventory and select Guest -> Install / Upgrade VMWare Tools
  4. Install tools, with necessary packages (I'm assuming you are starting in your home folder or somewhere equally appropriate for putting the tools installation directory)
    # Most commands need root access. You can use 'sudo'
    # where necessary instead
    sudo su
    
    # Update apt package database (if you didn't earlier)
    aptitude update
    
    # Install packages necessary to build tools
    aptitude install build-essential linux-headers-`uname -r`
    # note backticks around uname command, not ordinary
    # inverted commas
    
    # No suitable mount point existed in my default install:
    # create one
    mkdir /media/dvd
    
    # Mount tools image and extract tarball
    mount /dev/dvd /media/dvd
    tar -xzf /media/dvd/VMwareTools-*.tar.gz
    # You can use auto-complete above: it's just one file
    
    # Run install script
    cd vmware-tools-distrib/
    ./vmware-install.pl -d  # -d auto-accepts all defaults
    
    # Tidy up and exit root shell
    cd ..
    rm -rf vmware-tools-distrib/
    umount /media/dvd  # the script usually does this for you
    exit
    
Notes
  1. The kernel headers are installed by default on Ubuntu, so the linux-headers-* package is only necessary if the kernel has been changed since installation.
  2. The "uname" command in the install list ensures that the package for the running kernel is selected. If you've just installed a kernel using one of the metapackages listed above, it will be the latest one and headers can be installed simply with "linux-headers-virtual" (for example).
  3. To initialise the tools, the "/etc/bin/vmware-config-tools.pl" script needs to be run. If you used '-d' or allowed the script to run this itself (it prompts for this in interactive mode), this will already have been done, but it can be useful to know about this separate step in case of problems.
  4. If you put the second set of commands into a script, you'll need to remove "sudo su" from the start and run the script as root. "su" opens a new shell and the commands from the rest of the script will not be passed into it if you run as-is.
Once the tools are installed, updates can be performed automatically from the host so there is rarely a need to refer back to this process for an existing machine

Wednesday, 27 April 2011

Temporary PATH Additions: Modifying the standard CMD Here Extension

A relatively common shell extension for Windows systems is to have right-click for folders in Explorer offer the option to open a cmd prompt window with that folder as the current working directory (CWD). I seem to have added this to any Windows installation that I've used for any period of time. This is often called "cmd here" or "command here" and Microsoft provide an installable for this function in their PowerToys collection.

In fact, this feature is so useful that it's built-in to OSes from Vista onwards, but to access it you need to hold "shift" while you right-click the item and it only works for folders (some versions of the extension let you right-click a file and have the command prompt open in the folder containing that file). 

Something I find useful from time to time (and which I've never seen elsewhere) is to have file and folder context menus open cmd windows with the folder concerned added to the PATH environment variable, just for that session. This is great for uncommon or temporary use of a folder containing one or more exes without permanently bloating your PATH variable. Some programs with command line interfaces (such as Visual Studio) provide Start Menu shortcuts that open a cmd window with PATH modified for that window only and what I'm suggesting here is similar (but more dynamic).

All that happens when installing the "cmd here" extension is the addition of a few registry entries, and so I made a typical version of this add-on (based on the entries in the Win 7 registry and this web page) and adapted it. Save the following lines as a .reg file and you can add this to your file / folder context menus too:
Windows Registry Editor Version 5.00

    [HKEY_CLASSES_ROOT\*\shell\pathhere]
    @="Cmd with &Path here"
    ;"Extended"=""
     
    [HKEY_CLASSES_ROOT\*\shell\pathhere\command]
    @="cmd /k path %W;%%PATH%% && pushd %%USERPROFILE%%"

    [HKEY_CLASSES_ROOT\Directory\shell\pathhere]
    @="Cmd with &Path here"
    ;"Extended"=""

    [HKEY_CLASSES_ROOT\Directory\shell\pathhere\command]
    @="cmd /k path %L;%%PATH%% && pushd %%USERPROFILE%%"

    [HKEY_CLASSES_ROOT\Directory\Background\shell\pathhere]
    @="Cmd with &Path here"
    ;"Extended"=""

    [HKEY_CLASSES_ROOT\Directory\Background\shell\pathhere\command]
    @="cmd /k path %V;%%PATH%% && pushd %%USERPROFILE%%"

The entries for \Directory\Background enable the same effect by clicking in the empty space in an Explorer window: you just get the current folder the window is displaying.

As an aside, Raymond Chen explains the difference between the \Folder and \Directory classes in this blog post. Note that what the registry (and Raymond) are referring to here as "directories" are called "file folders" in parts of the Windows UI. We have used the "Directory" branch because it makes no sense to have virtual folders as targets for this sort of extension.

You'll have noticed that each entry's root has a commented-out, empty string called, "Extended". If these are un-commented (and you're using Vista onwards), commands will be added only to the extended context menu, available with a Shift-right-click.

You could copy the \Directory\shell entry to the \drive\shell branch if you wanted to provide the same facility for drive roots. You may also want to specify a different CWD, and this is controlled by the appended "pushd" command. If you delete this (everything from the first ampersand onwards, but don't forget to retain the closing quotation mark), the CWD will be the folder in which the context-menu was opened.

One thing I haven't sorted out completely is the variable expansion. When context menu entries for files are activated, %D, %L and %V all hold the filename with its path and %W just holds the path; some other letters hold other cryptic values. The details for directories seem similar, but my tests for \directory\background consistently crashed Explorer. The MS implementation of "cmd here" in Win 7 uses %L for \directory and %V for \directory\background. I can't find any documentation listing all these variables and I'd be interested to know if anyone's come across any.

Usual disclaimers apply (although it's highly unlikely to do anything you don't want) – specifically, I haven't tested this on anything other than one Win 7 Pro 32-bit installation. However, I'd expect it to work pretty much across the board, although older OSes (XP / 2003 and previous) will probably ignore the "Extended" key.

Tuesday, 15 March 2011

The Homeopathic Database

A few friends and I were discussing databases the other day. A colleague of one of us had tried to persuade him that a memory-based DB would be ideal for their project because of the increased commit speed compared to a disk-based system. Data would be eventually written to disk "at some point". My friend pointed out that /dev/null was even faster for writes and only moderately less useful if you need a cast-iron guarantee that all committed data will be available in the future.

If, instead of writing to /dev/null, you write to /dev/zero, it has much the same effect on your data, but reading from /dev/zero produces an infinite stream of zeros. Immediately, we realised this was the answer to every database user's dreams – dilute your data in an infinite sea of zeros: the Homeopathic Database.

Think about it. All those ones interspersed with zeros you started out with may seem important, but the advantages are worth considering. First of all, we know from the countless randomised, double-blind trials done on all homeopathic medicine* that it's a very effective idea. The fact that you only get zeros out at the end is not important because they have absorbed information from all the ones that have been diluted in them. As we know from homeopathic practice, the more zeros we have to dilute the ones in, the more effective the mixture, so the infinite number of zeros in /dev/zero means that what is stored in the database will be really good data.

Secondly, something that all DBAs worry about, backups, are really easy because the data is particularly well suited to compression: although there's an infinite amount of data in /dev/zero, as it's completely predictable, it's infinitely compressible. Backups therefore take no time at all.

The one thing you must remember to do is invert all your data before writing it to the database: the "law of similars" means that retrieved data will have the opposite effect in homeopathic concentrations as it did originally. And you may have to hit your server with a leather cushion while transactions are being committed.

Thanks to Mark, Steve and Alistair.

*They do do that, don't they? I mean surely no one would let people just sell any old rubbish without proper scientific investigation into whether or not it was better than placebo, would they? People who market it are able to make such grand claims for it, it seems certain they have data from repeatable, peer-reviewed trials or they wouldn't hold such strong beliefs.

Tuesday, 1 March 2011

.net Graphics in Windows Forms – Part 2: Anti-Aliasing Your Primitives

I promised this second instalment on Windows Forms graphics would be on rendering settings. Like the previous part of this series (on ControlStyles), this topic will not show any tricks or undocumented features, but I will discuss a few of the framework settings you can alter that change its behaviour and which I don't see discussed very often. The third instalment will discuss text rendering.

If you've never used any of the vector drawing primitives, you might like to try one or two out – just create an empty Forms project, override OnPaint() and you can draw (outlines) or fill (solid colour) several different shapes using methods in the Graphics object. The instance of Graphics you need is passed to OnPaint() in the PaintEventArgs object.

The basic things you need to know are:

  • Coordinates start at the top left of the control's client area and increase to the right (x) and down (y)
  • Angles are measured from the positive X axis (i.e. horizontally, to the right); angles increase clockwise and are given in degrees (many graphics systems and System.Math use radians!)
  • Brushes and Pens (which you'll find you need to fill and draw shapes, respectively) should always be disposed of if you create them. I've never looked into the mechanisms at work here, but the essentials are that they wrap non-managed resources and we're led to believe that the garbage collector can't be relied upon to release these resources before the OS runs out of them.
  • Manual disposal applies to several other objects associated with drawing so make sure you check. The golden rule is that if you make it, you break it. If you were passed it from elsewhere (e.g. the Graphics object in the PaintEventArgs) you should NOT call Dispose() on it yourself
  • You can use ready-made brushes and pens, available in the Brushes and Pens static classes with a myriad of colours to choose from. As these are pre-existing objects don't call Dispose() on them yourself (you can't anyway – you'll get an exception)
  • If you don't need to draw everything, don't! You are given ClipRectangle as part of the PaintEventArgs object and if you can get away without drawing outside this rectangle, that can speed things up

There are a wealth of drawing tutorials available, so I don't want to say anything more about the basics. Instead I want to look at a few of the properties that are available in the Graphics object and discuss what they do. The ones we are interested in are as follows:

  • SmoothingMode
  • PixelOffsetMode
  • InterpolationMode

SmoothingMode tells the rendering engine to use anti-aliasing when drawing lines and shapes. As I'm sure many of you will know, (at least in this context) anti-aliasing reduces jagged edges between areas of different colours by inserting transitional pixels of an intermediate colour where the shape of the (idealised) edge would cover only a proportion of the entire pixel.

The enum which defines the possible values for SmoothingMode has six members. However, one (Invalid) can't be used and of the other five, two are synonyms for "use anti-aliasing" (AntiAlias, HighQuality) and three for "don't use ant-aliasing" (Default, None, HighSpeed). Note that within each of these sets the results are identical: there is no difference at all between rendering done with AntiAlias or with HighQuality. Notice also where "Default" is: by default, anti-aliasing is switched off.

Anti-aliasing is not always the answer. It's slower, can make shapes look blurred and it doesn't really do anything if all your lines are vertical and horizontal, but I would strongly suggest that you at least try switching it on and examining the results if you are drawing anything remotely complex in your controls.

PixelOffsetMode tells the rendering engine how to align pixels on the screen with the coordinate system used to define points on the drawing surface. Like SmoothingMode, several of the enum members are equivalent (Default, HighSpeed and None are the same and HighQuality and Half are also equivalent). "Invalid" is again present but unusable. This setting is also dependent on the use of SmoothingMode.AntiAlias (or one of its synonyms) – PixelOffsetMode makes no difference if anti-aliasing is not being used.

The difference between the two modes is that any integer coordinate is considered to be at the top left of the pixel (None) or the centre of the pixel (Half). PIxelOffsetMode.Half takes longer to process but theoretically offers higher quality rendering because it is easier to follow the idealised course of a line if it goes through the centre of pixels rather than butts up against their edges. However, you may find it makes lines look softer than you want, especially verticals and horizontals.

In my opinion, it makes only a marginal difference to your results and given the extra calculation work it generates, is definitely in the "try it out" rather than "switch it on regardless" category. If anyone knows of a class of problem in which it makes a noticeable (and useful) difference to the results, I'd love to hear about it.

The final setting I want to briefly discuss is InterpolationMode. This property allows you to tell the rendering engine how you want it to treat images if they are scaled or rotated and it's only of use if you are using a bitmap in this way. I'm not going to go into the different modes available: it's enough to know that you don't have to stick with the defaults and a proper treatment of each method can easily be found elsewhere. For an interesting comparison of quality and speed, Bertrand Le Roy benchmarked the different methods: timings and rendered results can be found here.

For completeness, I'm also going to nod towards CompositingMode and CompositingQuality. These properties are worth looking into if you are creating images by adding partially transparent layers to your drawing surface, but are fairly self-explanatory and I'm not going to go into any more detail in this post.

Tuesday, 22 February 2011

.net WaitCursor: how hard can it be to show an hourglass?


I've seen a couple of different ways of using the 'Wait' cursor (aka the 'Hourglass') and several forum posts discuss the problems people have when they haven't been able to work out how to use it properly. Hopefully this is a comprehensive discussion of this small but seemingly complicated topic.

When your program is doing something which stops users from accessing the UI, you should display a 'Wait' cursor. There are three different things you can do to get this (and usually none of them work the way you'd want on their own):
// Method #1
Control.Cursor = Cursors.WaitCursor

// Method #2
Cursor.Current = Cursors.WaitCursor

// Method #3
Control.UseWaitCursor = true;
For the first two, Cursors.Default can be used to return to the expected arrow after the operation has finished; UseWaitCursor should simply be set to false again.

Controls all have a Cursor property and this sets the cursor shape when the mouse pointer is over a control. This property is examined and acted upon only when a Windows message (WM_SETCURSOR) is sent to a window. This means that until the next time this message is sent (perhaps when the pointer is moved away from and back over the control in question), updating this property won't have any effect. To exacerbate the problem, if the UI thread is blocked by whatever operation the WaitCursor would be displayed for, any WM_SETCURSOR messages that are generated won't be processed until the operation has finished.

The other problem with using this alone is that it is a per-control setting: set it for a form and all the child controls on the form will still display the default cursor unless you update all of their Cursor properties as well.

The solution for this problem (and the suggested 'proper' way of displaying the WaitCursor) is to set the Form's UseWaitCursor property. This has the advantage of working for any given control and all its child controls, so set it for a Form and the whole UI for the form will display the WaitCursor when the pointer is over it, regardless of the control under the mouse. There is also an Application.UseWaitCursor which has the same effect across all the windows of a running application. However, this still suffers from the problem of needing a WM_SETCURSOR message before the cursor shape will change.

So what about the other option? Cursor.Current is a static member of the Cursor class and accesses the OS to change the current cursor immediately. This is great… until the next WM_SETCURSOR message is processed and it goes back to whatever the control underneath is supposed to display.

Problems with all these approaches are made less predictable with UI thread blocking too: Cursor.Current will affect a change for some time if UI messages aren't being processed, for example, and then might suddenly change back for no reason that is obvious as a message gets handled.

So the best approach looks like it is to set MyForm.UseWaitCursor to true and then set Cursor.Current. As well as putting any long-running activities in a separate thread, of course. Well, that does solve the problem, unless you want the relatively common ability to cancel a long-running activity and you want the default cursor shape (i.e. an arrow) over your cancel button.

If you look into the Control.UseWaitCursor setter in Reflector, you'll see that it sets its flag (a bit in the private 'state' field in Control) and then recurses into the UseWaitCursor setters in each of its child controls. You might think (i.e. I thought) that all that's then needed is to reverse the setting of this flag in the button in which you want to display a normal arrow and all would be well. Unfortunately, this doesn't work – you still get WaitCursor everywhere. So how can you do it?

Well, it turns out that if you use all of:

MyForm.UseWaitCursor = true;
CancelButton.UseWaitCursor = false;
CancelButton.Cursor = Cursors.Default;

Then you can get the desired behaviour. And of course, a Cursor.Current call would also be in order if you find the cursor shape isn't changing until the mouse is moved.

I don't know how this works: I would have thought that if Control.UseWaitCursor = true sets Control.Cursor to the WaitCursor (which appears to be the case) then setting it to false would have the opposite effect, but I found that CancelButton.Cursor was still set to WaitCursor even after the UseWaitCursor flag in the control (not the form) had been reset.

This solves the problem and is not overly arduous but if you can explain the behaviour, leave a comment and I'll update the article accordingly!

Friday, 18 February 2011

.net Graphics in Windows Forms – Part 1: ControlStyles

One of the areas of .net development that doesn't get the attention I think it deserves is graphics. At the UI level, a lot of software is effectively just a bolting together of existing components that all draw themselves and so this topic is often considered a sideline. Maybe it is beyond the scope of a lot of programs, but occasionally it's very useful to know about the default drawing behaviour and how it can be altered.

Although it isn't hard to draw to a control's surface in Forms applications, the default configuration for the GDI+ system can limit the quality of the result – mainly for reasons of safety (speed, making sure something gets drawn and previously painted stuff gets erased, etc.). This is the first of three posts on graphics in Windows Forms applications. In this post, I'll discuss the ControlStyles flags held internally in objects derived from Control and how they can be set to help controls that draw themselves by overriding OnPaint(). In the next post I'll talk about painting lines and shapes (and in particular about smoothing) and in the last one, the discussion will move on to drawing text directly to the screen. On to the first topic…


System.Windows.Forms.Control has a private enum variable called controlSyle, which is of type System.Windows.Forms.ControlStyles. Amongst other things, the flags contained in this enum can alter the way the control is painted in several respects. In particular, you can use it to:
  • Inform the Window Manager that you want to be in charge of drawing a control if you don't need it to do anything (which removes pointless and potentially time-consuming calls during painting)
  • Use Double-buffering for complicated (i.e. time-consuming) drawing
  • Ask for painting to be redone any time the control is resized
The really good news about these settings (even the relatively complicated double-buffering) is that all you need to know is what the settings are for: any heavy lifting is done for you by the framework and you just have to ask.


As I said, ControlStyles is a flag-style enum (i.e. the possible values are independent of one another and several can be set at once). The variable itself is private, but it's exposed using the getter / setter methods GetStyle() and SetStyle(). However, these are protected, so deriving from an object in the Control hierarchy is the only way to access them. This isn't a problem because deriving a new control from an existing one (or simply from Control itself) is really what you want to do any time you're changing behaviour like painting.


Don't forget when you're dealing with a derived control that the recommended way to access events is not to subscribe to them, but override the OnEvent() methods of the base class (OnPaint(), OnClick(), etc.). Whether you then call the base method is up to you, but as the base method actually raises the event, make sure there are no subscribers if you want to miss this out.
 

The flags that are of interest to this discussion are detailed in the table below:

Flag
Description
UserPaint
Cause the Paint event to be raised when repainting is necessary
AllPaintingInWmPaint
Prevent the control from being erased by the OS before Paint is raised
OptimizedDoubleBuffer
Construct control in buffer before outputting to screen
ResizeRedraw
Repaint the control automatically if resized

UserPaint causes the Paint event to be raised when redraw is necessary. This flag will be set to false on controls the OS draws without input from the program. See the note below about this flag in combination with some of the others, but in general, if you're painting, you'll need this set to true.
 

AllPaintingInWmPaint simply says that the control wants to be responsible for the entire painting operation. What this means in practice is that the PaintBackground event is not raised when a WM_ERASEBKGND message is received from the Windows message pump. Running the (potentially lengthy) implementation of OnPaintBackground in Control is pointless if you are just going to replace the entire area with something else (a picture or some other graphic that fills the control). This flag can greatly reduce flickering on controls where OnPaint() always redraws every pixel of the invalidated region (or can easily be made to).
 

In fact, MSDN suggests that when this flag is true, PaintBackground is raised instead from the WM_PAINT Windows event, and you sometimes see a recommendation to override OnPaintBackground with an empty method specifically so the base method is never called. When I experimented with this, I found OnPaintBackground was never called if AllPaintingInWmPaint was set to true. Your mileage may vary, of course, and I'd be interested to hear any counter-examples. However, I can't see the point of making it possible to not call it from one Windows message just to call it from a subsequent one instead.
 

OptimizedDoubleBuffer means the PaintEventArgs passed to OnPaint() references the Graphics object of an off-screen surface and swaps that onto the display only when painting has been completed. This is another flicker-reducer and works especially well if the painting takes any length of time and / or is made up of layers (which the user can often see being constructed). Obviously, it does introduce an extra step into each Paint event and may not be necessary so should be experimented with rather than simply used all the time.
 

There is another member of the enum, "DoubleBuffer" which you may come across (previous to .net 2.0, it was all that existed). In current versions of the framework, it is hidden from Intellisense so you won't see it in the IDE. Despite this, there is a (presumably erroneous) note on the MSDN ControlStyles entry saying it is preferred over OptimizedDoubleBuffer.
 

MSDN says that UserPaint must be true for either of AllPaintingInWmPaint and OptimizedDoubleBuffer to work and that in order for OptimizedDoubleBuffer to work fully, AllPaintingInWmPaint should also be set to true when double buffering is desired.
 

ResizeRedraw is not really a way of improving display, but does provide a handy method for forcing the Paint event to be raised whenever a control's size has been changed. I've mentioned it here because you would normally set it together with the other flags if it is useful for your control.
 

I've seen a few contradicting examples and explanations in this area and the problems are exacerbated by the current (as of Feb 2011) MSDN literature. As well as contradiction of my findings with the PaintBackground event and the surprising recommendation of DoubleBuffer (i.e. the older flag, currently hidden from Intellisense in the IDE) over OptimizedDoubleBuffer, the ControlStyles page is ambiguous about UserPaint and DoubleBuffer (there is a statement that it should be set for the DoubleBuffer flag to work and another that says it is implied by the DoubleBuffer flag).
 

I thought it would be interesting to see which flags are set by default on some of the familiar controls. Here is the full set of flags, the numeric value of each within the enum and those set by default ('*' in the 'T' column) for a control derived from Panel:

Hex      Binary                            T Name
=========================================================================
00000001 00000000000000000000000000000001: * ContainerControl
00000002 00000000000000000000000000000010: * UserPaint
00000004 00000000000000000000000000000100:   Opaque
00000010 00000000000000000000000000010000:   ResizeRedraw
00000020 00000000000000000000000000100000:   FixedWidth
00000040 00000000000000000000000001000000:   FixedHeight
00000100 00000000000000000000000100000000: * StandardClick
00000200 00000000000000000000001000000000:   Selectable
00000400 00000000000000000000010000000000:   UserMouse
00000800 00000000000000000000100000000000: * SupportsTransparentBackColor
00001000 00000000000000000001000000000000: * StandardDoubleClick
00002000 00000000000000000010000000000000:   AllPaintingInWmPaint
00004000 00000000000000000100000000000000:   CacheText
00008000 00000000000000001000000000000000:   EnableNotifyMessage
00010000 00000000000000010000000000000000:   DoubleBuffer
00020000 00000000000000100000000000000000:   OptimizedDoubleBuffer
00040000 00000000000001000000000000000000: * UseTextForAccessibility
 

I looked at the default flags for several different types of control and found varying results:

                        Name Panel Form Button TextBox Control
==============================================================
            ContainerControl   *     *                        
                   UserPaint   *     *    *               *   
                      Opaque              *                   
                ResizeRedraw              *                   
                  FixedWidth                                  
                 FixedHeight                      *           
               StandardClick   *     *                    *   
                  Selectable         *    *       *       *   
                   UserMouse              *                   
SupportsTransparentBackColor   *          *                   
         StandardDoubleClick   *     *                    *   
        AllPaintingInWmPaint              *       *       *   
                   CacheText              *                   
         EnableNotifyMessage                                  
                DoubleBuffer                                  
       OptimizedDoubleBuffer              *                   
     UseTextForAccessibility   *     *    *               *   

Lastly, it's worth just making a note about the best way of calling SetStyle(). The method declaration is:


protected void SetStyle(ControlStyles flag, bool value)

which at first glance might lead you to suspect that only a single flag can be set at once and you may see discussions of setting these flags which list several calls to this method. However, because of the way the SetStyle method is implemented, you can combine flag values as you normally would (i.e. with the '|' OR operator) and set several at once, providing you need to set each to the same Boolean value:

SetStyle (ControlStyles.UserPaint |
ControlStyles.AllPaintingInWmPaint |
ControlStyles.OptimizedDoubleBuffer,
true);

Clearly, a single call is the more efficient way to change several flags at once.

Thursday, 10 February 2011

Visual Studio Tools for Office Notes

Recently, I've been writing a Visual Studio Tools for Office (or VSTO) application; specifically an addin for Outlook 2007. The relevant versions are Visual C# 2008 and VSTO 3.0. It's been an interesting process and I've discovered a few things about the library that I wanted to make a note of. Specifically, it's worth noting two limitations that anyone needs to be aware of when doing this kind of thing.

Firstly, presumably to aid end users with a more pleasant experience, exceptions are not handled the way you might expect… in most cases they are handled silently by the host application. This is definitely true for the Outlook 2007 plugin I've been building, but from Internet searches it looks like it also happens in at least some (i.e. document-level) Word plugins too. I'm not sure about other types of addin.

Visual Studio creates a class and default method,
ThisAddIn.ThisAddIn_Startup()
and this is the equivalent of Main() for addins as it gets called automatically when the addin is initialised (which usually happens when the host application is loaded). Interestingly, exceptions in the call to this method (and obviously, any nested calls within the addin at this point) operate as you would expect and are reported, but other entry points to your dll through other events do not result in an exception if / when a problem is hit.

Significantly for the developer, this means that whether or not you run the code in debug mode from the VS debugger, you don't get notified of exceptions – instead, the code just silently stops running and control is taken back by the host application (i.e. Outlook) instead of passed to the debugger. Obviously this is a major pain if you're not sure whether something's completed successfully or not, although it's worth pointing out that breakpoints and stepping through the code do still work fine (until you hit an exception, anyway).

The second problem I've had is Outlook specific. I was surprised to find that I cannot easily hook into the Send & Receive functionality. My original idea was to create an account within the standard Outlook account structure and allow users to add it to send / receive groups along with the standard email accounts (it's not actually an email account, but syncs note items). However, it seems that this part of the Outlook 2007 object model is not exposed in the VSTO .net interface: I can't add an account and there is no event when a send & receive takes place, let alone a notification that some specific account should undergo that process.

Now, VSTO is a .net wrapper for a more complex COM API, so it might well be possible using either a pure COM addin or manually accessing the underlying COM interface from a .net language (which I might look into), but it seemed to me to be a surprising omission for a relatively mature API. If anyone can offer any insights (particularly if I've missed anything obvious!), I'd love to know.

Wednesday, 9 February 2011

Computer Clocks and VMWare ESXi

I recently noticed that my home Linux server, which now runs on VMWare ESXi was losing time when it was suspended by the ESXi server. Every time the guest was suspended, the clock just stopped and started again, possibly several days later, still using the time when the virtual machine was halted.

I checked the hardware clock (/sbin/hwclock – I needed to be root to do so on my Ubuntu distribution) and it was correct – like other VMWare products, ESXi mimics the CMOS clock of a real platform and uses the hardware clock to do so (and it can update this regularly using an NTP server) but the response from the date command was still stuck to the time a few days ago when the machine was switched off. Several of the “clocks” in our house show the time from this server, so it needs to be accurate. I waited half an hour to see if the time would sort itself out and so it would cross an hour boundary to see if that would help. It didn’t – all my clocks were still wrong.

I learned several things in my subsequent investigation which may prove useful or interesting to other people:
  • OSes only read the time from the hardware clock when they are booted up (or only every hour or so). After that, they count several times a second and derive the time and date from this counter. The CMOS clock only reports time to the nearest second and it is presumably quite slow to access so the OS sorts out the time after bootup itself. The ticker is often a hardware device (and several different ones can be used, depending on OS and available hardware), but will generally cause an interrupt that needs to be serviced on each tick.
  • Windows (NT derivatives only) updates the ticker once an hour from the system CMOS clock (correcting only if it is more than 60 seconds different). Linux generally does not do this.
  • Linux systems can cause resource-hogging problems depending on the kernel used. Earlier Linux systems used to tick at 100Hz, but newer (2.6 kernels) use a 1000Hz timer by default. In a virtual environment, this imposes a more significant resource drain since the whole VM must be switched into context on the host, rather than just whichever CPU mode is required to service the timer interrupts on a physical system. Kernel directives can force the timer to use 100Hz on a virtualised installation which can often be a good idea, especially if the number of guests running simultaneously is relatively high. Kernels newer than 2.6.28-7.18 are more VM friendly and perform timing operations in a different manner
  • As you might expect, if VMTools is installed and running correctly (which I thought it was, but for some reason I still haven’t been able to replicate, it wasn’t running when my machine came out of suspend) this is all taken care of automatically.

MS Word: Automatic date fields


This is a post detailing the way I managed to get Word to create a date automatically with ordinal suffixes and superscript and is written in something of a tutorial style. If you already know about / aren’t interested in the details of Word fields, skip to the highlighted block about 2/3 of the way down and you’ll find the field codes necessary to achieve the result.

I recently decided to write myself a letter template. Even these days, everyone writes letters from time to time and so it seemed a good idea to have a .dotx file that would generate a suitable blank whenever I wanted to start a new one. I like to have the numbers in a date be followed by “st”, “nd” etc., and for these postscripts to be in superscript. This doesn’t seem difficult; after all, word adds the suffix automatically if you type the date in manually and you can insert a field that will fill the date in automatically. Unfortunately, these two techniques do not work together.

The first problem is getting the right date. Inserting a date using the ribbon (I’m using Word 2007 – I expect earlier versions are similar but I haven’t tried, nor have I experimented with 2010 yet) gives you the current date. That means that the date will never change, or if you tick the box that says “Update Automatically” will change every time the field is updated – not much good if you go back to the letter later for reference purposes and cannot tell when you wrote it because it has today’s date on it.

The second is that the list of formats you can choose from does not include a format which produces the “st”, “nd”, etc. suffixes. A number followed by these letters is said to be in ordinal format in Word-speak, incidentally. Perhaps elsewhere too, but I’m not sure it’s universal nomenclature.

Both these problems can be solved rather easily by amending the field directly. If you don’t know, the contents of Word’s fields are represented internally in a slightly cryptic written language. You can view this by right-clicking a field and selecting “Toggle Field Codes”. Alt-F9 will toggle code viewing for the entire document and is quite useful for fields within fields (nested fields) as they don’t all expand at once if you go down the right-click the path.

If you insert an automatically updating date and view the field code, you will see that it is created like this:
{ DATE \@ "dd MMMM yyyy" }
What’s between the inverted commas will be different unless you’ve picked the same date format as me. You can now directly change the code that generates this field.

The first problem (constantly updated date) is solved by changing the word “DATE” to “CREATEDATE”. This keyword determines what the field code does (and you will see one at the start of most fields if you look at the code). In this case, CREATEDATE generates a date based on the time the document was first created. Be aware that this is determined by the “Save As” dialog being used, so if you save an existing document with another name from within Word, this date will change in the newly named document at that point.

The “\@” merely tells Word that what comes next is a template for the date. Word calls this template a picture and the “\@” is the “picture switch”. In more general terms, a switch is a modifier to a command and is indicated here by the backslash. The picture template for this and other date fields has many components to do with date and time and this is beyond our scope here.

We can also use a switch to use the ordinal format for numbers:
{CREATEDATE \@ "d" \* ordinal}
This switch, however, modifies any numeric field to use the ordinal suffix, not just dates. As such, it does not understand the special needs of a date field and produces rubbish if you include it in a full date (with months and years in it). This is why the example above includes a directive only to show the day of the month.
To show a complete date, two adjacent fields must be employed:
{CREATEDATE \@ "d" \* ordinal}{CREATEDATE \@ " MMMM yyyy"}
Note the space before the “MMMM” inside the quotes – this gives us the space we need between the ordinal suffix and month name.

In order to form this into a single field you can use a “QUOTE” type field to encapsulate the other two. QUOTE fields simply repeat what is in them, but they can include embedded fields and so you end up with a single field that can be moved and formatted as a single object in your text:
{ QUOTE{CREATEDATE \@ "d" \* ORDINAL }{CREATEDATE \@ " MMMM yyyy" } }
So this is almost what we want. What we have now is something like this:
9th February 2010
The last problem is the superscript we want for the ordinal day suffix. Relatively simple? I’m afraid not. The \* ORDINAL switch must apply to a number, which is also printed. There is no way to make it print the suffix on its own, which means that we cannot simply create a separate field for the suffix and superscript it. There is no switch to make the suffix superscript on its own and so we cannot separate the two to allow us to format the digits and the suffix separately.

The only way to do it is to write some relatively complicated code to produce the suffixes (suffices?) based on the number from scratch and then formatting (i.e. superscript) can be applied to these letters within the field and when they are printed they will be as we want. Here is the code (and note the use of superscript within it):
{ QUOTE { CREATEDATE \@ “D” }{ IF { =MOD( { CREATEDATE \@ “d” },20) } = 1 “st” “{ IF { = MOD( { CREATEDATE \@ “d” } ,20) } = 2 “nd” “{ IF { =MOD( { CREATEDATE \@ “d” } ,20) } = 3 “rd” “{ IF { CREATEDATE \@ “d” } = 31 “st” “th” }” }“ }“ }{ CREATEDATE \@ “ MMMM yyyy” } }

Finally, we have the format I was originally after:
9th February 2010
I’ve seen some more complicated versions of this in other places, but this is my take on it. Do note, however, that the suffix creation algorithm is only suitable for dates – it will fail as soon as you count above 31! In a more easily read form it looks like this (and there’s no reason not to put it like this into your documents – it won’t spoil the layout once the field codes are hidden):
{ QUOTE
{ CREATEDATE \@ "D" }
{ IF { =MOD( { CREATEDATE \@ "d" },20) } = 1
"st"
"{ IF { = MOD( { CREATEDATE \@ "d" } ,20) } = 2
"nd"
"{ IF { =MOD( { CREATEDATE \@ "d" } ,20) } = 3
"rd"
"{ IF { CREATEDATE \@ "d" } = 31 "st" "th" }"
}"
}"
}
{ CREATEDATE \@ " MMMM yyyy" }
}
The new things introduced here are IF and MOD, as well as the “=” just before the MOD operator.

MOD is not a type of field, so we cannot put it at the start of a field as a type identifier. Instead it is a function – something we might choose to use in a mathematical expression. So, we need to know how to make a field produce the result of a mathematical expression and it will come as no surprise from the usage above that “=” is the field type identifier that does that: whatever is displayed from an “=” field is the result of a mathematical expression.

Those of you familiar with the MOD function will know that it simply divides one number (in this case the first number in brackets after the word) by another (the second number in the brackets) and keeps only the remainder. You will see that the first number (or argument) to the MOD operation is generated by a nested field (the CREATEDATE field just after the opening bracket. In this case we are using MOD to ensure that 21, 22 and 23 but not 11, 12 and 13 are treated the same as 1, 2 and 3, which is what we want (so we don't need separate tests for 1 and 21). There is also a special test right at the end for 31, which is an exception to the rule we have created (which generalised is that numbers in the 0s, 20s, 40s etc. receive special treatment and 10s, 30s, 50s etc. do not).

The IF field type prints one of two things depending on a comparison:
{ IF a = b "True Text" "False Text" }
Although I have written it as an equality test here, the comparison can be one of the other mathematical operations, such as greater than, less than or equal to, etc. The result is whatever is in the first set of quotes if the comparison is true and whatever is in the second if it is false. And you can nest other fields in these quotes which is how the composite field above is created. In the spaced out version, each new line starts further indented if the field is nested inside the previous one. As each field closes, the indentation is also reduced.

Just as a reminder, you cannot simply type (or copy and paste), unfortunately – each field must be created as a field in word. There is a quick way and that is to write the contents of the field (don’t include the outer curly brackets – these are displayed by word only to tell you that you are looking at field codes and are not part of the field), select the text that will form the field and press Ctrl-F9. You can also copy and paste fields, of course, which makes the plethora of identical CREATEDATE fields easier to deal with. Also, watch your spaces - you may end up quoting some you don't want as you construct the fields.

As a final note, when you are experimenting, don’t forget to update the field if you make changes to the field code or your changes may not be processed (F9 does this). If you are working with more than one field, the contents of all of them might not be updated together.