LCA laptop bag even cooler than previously thought

I have gushed about the 2007 Linux Conf Australia laptop bag before, but I just found out that it’s even cooler than I thought: My Macbook Air fits in it! Photographic proof:

I mean, this is the bag that was so eensy that no one believed I could have a laptop inside it at all. I carried around my Panasonic CF-R5 in this bag for a long time, but that laptop was about half the width of the Air.

Not to diss the LCA 2008 bag. It’s also a great bag that I use on a regular basis, it just doesn’t bend space and time the way the 2007 bag does. Thanks again to Brandon Philips for giving me his extra LCA bag after my first one died.

I am sadly not sure I can attend the 2009 LCA, but I highly encourage people to go if at all possible. I encourage people even more to submit talks for 2010 – speakers get wined and dined, taken on adventures, and even picked up and dropped off at the airport. If you’re going to visit Australia, this is the way to go.

To SSD or not to SSD?

I own a Macbook Air. I am a cool file systems person, so naturally I would buy the version with the SSD rather than the boring old crappy disk, right?


SSD – the kind built out of flash memory – is at present far less reliable than old-fashioned spinning disks. My direct personal experience, and that of my business colleagues and friends, is that flash-based storage suffers silent data corruption at an extraordinary rate. Why is this? A few reasons:

Flash-based storage has a limited number of write cycles before it wears out and stops storing the correct data. Supposedly this is not a problem because (1) current flash can do millions of erase/write cycles, (2) SSDs implement hardware wear-leveling to spread out writes evenly over all the cells.

Let’s start with hardware wear-leveling. Basically, nearly all practical implementations of it suck. You’d imagine that it would spread out writes over all the blocks in the drive, only rewriting any particular block after every other block has been written. But I’ve heard from experts several times that hardware wear-leveling can be as dumb as a ring buffer of 12 blocks; each time you write a block, it pulls something out of the queue and sticks the old block in. If you only write one block over and over, this means that writes will be spread out over a staggering 12 blocks! My direct experience working with corrupted flash with built-in wear-leveling is that corruption was centered around frequently written blocks (with interesting patterns resulting from the interleaving of blocks from different erase blocks). As a file systems person, I know what it takes to do high-quality wear-leveling: it’s called a log-structured file system and they are non-trivial pieces of software. Your average consumer SSD is not going to have sufficient hardware to implement even a half-assed log-structured file system, so clearly it’s going to be a lot stupider than that.

How about those millions of write cycles? Let’s start with a higher-level observation: Disks have been around for decades. We have decades of experience with accelerated aging of disks to predict long-term failure rates – you stick ’em in a hot room and vibrate them around and write lots of data to them, and the failure rate under these conditions can be successfully extrapolated to long-term failure rates. When you buy a disk, you can reasonably expect that it will store your data correctly, and – almost as important – that if it doesn’t store your data correctly, you’ll know because you either get an IO error or because it makes a loud clunking noise and your computer hangs.

When it comes to flash, manufacturers are handicapped when predicting long-term failure rates for a number of reasons. First, it’s hard to extrapolate failure rates under stress tests to long-term failure rates. In particular, the failure mode of a flash cell is that the charge leaks out of the cell – slowly, over time. Stress testing by writing to the cell a lot and then reading the data back is not going to test this situation. In general, we simply don’t have a lot of experience with testing flash and it will take a few years to build it up. Second, manufacturers of the device using flash are constantly switching suppliers for the actual flash memory itself. When it comes to consumer-grade flash, manufacturers have strong pressure to drive the price down and very little pressure in the direction of quality. Frequently, the manufacturer won’t have any idea where the flash chips came from for a particular device based purely on the model number because it used more than one supplier for that model. Third, failure rates will be heavily depending on the pattern of both reads and writes, so a device that checks out fine under one test pattern will failure miserably under another load – not generally a characteristic of disks. Another fascinating aspect of flash-based SSD is that you don’t seem to get any report of checksum failures on corruption – at least I haven’t seen one in the three confirmed cases of flash corruption I’ve seen. I don’t know if this is because the device isn’t reporting it or because the OS driver isn’t listening for it, but it’s what happens.

The exception to these observations are any flash device that costs a lot of money – commercial-grade flash as opposed to consumer-grade flash. Disks vary in quality too, but it’s usually much more along the performance axis than the reliability axis. Speaking of performance, the performance of flash-based SSD has not been the huge leap over spinning disk that we expected. At present time, many disks still have higher bandwidth than many SSDs. SSDs still have a performance penalty for non-sequential IO vs. sequential IO – not as high as a disk seek, but enough to drop throughput by a factor of two or so. They also have high overhead for small random writes due to the need to erase the entire erase block the target block is located in. So SSD will beat the pants off of a disk on an uncached random read workload (e.g., system boot-up), but disks have the advantage on streaming reads and both streaming and random writes, generally speaking.

Another purported advantage of flash is lower power usage than disks. This isn’t a straight-forward equation; it depends on your usage pattern and the sophistication of the device and OS’s power-saving mode. Disks can not only be spun down, the internal electronics can be put into power-saving mode – as can elements of the host-side adapter and link. Don’t automatically assume that your SSD will use less power than a sophisticated disk in power saving mode.

Note that I am explicitly *not* talking about DRAM-based SSD. Those babies are fast, reliable, and very very expensive. If you have one, more power to you.

All of these equations will change as flash-based SSD gets better. Manufacturers will figure out better quality control, hardware wear-leveling will either get better or people will use log-structured file systems at the OS level, performance will improve, prices will drop. But if you are running out and buying storage today, you should buy a disk unless you fit one of the following categories: (a) You have a lot of money and there is some particular feature flash-based SSD gives you that is worth spending that money on, (b) You don’t care much about data integrity, (c) You won’t be doing a lot of writes, (d) You’re using a full-featured log-structured file system with built-in checksums.

Let’s come back to the Macbook Air. Supposedly, one would buy the SSD version because you wanted lower power consumption, better shock-resistance, and higher performance. You wouldn’t get the SSD because it costs a hell of a lot of money (about $1000 more) or because it has lower capacity than the hard drive version (I think it’s 40GB for the SSD and 80GB for the disk at present). The reports are that you really only notice the performance difference at boot. I’ve personally dropped my Air about a dozen times, once hard enough to dent a corner, and so far the disk is fine. Laptop disks in general have become quite reliable and it’s been years since I had one fail, even though I’m what they call a “digital nomad” – my laptop is my primary machine and I travel all the time. The battery life on my Air is stellar – 4 or 5 hours – and almost completely dominated by the display brightness. Dialing it up to max approximately halves the battery life.

Overall, I think people buy the SSD-based Air because it’s cool and new (a perfectly good reason) and because if it costs more, it must be better right? It’s also a status symbol. My personal recommendation: Buy the disk version of the Air. If you did buy the SSD version, back up frequently.

Postscript: Yes, this analysis is based on anecdotal evidence and personal experience, but I can’t afford the time to do real research unless someone pays me to. If you know someone who will, send me email!

Focus follows mouse on Mac OS X: Only $14.95!

Okay, this is it for Mac OS X for me. I want (like any sane person) focus follows mouse. A few googles later, what do I find?

Yes, for only $14.95, I can have a trivial obvious window manager feature! Plus a few other trivial obvious window manager features! Because Apple hates useful software!

No no no no no no. I’ll keep using Mac OS long enough to get a feel for the ways in which it is better than other desktop systems, and then I’ll reinstall the Airbook with Linux.

On a related note, I just installed a very nice piece of free software, Menumeters, which while GPL still had a request for donations on its page – not unheard of, but not normal either. It got me thinking about my experience turning Linux file systems consultant. I’ve had a delightful time discovering how much capital I’ve built up over the years in terms of both knowledge and reputation. This is what you’re building when you work on open source, skills and reputation – not some trivial piece of software you’ll sell for $14.95 a pop. Once you have that capital, you have lots of options, including convincing other people to help you build some nontrivial very valuable piece of software.

Post Script: Yes, you can turn on focus follows mouse for X11 apps, but X11 apps are only a fraction of the windows I use and are poorly integrated with the rest of the system – e.g., when my focus in an X11 window, the screen gradually dims because it thinks I’m not typing. Drove me nuts until I figured it out.

Post Post Script: Argh! And it won’t charge my BlackBerry 8100 via USB out of the box! I thought this stuff was just supposed to work!

Macbook Air part 1: Keyboard and mouse

So I’m running Mac OS X for the first time in about 4 years. I’m spending nearly all my time desperately finding keyboard shortcuts because I absolutely cannot work if I have to use the mouse on a regular basis. This is made harder because I’ve been using my teeny little Panasonic CF-R5, which is designed to match the fingers of Japanese teenage girls. This turns out to be a feature because I don’t have to stretch my hand as far to do things, especially since I use emacs and have to reach for ESC all the time. But the result is that I’m having to relearn how to use all the modifier keys, and some of them are just not in the right place. I have Caps Lock already remapped to Control, at least (thanks, Nathan) and I immediately increased the repeat rate. Oh, and Quicksilver is teh bomb for keyboard-y goodness.

The mousepad is… better than I expected. I turned on tap-to-click, but I can’t seem to triple-click when tapping, only with the button. And apparently my fingers aren’t sweaty enough, because I have to push really hard to get the mousepad to respond sometimes. Otherwise, the scrolling and pinching stuff is pretty cool.

That being said, the keyboard and mouse are quite nice from an objective point of view. Nice response, hard to fat-finger things, really big mousepad.