Operating systems war story: How feminism helped me solve one of file systems’ oldest conundrums

A smiling woman with pink hair wearing a t shirt with the word "O_PONIES" in Courier font
Valerie Aurora in 2009 (keep reading to find out why my shirt says “O_PONIES”)
CC BY NC-SA Robert Kaye
Hi, my name is Valerie Aurora, and I am the inventor of a software feature that has prevented billions of unnecessary writes to hard drives, saving energy and making our computers faster. My invention is called “relative atime,” and this is the story of how my feminist approach to computing helped me invent it – and what you can do to support women in open source software. (If you’re already convinced we need more women in open source, here’s a link to donate now to the Ada Initiative’s 2014 fundraising drive. My operating systems war story will still be here when you’re finished!)

Donate now

First, a little background for those of you who don’t live and breathe UNIX file systems performance. Ingo Molnar once called the access time, or “atime” feature of UNIX file systems “perhaps the most stupid Unix design idea of all times.” That’s harsh but fair. See, every time you read a file on a UNIX operating system – which includes OS X, Linux, and Android[1] – it is supposed to update the file to record the last time it was read, or accessed. This is called the access time or atime. Cool, right? You can imagine why it’s helpful to know when was the last time anything read a particular file – you can tell if you have new mail, for example, or figure out which files you haven’t used in a while and can throw away.

The problem with the atime feature is that updating atime requires writing to the disk. So every read to a file creates a tiny disk write – and writes are expensive and slow. (SSDs don’t get rid of this problem; you still don’t want to do unnecessary writes and most of the world’s data is still on spinning disks.) Here’s what Ingo said about this in 2006: “Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_.

So, atime is terrible idea – why don’t we just turn it off? That’s what many people did, using the “noatime” option that many file systems provide. The problem was that many programs did need to know the atime of a file to work properly. So most Linux distributions shipped with atime on, and it was up to the user to remember to turn it off (if they could). It was a bad situation.

A cartoon of a woman driving a robot penguin
LinuxChix logo
In 2006, I was a Linux file systems developer and also an active member of LinuxChix, a group for women who used Linux. LinuxChix existed in part because it was impossible to have technical discussions about Linux on most mailing lists without people insulting and flaming you for asking the simplest questions – and it was ten times worse for people with feminine usernames. Tell a cautionary story about installing RAM correctly, and the response might be a sneering, “Oh, you didn’t let out the magic smoke, did you?” On LinuxChix, that kind of obnoxiousness wasn’t allowed (though we still got a lot of what is now called mansplaining.)

So when I advised several people in LinuxChix to turn off atime, a friend felt safe telling me that hey, performance on her laptop was better, but now Mutt, the email reader we both used, thought she always had new email. This is because in her configuration, Mutt would look at an email file and compared its atime with the file’s last written time to figure out if any new email had arrived since the last time it read the file.

Now, the typical answer to “Mutt doesn’t work with noatime” was “Switch to a slower directory-based method,” or “Use a file size hack that had bugs,” or any number of other unhelpful things. Mostly, people just wouldn’t bother reporting things that broke with noatime. But I was part of a culture – a feminist culture – in which I respected people like my friend and programmers that attempted to use fully defined, useful features of UNIX in order to implement features efficiently.

I decided to look at the problem from a human point of view. What my friend and the Mutt programmers really wanted to know was this: Has this file been written since the last time I read it? They didn’t particularly care about the exact time of the last read, they just wanted to know if it had been read before or after the last write. I had an idea: What if we only updated a file’s atime if it would change the answer to the question, “Has this file been read since the last time it was written?” I called it “relative atime.”[2]

The amazing thing is: it worked! Matthew Garrett (also a known feminist), Ingo Molnar, and Andrew Morton made some changes to patch, including updating the atime if the current atime was more than 24 hours ago. Other than that, this incredibly simple algorithm worked well enough that in 2009, relative atime became the default in the mainline Linux kernel tree. Now, by default, people’s computers were fast and their programs worked.

I came up with this idea and the original patch in 2006, when the atime problem had been known for many years. Previous solutions had taken a very file-system-centric point of view, mainly along the lines of buffering up atime updates in memory and writing them out when we ran out of memory. What led me to a creative, simple, and extremely fast solution was being part of a feminist community in which people felt comfortable sharing their technical problems, wanted to help each other, and respected each other’s intelligence. Those are all feminist principles, and they make file systems development better.

I try to take that human-centered, feminist approach with other topics in file systems, including the great fsync()/rename() debate of 2009 (a.k.a “O_PONIES”) in which I argued that file systems developers should strive to make life easier for developers and users, not harder. As recently as 2013, a leading file systems developer was still arguing that file systems didn’t have to save file data reliably by mocking users for playing computer games.

I was working on another human-centered file system feature, union mounts, when I heard that a friend of mine had been groped at an open source conference for the third time in one year. While I loved my file systems work, I felt like stopping sexual harassment and assault of women in open source was more urgent, and that I was uniquely qualified to work on it. (I myself had been groped by another Linux storage developer.) So I quit my job as a Linux kernel developer and co-founded the Ada Initiative, whose mission is supporting women in open technology and culture. Unfortunately, as a result of my work, several more Linux storage developers came out publicly in favor of harassment and assault.

That’s one reason why I’m so excited that Ceph developer Sage Weil challenged the open storage community to raise $8192 for the Ada Initiative by Wednesday, Oct. 8 – and he’ll personally match that amount if we reach the goal! UPDATED: Sage and Mike Perez raised this to $16,384!!! The number of Linux file systems and storage developers who both donated to Sage’s challenge and wanted to be listed publicly as supporters is reminding me that the vast majority of the people I worked with in Linux want women to feel safe and comfortable in their community. I love file systems development, I love writing kernel code, and I miss working with and seeing my Linux friends. And as you can tell by the lack of something like union mounts in the mainline kernel 21 years after the first implementation, Linux file systems and storage does not have enough developers, and can’t afford to keep driving off women developers.

A woman sitting at a table explaining soemthing with her hands
Me teaching the Ally Skills Workshop
The Ada Initiative is capable of changing this situation. In August 2014, I taught the first Ally Skills Workshop at a Linux Foundation-run conference, LinuxCon North America. The Ally Skills Workshop teaches men simple everyday ways to support women in their workplaces in communities, and teaching it is my favorite part of my work. I was happy to see several Linux file systems and storage developers at the workshop. I was still nervous about running into the developers who support harassment and assault, but seeing how excited people were after the Ally Skills Workshop made it all worthwhile.

If you’d like to see more people working on Linux storage and file systems, and especially more women, please join Sage Weil and more than 30 other open storage developers in supporting the Ada Initiative. Donate now:

Donate now

Edited to add 10/6/2014: Sage made his goal, hurray! And here’s my favorite comment from the HN thread about this story, the only one actually flagged into non-existence (plenty of other creepy misogyny elsewhere though):

Screen Shot 2014-10-05 at 10.18.37 PM

Also, I had no idea Lennart Poettering planned to post this detailed description of the abuse, harassment, and death threats he’s suffered as an open source developer.

We’re still raising money for Ada Initiative to fight this kind of harassment, so feel free to donate:

Donate now

[1] Yes, Android is Linux too, I’m just naming the brands that non-operating systems experts would recognize.

[2] “Relative atime” isn’t so bad, but the name of the option that you pass to the kernel, “relatime”, showed my usual infelicity with naming things as it looks like a misspelling of “realtime”.

Here’s my favorite operating systems war story, what’s yours?

Val in her Dr. Pepper days
Val in her Dr Pepper days
When I was working on my operating systems project in university, I stayed up all night for weeks feverishly rebooting my test machine, hoping that THIS time my interrupt handler changes worked. I survived on a diet of raisin bagels, baby carrots, and Dr Pepper, and only left the computer lab to shower and sleep for a few hours before stumbling back in and opening up EMACS again. I loved it.

Julia Evans‘ blog posts about writing an operating system in Rust at Hacker School are making me miss the days when I thought everything about operating systems was magical. Julia is (a) hilarious, (b) totally honest, (c) incredibly enthusiastic about learning systems programming. (See “What does the Linux kernel even do?“, “After 5 days, my OS doesn’t crash when I press a key“, and “After 6 days, I have problems I don’t understand at all.”) I’m sure somewhere on Hacker News there is a thread getting upvoted about how Julia is (a) faking it, (b) a bad programmer, (c) really a man, but here in the real world she’s making me and a lot of other folks nostalgic for our systems programming days.

Yesterday’s post about something mysteriously zeroing out everything about 12K in her binary reminded me of one of my favorite OS debugging stories. Since I’m stuck at home recovering from surgery, I can’t tell anyone it unless I write a blog post about it.

VME crate (CC-BY-SA Sergio.ballestrero at en.wikipedia)
VME crate (CC-BY-SA Sergio.ballestrero at en.wikipedia)
In 2001, I got a job maintaining the Linux kernel for the (now defunct) Gemini subarchitecture of the PowerPC. The Gemini was an “embedded” SMP board in a giant grey metal VME cage with a custom BIOS. Getting the board in and out of the chassis required brute strength, profanity, and a certain amount of blood loss. The thing was a beast – loud and power hungry, intended for military planes and tanks where no one noticed a few extra dozen decibels.

The Gemini subarchitecture had not had a maintainer or even been booted in about 6 months of kernel releases. This did not stop a particularly enthusiastic PowerPC developer from tinkering extensively with the Gemini-specific bootloader code, which was totally untestable without the Gemini hardware. With sinking heart, I compiled the latest kernel, tftp’d it to the VME board, and told the BIOS to boot it.

It booted! Wow! What are the chances? Flushed with success, I made some minor cosmetic change and rebooted with the new kernel. Nothing, no happy printk’s scrolling down the serial console. Okay, somehow my trivial patch broke something. I booted the old binary. Still nothing. I thought for a while, made some random change, and booted again. It worked! Okay, this time I will reboot right away to make sure it is not a fluke. Reboot. Nothing. I guess it was a fluke. A few dozen reboots later, I went to lunch, came back, and tried again. Success! Reboot. Failure. Great, a non-deterministic bug – my favorite.

Eventually I noticed that the longer the machine had been powered down before I tried to boot, the more likely it was to boot correctly. (I turned the VME cage off whenever possible because of the noise from the fans and the hard disks, which were those old SCSI drives that made a high-pitched whining noise that bored straight through your brain.) I used the BIOS to dump the DRAM (memory) on the machine and noticed that each time I dumped the memory, more and more bits were zeroes instead of ones. Of course I knew intellectually that DRAM loses data when you turned the power off (duh) but I never followed it through to the realization that the memory would gradually turn to zeroes as the electrons trickled out of their tiny holding pens.

So I used the BIOS to zero out the section of memory where I loaded the kernel, and it booted – every time! After that, it didn’t take long to figure out that the part of the bootloader code that was supposed to zero out the kernel’s BSS section had been broken by our enthusiastic PowerPC developer. The BSS is the part of the binary that contains variables that are initialized to zero at the beginning of the program. To save space, the BSS is not usually stored as a string of zeroes in the binary image, but initialized to zero after the program is loaded but before it starts running. Obviously, it causes problems when variables that are supposed to be zero are something other than zero. I fixed the BSS zeroing code and went on to the next problem.

This bug is an example of what I love about operating systems work. There’s no step-by-step algorithm to figure out what’s wrong; you can’t just turn on the debugger and step through till you see the bug. You have to understand the computer software and hardware from top to bottom to figure out what’s going wrong and fix it (and sometimes you need to understand quite a bit of electrical engineering and mathematical logic, too).

If you have a favorite operating system debugging story to share, please leave a comment!

Updated to add: Hacker News had a strangely on-topic discussion about this post with lots more great debugging stories. Check it out!

2013: Tipping point for the Linux kernel community?

As the last days of the Ada Initiative fundraising drive come to a close ($103,000 so far!), I’m reflecting on what’s changed for Linux in the last 2 and half years. And it’s pretty awesome.

This is the year I’ve been waiting for in the Linux kernel community: the year that 7 new women are learning kernel development at once, the year that kernel developers directly called out the leader of the Linux kernel for fostering a hostile, shitty culture, the year that we as a community realized that the “graying of Linux” is a serious threat to the future of the kernel.

Back in 2011 when I quit my Linux kernel job at Red Hat and co-founded the Ada Initiative, making the Linux kernel community less hostile and more welcoming to women (and all people) seemed like an impossible task. Many Linux developers, past and present, donated to the Ada Initiative anyway, in an act of what I can only call rampant optimism. I decided to focus on more achievable goals, like never getting groped by another Linux kernel developer at a conference again. (So far, so good!)

So I was thrilled to hear about the 7 Linux kernel internships, and the call for civility, and the recognition of the lack of new kernel developers, but I didn’t think the Ada Initiative was directly involved in any of them. But then I kept learning more about ways that the Ada Initiative played an influential part in these events.

As one example, Marina Zhurakhinskaya, head of the fabulously successful Outreach Program for Women in open source, wrote last week about how AdaCamp brought her together with new mentors for the Outreach Program for Women. She told me that the connections she made and the discussions she had at AdaCamp were key to making this year’s 7 Linux kernel internships for women possible. She also credited the increase in the percentage of women speakers at GUADEC this year (from 7% to 21% in one year!) to skills she learned from reading a post on the Ada Initiative’s blog.

It took 2 years to make a noticable impact on the culture of the kernel community, but the Ada Initiative’s approach is working, thanks to people who believed in us back when we were just a web site and two programmers turned activists. My personal goal is to make the Linux kernel community as functional, productive, and enjoyable as the Django community or the Python community. Just imagine: What would a Linux kernel developer event with 20% women be like? What if Kernel Summit was dominated by polite people who just wanted to work together to make the kernel better? How many top developers who left the kernel community could we convince to come back?

For the first time, I’m starting to believe this idea could come true. You can make that day come faster by donating now. And when you meet the new OPW interns at LinuxCon, smile, say hi, and let them know that you’re on their side.

Donate now

Lifetimes of cryptographic hash functions table updated

I finally got around to updating the Lifetimes of cryptographic hash functions table to be current to 2012. I also licensed it CC-BY-SA. Some time when I’m REALLY bored I’ll include all the references inline, but Wikipedia now has pretty complete coverage of each hash function, including original sources. Here’s the current version.

The Linux community can’t remain silent while leaders make anti-woman comments

I’ve written about prominent open source software leader Ted Ts’o’s public comments about rape for the Ada Initiative over on their blog. This post is about my personal reactions.

I know, you don’t want to read about some Linux programmer’s opinions on rape. That’s the point, none of us wants to. But when a prominent open source leader publishes their opinions about rape on a public mailing list, we don’t have a choice. Especially when the author is the chair of the world’s most influential Linux conference, leads a huge chunk of Linux file systems development, and generally represents the Linux community publicly.

The short version: prominent Linux leader Ted Ts’o wrote detailed explanations of his opinions on rape on the 2011 linux.conf.au mailing list in February 2011. Among his beliefs: Rape is impossible if two people are drunk enough, and some rapes shouldn’t really count as rape. There’s more, but I won’t subject you to it.

The recent publicity around U.S. politicians making comments about rape should make it clear why comments like this are harmful: They diminish and excuse rape. It’s not really rape if it wasn’t violent, it’s not rape if you were drinking, etc.

But here’s what these comments mean for me, personally:

One of the people who runs the Linux Kernel Summit thinks that a woman can’t be raped if she and her rapist are both really drunk. Let’s work through this logically: I am a woman and a Linux kernel developer. Linux events usually have alcohol (often free). I drink alcohol, as does he. He goes to many Linux conferences. I go to many Linux conferences. Going to a Linux conference means that I have to spend several hours a day with someone who believes rape is impossible if both people are drunk enough, sometimes while we are given free drinks.

Does this sound like a miserable situation to you? It’s the reality of the Linux community for me during the last year and a half.

When I first read Ts’o’s comments, I couldn’t sleep for two nights. I wanted to throw up every time I thought about it. I was furious and frightened at the same time. Every time I think about this, even now, I literally have nightmares. I can’t bear the thought of working with him even over email, much less attending the same conferences. I worry that my career in Linux is over if I can’t work with him.

The open source community usually views itself as a force for social good: We want to make the world a better place, and we specifically oppose discrimination based on gender or race. At the same time, we have one of the most gender-unequal communities in the world: women make up only 2% of the open source community – compare that to 4% of Fortune 500 CEOs and 20% of the computing industry overall.

When leaders in a community make comments like this and they aren’t protested by other leaders in the community, women get the message: Women are not welcome here. If we as a community care about social justice, about giving our daughters the same chance at a career in open source software that we had, we have to stop this kind of behavior.

The first step is to speak up. If you agree that leaders in the Linux community should not make public statements belittling and condoning rape, please write a blog post, write an email, talk to other community members – but don’t be silent.


A final note: A lot of people in Linux like to say that free speech is an absolute – indeed, that’s the argument people used to defend Ts’o’s comments about rape in the first place. I’m betting the reaction to this post won’t be the same. I challenge every person who has spoken up to defend people diminishing sexual assault and harassment on Linux Weekly News and Linux mailing lists to speak up and defend my right to discuss this topic on Kernel Planet.

Has open source given to you? Give back so women have the same chance you did

This is a cross-post of a post I wrote for the Ada Initiative blog about what open source software and Linux in particular has given me.

Like many of you, open source software gave me a good job as a software engineer – for over 10 years now for me. My job as a Linux kernel developer let me do something good for the world – and make a lot of money doing it. My open source career let me help my family, do fun things like see erupting volcanoes in Hawaii, and donate to causes I care about, like women’s rights and protection of free speech. Open source software gave me amazing opportunities and a great salary to go with it.

Valerie Aurora
Valerie Aurora, lucky open source programmer
That’s why I donated so much of my time and money to help other women have the same chance at a career in open source software that I did. I first started donating my time in 2001 by volunteering for LinuxChix. Today, the Ada Initiative‘s work is so important to me that I donated $5,000 in cash and worked for free for 8 months to get it off the ground – more than $80,000 at senior open source software programmer rates.

Those of us who are in open source software today got lucky: We started programming at the right time, had access to computers, and got community support. But lots of other people – especially women – didn’t have these opportunities. That’s why women make up only 2% of the open source community (as opposed to 20% or more in computing overall). The open source community believes in fairness and social justice, and we should all be working to give women a fair chance at the same jobs we love.

Girls at computers
By Lorena Ceron CC BY-SA-3.0
If open source software gave you a career, a house, or a college fund for your children, please consider giving back to help women have the same chance at a career like yours. Is it worth 1% of your annual salary to know that your daughters, nieces, and other young women will have the same chance at an open source career like yours?

Donation progress bar: donate now

A reminder about employer donation matching: Many employers will match employee donations to non-profits like the Ada Initiative, including Google, Microsoft, Red Hat, Apple, and many smaller companies as well. Check with your manager or search your internal company web site to find out if you can double your money.

Google search loves me

I’m not sure why, but my personal web sites come up shockingly often as the first result in Google searches for not terribly obscure search terms. I’ve checked by searching in a browser in incogito mode and it looks like it’s not just that it’s biasing the search results for me specifically. Some search terms my web pages come up as the #1 result on Google for:

LASIK story
Linux kernel consulting
Linux file systems consulting

Which reminds me, am I the only freelance Linux file systems consultant currently taking contracts? I don’t know a single other Linux file systems developer taking contracts right now.

It seems strange since I don’t make any effort to improve my page rank – I just write stuff in the obvious way. Maybe it has something to do with the fact that I still write my web pages in static HTML? Or that many of them are years and years old? I have no idea, really. If I were an SEO consultant, I’d give some pretty strange advice.