File systems are irrelevant

I gave a lightning talk at the latest SuperHappyDevHouse designed to answer the question “What file system should I use?” for your typical web 2.0 startup. Answer? Make the file system irrelevant. Slides:

Storage for Startups

If you want the soundtrack to the slides, invite me to give a talk. :)

8 thoughts on “File systems are irrelevant”

  1. thanks for the slides…

    i have a question, maybe it’s really basic or insane.. but web 2.0 isn’t about community creation and modification? wouldn’t that be much more writing to the file system compared to a normal web site?

    thanks again and sorry if i’m asking something too insane..

    P.S.: i friended you, i hope you don’t mind

  2. Web 2.0.. now with pictures.

    Ohhh I like the drawings.. better than most I have to deal with. I will have to see the next time you give a talk in NM to hear the soundtrack. I also wonder what the long-winded version of this would be like :). [Ride of the Valkyries versus Goth dance]

    The lecture got me to thinking more about some issues I had to deal with in a former life.. but its more on the “I have to store 480 gigs a day of data..” however the breaking the files down into smaller parts and ‘hashing’ them would probably have been an improvement over 240 2 gig files in one directory.:).

  3. O(100G) does not mean what you think it does

    O(100G)==O(1), which probably was not what you wanted to say.

    Plus, using hashed names on VFAT, with 10partitions per terabyte just to work around fs limitations seems pretty stupid, and management nightmare.

  4. Re: O(100G) does not mean what you think it does

    Yet, everyone knows what I mean by O(100GB) instantly…

    With regard to organizing data for your startup, it all depends on where you want your points of failure to be.

  5. XFS

    Been using XFS for our latest projects/installs.

    Seems better than all the we’ve tested (JFS,ReiserFS,EXT3) on RAID based SATA systems, and even better than ReiserFS on small+many file handling (which we didn’t expect) without the large CPU overhead Reiser experiences.

    Also creation times for an XFS drive are extremely fast. Not that it matters but..

    Sequential reads/writes and delete performance of XFS is also outstanding.

    Try tweaking your filesystem to works closer with how your hardware (controller) works, that usually does the trick. If you have much memory; try larger buffers. If you have many controllers; separate the logging from the data on different controllers (since transactional systems write both).

    /Charlie Elgholm

Comments are closed.