Unicode, Windows and C++

Unicode, Windows, C++, it should all be good right? The answer seems to be that, well, actually, yeah, it is. But there are a lot of details. I do my best to walk through some of them below.

Peripheral Bandwidths – PCIe, Infiniband, QPI, SATA, SAS, DMI/ESI & Ethernet

I’m trying to figure out just how much data I can move through a modern computer. To grok that I need to understand the bandwidth capabilities of the various types of peripherals one can attach to a modern computer. I therefore run through below what appear to be the main peripheral types and what kind of bandwidth they can maintain. Wikipedia has a great summary of the bandwidths of various technologies. Note however that the bandwidths listed in that article are raw (except, confusingly, when they are not) bandwidth, not data bandwidth, below I try to find numbers that represent actual data bandwidth.

My computer memory cheat sheet

For various reasons I need to better understand the gory details of how modern hardware actually works, especially NUMA systems. My first stop on my tour of modern hardware is memory, specifically all the gory details of DDR3 memory (I’ll start worrying about DDR4 when I can buy it).

Losing exceptions in C#, there has to be a better way!

A nasty problem I've been tangling with for a while now is that C# likes to eat exceptions. If one is already in an exception context and another exception gets thrown then the first exception, by default, is just lost. I explore below some ways to deal with this and honestly they all suck. Does anyone have a better idea?

State diagrams for Paxos made simple

I was reading through Paxos made simple and I really wished there were state diagrams to help explicate the protocol. So I wrote them up and share them below. Please keep in mind that the diagrams just explore naive Paxos, that is single value, no distinguished proposer or distinguished learner. So this version of Paxos is pretty useless in practice but it completely captures the core mechanisms that make Paxos work (with the exception of how to pick a distinguished learner). Please note that this article is intended to be used by someone going through Paxos made simple. It is an adjunct, not a replacement.

Wrapped or Native Paxos?

So let's say I want to build a nice highly consistent multi-data center store, something like Megastore. Most everyone at this point has something like Bigtable already deployed in their data centers. What they typically don't have is a way to keep different instances of their table stores guaranteed consistent with each other across DCs. Megastore steps in to address this issue. But this begs a fundamental question - what's better, to wrap a Paxos coordinator on top of existing table stores or to build a new Paxos native storage service?

Average, percentiles and measuring service performance

Measuring the performance of services is tricky. There is an almost irresistible desire to measure average performance. But measuring service performance using averages is pretty much guaranteed to provide misleading results. The best way (I know of anyway) to get accurate performance results when measuring service performance is to measure percentiles, not averages. So Do Not use averages or standard deviations, Do use percentiles. See below for the details.

Distributed Storage Reading List

My technical wanderings of late at Microsoft have taken me into the realm of massively distributed storage. Of course, I've been here before but this time I need to bring some other folks along. So I was asked to put together suggested readings to help people come up to speed. I thought the list might be of general interest so I'm posting it here.

What do you think? Is this a good list? A bad one? What would you suggest?

How do I securely wipe my hard drive?

Ever since Gutmann published his original paper [3] in 1996 there has been an assumption amongst security types that to 'securely' delete a hard drive one had to overwrite it many times. While it's not entirely clear that this claim was true when Gutmann made it, nevertheless, changes in magnetic hard drive technology appear to have made the need for multiple overwrites completely unnecessary. As explained in gory detail in [1] there is no economical way known to recover data that has been overwritten just once from a modern magnetic hard drive. So a single pass writing zeros should more than handle things.

As explored in the Wikipedia article on data remanence it is possible for bad sectors to potentially be recovered because the zero pass wouldn't touch them. But keep in mind that no matter how many times one wipes a drive those bad sectors won't be written to. So if bad sectors are an issue then one will probably need to degauss, physically destroy the drive or use whole disk encryption.

Where things get more fun is with solid state drives (SSDs). As explained in [2] there are real problems with securely deleting SSDs. Right now there is really no good way for a normal person (e.g. someone who isn't a storage expert) to really know if they have successfully deleted everything off a SSD. Tricks like filling up the drive with data won't work because the drives have more capacity than they advertise and since flash cells fade the drive may have taken cells (with data) offline. The drives do support entire disk delete commands but as [2] points out, those commands aren't always appropriately implemented. Overwriting sometimes works but sometimes not and using a pattern of zeros is particularly problematic because some SSDs compress contents.

So if one wants to securely dispose of a SSD I suspect the only reasonable approach is software based disk encryption. Yes, some SSDs do implement hardware level encryption but given the lack of easy validation of the logic and updates when there are issues I wouldn't personally trust that approach. Of course this reduces the security of a 'wipe' to someone not being able to crack the password on the key file stored in the SSD. Personally I'd still do two wipes, at least one using some kind of random data, before disposing of a SSD just to be especially paranoid. Perhaps the only reasonable alternative with SSDs is physical destruction.

References

[1] Dave Kleiman Craig Wright. Overwriting Hard Drive Data: The Great Wiping Controversy. 2008. URL http://www.vidarholen.net/~vidar/overwriting_hard_drive_data.pdf.

[2] Laura M. Grupp Michael Wei. Reliably Erasing Data From Flash-Based Solid State Drives. 2011. URL http://www.usenix.org/events/fast11/tech/full_papers/Wei.pdf.

[3] Peter Gutmann. Secure Deletion of Data from Magnetic and Solid-State Memory. 1996. URL http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html.

Sharing sparse disk image bundles across OS X machines

Normally using my Mac is a simple joy. But recently I created a sparse disk image bundle on my main OS X box and wanted to share it with other OS X boxes. This is quite possible but requires some very arcane commands to make work. I explore those commands below.