Stuff Yaron Finds Interesting » Internet Protocols

State diagrams for Paxos made simple

Administrator — Fri, 11 Nov 2011 02:19:51 +0000

I was reading through Paxos made simple and I really wished there were state diagrams to help explicate the protocol. So I wrote them up and share them below. Please keep in mind that the diagrams just explore naive Paxos, that is single value, no distinguished proposer or distinguished learner. So this version of Paxos is pretty useless in practice but it completely captures the core mechanisms that make Paxos work (with the exception of how to pick a distinguished learner). Please note that this article is intended to be used by someone going through Paxos made simple. It is an adjunct, not a replacement.

1 Modeling assumptions

Following the lead of the paper, messaging is modeled as one way unicast with guaranteed reliability and with no repeats. Guaranteed reliability, btw, only means that the message will get there, it says nothing about when.

One way messaging also means that unless the receiver of the message explicitly decides to respond to the message there is no way for the sender to know when the message arrived or what happened as a consequence of receiving it.

Sending a message is executed as sendMessage(address, purpose of message, arguments...). Receiving a message is executed as messageReceived(purpose of message, arguments...).

The state diagrams below are assumed to be single threaded. In cases where an entity is waiting for messages only one message will be processed at a time and any remaining messages that have arrived will be put into an infinitely big queue to be processed one by one.

Finally, any time a value is assigned it is assumed to be assigned in some suitably persistent way (e.g. written to a disk, stored on tape, chiseled into a stone tablet, etc.)

These simplifications do not alter the algorithm, they just make the state diagrams easier to read.

2 Proposer

Figure 1 State diagram for a proposer

We begin by assuming that some client somewhere instantiates a proposer. How this happens is out of scope for the algorithm definition.

Notice that there is no failure logic in the state machine. This is because the default algorithm doesn’t send error messages when requests are rejected. So if a quorum’s worth of the proposal requests aren’t accepted then the proposer will stay in the wait state forever.

Similarly there is no obvious way for the proposer to know if their accept request was accepted by a quorum of acceptors. First, acceptors don’t send errors for rejected requests and second they send acknowledgements of accepted requests to learners, not proposers.

None of these issues alter the algorithm and there are plenty of ways to deal with them in practice so we follow the Paxos made simple paper and ignore them.

2.1 InitializeSystem

The arguments passed to the proposer are:

acceptorList This is the list of all acceptors in the cluster. Note that in multicast based systems this could just be the multicast address but in that case there would need to be a fourth argument to record how many acceptors there are in the cluster.

proposalNumber The proposal number that the proposer is supposed to start out proposing. Note that Paxos requires that different proposers use unique proposal numbers but doesn’t specify how this is to occur. There are a number of ways to achieve this but they aren’t important for understanding the core algorithm. The key thing to understand is that the initial proposalNumber will be unique amongst all other proposals.

proposalValue This is the value that is supposed to be proposed.

proposerAddress This is the address to which messages to the proposer can be sent, this is needed by acceptors to send in responses to successful prepare messages.

The proposer also has the following local variables:

acceptedPrepareCount This records how many acceptors have accepted the proposer’s prepare request

seenAcceptedProposalNumber This is the highest proposal number that was returned in a response to a prepare message seen so far.

2.2 A question of identity

There is an assumption that the proposerAddress is absolutely unique and used only once with a single proposal. Put another way, each time a proposer is instantiated it conceptually created a universally unique address for itself that it will never re-use for any other proposals. That way when it receives a message it knows that the message is for it. This is important because otherwise a proposer could receive a message that got ’stuck’ in the system for an old version of itself that was proposing something else and got re-used for this proposal.

Of course in practice one doesn’t have to get anywhere near this fancy. Just sticking the proposalNumber being responded to in the ProposalAccepted message would a long way to taking care of the potential naming issues. But it would also make the diagram more complex.

It’s interesting to speculate however as to what would happen if two instances of a proposer existed which had the same address. This is entirely plausible, especially in a distributed cloud environment. This isn’t something the algorithm was designed for, there is an assumption that messages fined their way to the ’right’ place, eventually. As long as messages don’t get duplicated (which, for protocols like UDP is also entirely plausible) it’s probably isn’t a big deal but if message can get duplicated then the protocol could entire an ’impossible’ state where two or more acceptors have accepted the same proposalNumber but with different values. That isn’t a legal state in the protocol.

3 Acceptor

Figure 2 State diagram for an acceptor

The guard conditions at the start of AcceptPrepare and AcceptAccept should really be on the links and not in the states. But doing it ’right’ made the picture enormous so I had to push them into the states for readability.

3.1 InitializeSystem

When an acceptor is created it is passed in:

learnersList Specifies the list of entities to notify when a value is accepted.

acceptorAddress The acceptor’s own address, this is used when notifying learners

4 Learner

Figure 3 State diagram for a learner

Other than talking about distinguished learners in the context of scalability the Paxos Made Simple paper doesn’t actually say much about learners. So I’ve taken a very conservative approach and just assumed that learners just record what they learn, no more. There is one state variable, learnedValue. It’s a dictionary where the key is an acceptor’s address and the value is whatever value the acceptor has accepted. Presumably anyone who wants to use what a learner has learned would look at the dictionary and see if any value in the dictionary is quorate.

Wrapped or Native Paxos?

Administrator — Wed, 09 Nov 2011 22:49:59 +0000

So let’s say I want to build a nice highly consistent multi-data center store, something like Megastore. Most everyone at this point has something like Bigtable already deployed in their data centers. What they typically don’t have is a way to keep different instances of their table stores guaranteed consistent with each other across DCs. Megastore steps in to address this issue. But this begs a fundamental question - what’s better, to wrap a Paxos coordinator on top of existing table stores or to build a new Paxos native storage service?

1 The architectures

Figure 1 Two possible Megastore style architectures

In the first architecture, the one on the left, each DC contains a machine that is running Paxos. Think of this as a combination of the replication server and coordinator in the Megastore paper. All reads and writes go through the wrapped Paxos instance. Writes must always be relayed to the local table store instance (which runs on the existing table store infrastructure in the same DC) for persistence. Reads, however, can often be handled directly out of the Wrapped Paxos Instance via its local cache.

In the Native Paxos Instance case there is no existing table store that is being wrapped. Instead each Paxos instance has its own local table store with its own persistent storage (Read: disk) physically on the same machine. There is no call out to a completely stand along table store service.

So really the only difference between the two architectures is that the Wrapped Paxos architecture uses the Paxos boxes as caches with a separate table service and all of its infrastructure handling persistent storage. With the Native Paxos architecture there is no separate table service, no separate naming/replication/locking infrastructure, just a local single instance Table Store.

An example of a wrapped Paxos instance could be a VM running Paxos in Windows Azure that talks to Windows Azure Table Store or a machine in a data center running Paxos talking to a local installation of HBase or Mongo or CouchDB. An example of a Native Paxos instance could be a machine running Paxos who records all data persistently to the local machine using MySQL or single instance (e.g. one box) deployments of HBase, Mongo or CouchDB.

2 Issues to consider in choosing an architecture

2.1 Are local machines recycled with abandon?

If running in a cloud that doesn’t provide good guarantees that machines (and their local disks) won’t be recycled with abandon then one has little choice but to use a wrapped approach that leverages some existing table store provided by the cloud provider.

It’s tempting to argue that the recycling issue isn’t that big a deal. Imagine, for example, that one’s cloud provider has five DCs in a particular region and one has deployed one’s Native Paxos Instances across all 5. In that case the probability of losing all five machines is really low so it’s o.k. if a few get recycled here and there (with total data loss), we can just resynch from the survivors. But to think this way is to ignore the very real possibility of what’s called a “poisoned value.”

A not uncommon bug in replicated systems is that one gets a request with some value that triggers a bug that causes the local machine processing the request to fail hard and possibly get fully recycled by the underlying cloud infrastructure. The problem is that replicated systems like to retry so the message is likely to get repeated to the next instance of the system who then crashes unrecoverably and so on. No amount of replication can protect against this kind of systemic error. So it’s usually considered best practice to have some kind of durable storage, just in case. But if all the crashed machines can potentially be fully recycled with their local disk state lost then there is no real durable storage. So yes, this issue probably matters.

2.2 Lowering cost

The Wrapped Paxos Instance can be made potentially cheaper than the Native Paxos Instance. The reason for this is that with the Native Paxos Instance the capacity of the cluster is limited to the memory and disk space on the smallest machine in the cluster. In the case of the Wrapped Paxos Instance one can represent substantially more data than would fit on a single box’s RAM or disk. And typically, especially for cloud based solutions, the cost of storing data in a cloud provider’s table store is typically much less than running a VM and storing the data on the VM’s disk. So money is saved both in having fewer Paxos clusters but also in storing the persistent data more cheaply. This assumes, of course, that one is willing to take the latency penalty of cache misses, but that is a configurable choice.

Note that in theory the cost savings shouldn’t be there. For example, most table stores will replicate a value three times inside of a data center. So if one has one’s store replicated across three DCs then a Wrapped Paxos approach will end up with three copies in each DC for a total of six copies. But a Native Paxos approach would just have three copies total (one per DC). But in reality one is highly unlikely to be happy with just three copies in a distributed system (for latency reasons, if nothing else, a single reboot on a machine due to a software or OS upgrade means a DC doesn’t have any local representatives). So what’s substantially more likely is that one will have either five Paxos instances (two DCs have two representatives and one DC having one, mostly for quorum reasons). So in practice the replication costs between the two solutions aren’t that different in pure theory and given the practical realities of the lower cost of storing data in a cloud providers table store the Wrapped Paxos approach is likely to be cheaper.

Note however that this all assumes that it is practical to ’under provision’ the Wrapped Paxos Instances. In other words it’s o.k. that they don’t have copies in their caches of all data. It’s o.k. that the write capacity of the system is limited to the capacity of the under provisioned cluster, etc. If latency and throughput requirements prevent under provisioning the Wrapped Paxos Instances then there is likely no real cost savings.

2.3 Increased availability

All things being equal (and when are they ever that?) the Wrapped Paxos approach should be less available than the Native Paxos approach. The reason is that the Wrapped Paxos approach introduces a whole level of extra complexity - a full distributed (within a single DC anyway), fully replicated table store. This is an entire major subsystem whose problems will at best just kill all writes and can easily also kill all reads (if the Wrapped Paxos Instances are under provisioned). Now one of the advantages of a multi-DC approach is that if one DC’s table store is having a bad day at least the other DC’s table stores are hopefully still working (unless one has a poisoned value). But losing a full DC is likely to do some pretty bad things to latency and throughput thus reducing availability.

How critical this factor is really depends upon the maturity and performance of the table store being used. If it’s known to be highly available and highly reliable then in practice this consideration may not necessarily apply.

3 A non-issue - lowering latency

In theory the Native Paxos Instance should be faster than the Wrapped Paxos Instance for writes in particular. After all the Native Paxos Instances only need to write to their local disk drives while the Wrapped Paxos Instances must go through a full write to the table store service which is almost certainly on a separate machine. In the worst case a write to the table store will require a full name resolution on the table store service to figure out what machine currently is the table store master for the desired value and then having to send a write to the table store master who then has to replicate it to its two slaves. But given the latencies involved with doing cross data center writes as part of the Paxos algorithm it isn’t clear if this extra overhead on writes really makes all that much of a difference.

Reads may or may not be faster depending on the cost tradeoff made in deploying the Wrapped Paxos Instances. If low latency is a priority then each Wrapped Paxos Instance can have enough cache to hold all the values it is responsible for overseeing (even if they have to spill over to disk).

So one suspects that in practice latency is not a compelling reason to choose one architecture over the other.

4 Conclusion

In the end the choice is not eternal, one should be able to switch from one architecture to the other. So one could start with a wrapped approach since the table store infrastructure might already be available and then switch to native if that should prove to have useful advantages. My general guess is that anyone who can’t under provision will probably want to run native mostly because it removes a whole layer (the table store service) of things to go wrong.

Average, percentiles and measuring service performance

Administrator — Thu, 27 Oct 2011 21:34:40 +0000

Measuring the performance of services is tricky. There is an almost irresistible desire to measure average performance. But measuring service performance using averages is pretty much guaranteed to provide misleading results. The best way (I know of anyway) to get accurate performance results when measuring service performance is to measure percentiles, not averages. So Do Not use averages or standard deviations, Do use percentiles. See below for the details.

1 Averages for service performance are typically wrong

So let’s say you have your shiny new service and you want to know how its performing. You likely set up some kind of test bench and start firing off a bunch of requests to the service and record the latency and throughput for the requests over some period of time. But how should one present a summary of the performance data?

In most cases what I see people do next is calculate average latency and average throughput. If they are particularly fancy they might even throw in standard deviations for both.

Unfortunately in most cases both the average and the standard deviation don’t accurately represent the performance of the system.

The reason for this is pretty straight forward - the average and standard deviation attempt to describe the characteristics of a set of data assuming they describe a normal curve. This is the famous bell shaped curve. But service performance is almost never normal. In fact performance distribution tends to be pretty flat for most requests and then fall off a cliff. this is not what you would call a normal distribution. There are lots of reasons for this.

For example, most services have some kind of caching and typically caching is a pretty good technique but cache misses are expensive. So while most requests will be serviced quickly out of a cache some number of requests will cause a cache miss and will be substantially more expensive to handle. This behavior isn’t really a curve, it’s more like a step function.

Other reasons are queuing behavior. Most services can be thought of as a series of queues and so long as the load is within the queue’s capacity then everything is fine but once that capacity is exceeded then time outs, failures, etc. will start to happen since the system can’t recover until incoming requests fall enough to let the system catch up. So pretty much the system just shuts down until the queues are cleared.

Now there are ways to torture normal distributions to get behavior that is closer to what one sees in service behavior. We can start to talk about Kurtosis, skew, etc. But these are all attempts to force the data to be summarized by some model and if that model isn’t appropriate to the data set then the information the model is giving is just plain wrong. So if one is going to start playing around with different types of distributions then this still means one has to collect the same kind of data that percentiles (discussed below) require in order to prove that the distribution is accurate. Well if one is going to do all the work to collect percentile data then why not just use the percentile data?

But the punch line is this - characterizing a service’s performance across many requests using averages will almost certainly produce misleading data. So please, just say no to averages.

2 Even small numbers of customers having a bad experience costs real money

O.k. o.k. so averages are wrong. But they are really easy to calculate and as long as they ’close enough’ aren’t we happy? This is actually something that has been studied and the answer is - no. To help frame this discussion consider the following:

Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO– 2007 - 100 ms delays caused a 1% drop in sales at Amazon.

Performance Related Changes and their User Impact– 2009 – ½ second delays saw a loss of 1.2% of revenue/user. This went to 2.8% at 1 second and 4.3% at 2 seconds.

Why Web Performance Matters: Is Your Site Driving Customers Away?– 2010 – Between 2 – 4 seconds 8% of users abandon the site, from 2 -6 seconds it’s 25% and by 10 seconds it’s 38%.

With this data in mind let’s go back to look at those averages. If the average is say a 50 ms delay shouldn’t everyone be happy? Well not if say 20% of users are seeing 100 ms plus latencies. This wouldn’t show in the average and the standard deviation is largely meaningless anyway since it’s describing probability for the wrong curve.

In that case the 20% of users seeing 100 ms plus latencies, using the Amazon number, 20% * 1% = 0.2% of sales just walked out the door. That isn’t a healthy way to run a business. In fact major service companies measure the experience of their users up to 99.9% (as will be explored below) because bad experiences for even small numbers of users have significant financial consequences.

Put another way, it’s cheaper to create systems that have predicable performance into the 3 9s than to lose sales caused by bad performance at the end of the performance curve.

3 Percentiles

So typically the way we will accurately represent system performance is using percentiles. The idea behind percentiles is pretty straight forward - what percentages of users had a particular experience?

Imagine, for example, that we ran a test 10 times and the latencies we got back were 1, 2, 1, 4, 50, 30, 1, 3, 2 & 1 ms. The first thing we would do is order the latencies from smallest to largest - 1, 1, 1, 1, 2, 2, 3, 4, 30, 50.

The median or 50th percentile is the best latency that 50% of the requests experienced. In this case we would count 1/2 the results or 10/2 = 5 results and that is the median. In this case it’s 2 ms. So this means that 50% of the requests had a latency of 2ms or better.

The 90th percentile is the best latency seen by 90% of the requests. In this case that’s the 9th result (0.9 * 10 = 9) which is 30 ms.

Typically the results will show a graph from 1 percentile through 90% in increments of 1% followed by 99.9% or higher if appropriate. In most cases the results have to be shown on a logarithmic scale to be easily viewable.

Throughput is measured in a similar way. The big difference is that for throughput one is measuring the number of requests completed over some window of time. 1 second is a pretty typical window. I usually just measure how many requests completed during a particular window.

Distributed Storage Reading List

Administrator — Tue, 25 Oct 2011 23:26:28 +0000

My technical wanderings of late at Microsoft have taken me into the realm of massively distributed storage. Of course, I've been here before but this time I need to bring some other folks along. So I was asked to put together suggested readings to help people come up to speed. I thought the list might be of general interest so I'm posting it here.

What do you think? Is this a good list? A bad one? What would you suggest?

Required Reading

Google's Stack - GFS, Big Table & Megastore

An alternate way to building large scale stores without consistency, worth having in mind - Dynamo

Working in highly distributed storage systems almost certainly means dealing with Paxos (or one of its variants) so - Paxos made simple (make sure to also check out my companion article as well) & an outstanding Google paper on Paxos called Paxos made live.

Extra Credit

I suspect that anyone who ends up working in this area for any period of time will end up reading the following list of articles but they aren't necessary from day one:

Wikipedia has a great summary of the various variants of Paxos that are running around.

Chubby is Google's abstraction layer to make Paxos easier for developers to handle.

A great summary of ZooKeeper, this is Hadoop's answer to Chubby.

A summary of ZAB, the 'Paxos Like' mechanism that is underneath ZooKeeper.

OAuth 2.0 Bearer tokens – unsafe at any speed?

Administrator — Sat, 02 Oct 2010 01:01:42 +0000

Eran’s latest article raises a number of specific security threats by way of arguing that bearer tokens are irredeemably insecure. In this article I examine the attacks Eran calls out and demonstrate that they are already addressed by OAuth 2.0. Eran’s article does bring up the interesting question of - do we need defense in depth for the tamper resistance and confidentiality provided by SSL/TLS?

1 The security threats Eran raises

Below I list the specific security threats I found Eran raising in his article:

Developers turning off error checking in SSL Some sites do not have up to date certificates and a few try to issue certificates that aren’t signed by a certifying authority trusted by the operating system. To deal with these errors client authors sometimes just turn off SSL cert error checking. Done in a blanket way this invalidates SSL security since any man in the middle can insert itself, send a bad cert and the client will accept it.

Bearer tokens can be replayed Eran claims that if SSL is screwed up (such as above) then an attacker can take the bearer token and use it to make any request they want.

Typos Eran claims that if the client developer puts in a typo (say http://foo.com instead of https://foo.com or https://boo.com instead of https://foo.com) then SSL’s security guarantees are compromised.

Client developers don’t do security Eran believes that client developers can’t get security right and therefore must have libraries to protect them from themselves.

The first and last threats obviously contradict each other. Eran’s attack on SSL is about people using the libraries wrong and then he suggests that libraries can fix security? The reality is this - developers who do not pay attention to the security consideration sections and understand the security threats they are under will get it wrong. So either we educate developers or security will fail.

That bearer tokens can be replayed is absolutely true but equally true is that properly designed bearer tokens significantly reduce the damage done. First, they are short lived. Second, they have an audience in them which in most interesting cases (as I discussed previously) kills replay attacks before they can even get started.

The typo attack sounds the most scary. If a developer mistypes a single character the entire security of the system might be forfeit. But is this threat realistic? In OAuth 2.0 clients are required to go through a two step. In the first step they present their credentials to a token endpoint who issues an access token. In the second step they present the access token to an application endpoint to actually do something.

Let’s say the developer put in the wrong token endpoint but the right application endpoint. If so then the access token the wrong token endpoint produces won’t work on the right application endpoint and the client will fail.

Let’s say the developer put in the right token endpoint but the wrong application endpoint. A properly designed OAuth token endpoint request includes the URL of the application endpoint the token is going to be used for. This allows the token endpoint to validate that this is a supported application endpoint. Typically the way systems I’m involved with handle this situation is by putting the base URL for requests into the scope field. But, so long as the token endpoint checks the application endpoint URL and sees that it isn’t a supported endpoint then no access token will be issued and no damage is done.

2 Security is insurance, please don’t buy more than you need

OAuth 2.0 depends on SSL/TLS to provide two key features - message tampering protection and confidentiality. If SSL/TLS is broken then one or both of those features will be lost. As explored in the appendix below mechanisms like OAuth 1.0’s signature protocol don’t really provide much defense in depth against SSL/TLS failures. So if we are going to get additional protection it’s really only going to be by essentially re-inventing SSL/TLS like capabilities somewhere else in the stack. This would most likely occur by introducing a generic mechanism to sign/encrypt HTTP messages.

But inventing such a mechanism is a non-trivial endeavor. Just look at the complexity of SSL/TLS itself to get some idea of how hard getting a HTTP level message signing/encrypting mechanism right will be. So if we are to invent such a mechanism we need one heck of a good use case. I haven’t seen such a use case but if someone has one I’d love to see it because I have some ideas on how to implement HTTP message signing/encrypting.

A Appendix - A quick look at what OAuth 1.0 signatures buy in terms of security

Let’s say that a developer has, as Eran realistically describes in his article, turned off SSL cert error checking. Let’s further assume that the developer is using OAuth 1.0 signatures. Finally let’s assume that a man-in-the-middle (MITM) attack is underway. As soon as the client tries to connect to the server the MITM will redirect the request to their own machine and present a bad cert which will be accepted because cert checking is off. Now let’s examine what the attacker can do even though OAuth 1.0 signatures are being used.

OAuth 1.0’s security considerations section already points out two things the attacker can do. In section 4.2 it points out that the attacker can silently pass on requests and responses thus allowing them to eavesdrop. But even more fun is section 4.3 which points out that since responses aren’t signed the attacker can change the response to be anything they want opening up a Pandora’s box of security threats. Trying to check on the status of your web service via it’s OAuth protected management interface? The attacker can make it look like everything is fine with your service even as the attacker is taking it down. Looking for the storage location to upload your secret document? The attacker can re-write the response to your query for the directory location to point at a URL they control.

Or check out the warning in section 3.4.1, the request body is only protected by the signature if it’s a HTML form. In other words if the request protocol is JSON, XML, etc. then the attacker can not only change the response, they can change the request too without any fear of detection.

In addition, as a practical matter, an attacker can repeat the same request message multiple times if it wants to. OAuth 1.0 tries to prevent replays by using nonces. These are unique values generated by the client that the server is supposed to record for each and every request received from each and every client (at least until the time stamp in the message has passed). The idea is that before processing a request the server will check the nonce in the request and see if it has been seen before and if so will reject the request.

In reality distributed systems will do no such thing. This is because keeping a database of nonces for every single request received from all clients is so expensive to implement and so hard to keep consistent (we run right into the CAP theorem) that in practice scalable systems just won’t do it. Instead what they will do is check the time stamp and that is it. So as a practical matter attackers can in fact replay requests.

Why does OAuth need request tokens?

Administrator — Tue, 26 May 2009 00:00:00 +0000

OAuth's current access dance is based getting a request token that is later exchanged for an access token. Introducing the request token takes what could have been a 4 round trip protocol and makes it into a 6 round trip protocol. Couldn't we just simplify OAuth down to 4 round trips by getting rid of the request token all together? Or is there some critical use case enabled by request tokens that makes all the complexity worth the price?

[5/26/2009 - Updated with Q&A on open redirectors]

[6/2/2009 - Updated with a note from Allen Tom on another way to prevent open redirector attacks]

To set the stage for the conversation it's useful to first review the newly proposed three legged OAuth message exchange pattern. The picture below is based on OAuth Core 1.0 Rev A (Draft 3):

There are several things that concern me about the current OAuth pattern:

State – This pattern forces both the consumer and service provider to store someone else's state for a potentially non-trivial period of time. That is always a messy proposition at best. The service provider is forced to associate a call back with a request token and the consumer is forced to associate a verifier with a request token.

Performance – The service provider better implement messages 2/3 really quickly because the user is going to have to swallow all the latency of messages 1-4 before anything useful happens.

Complexity – Having 2 extra round trips means extra code, extra testing and extra chances for things to go wrong.

What if we instead use a pattern like:

Now the only long term secret is the access token and only the consumer has to record it. The request/response pairs go down to four and the performance issue with requesting the request token is gone.

I suspect there is a good reason that OAuth felt it had to have the request token but I honestly am not clear what that reason is and if its important enough to justify the complications it introduces. Hopefully the article will set me on the path to finding out.

Q&A

Isn't the newly proposed pattern susceptible to phishing?

Absolutely. The short answer is that the consumer will need to associate a value with the callback URL that lets the consumer make sure that the user coming back on the call back URL with an authorization token is the same user the consumer sent out. This can be trivially handled via random numbers or encrypted ID. See below for details.

The attack is that a black hat user gets to message 2 (the redirect) in the previous chart and then doesn't actually execute the redirect but instead takes the call back URL and publishes it on a web page. The black hat convinces a white hat user to click on the link in a context where the white hat user expects to grant permission to the service provider. The white hat user grants permission and is redirected back to the call back URL specified by the consumer originally for the black hat user with the access token.

In theory the attack would be if the consumer somehow got confused and thought that the access token returned by the white hat user belonged to the black hat instead because the callback URL that was used was the one that the consumer had originally issued to the black hat.

It might seem trivial to prevent this attack – don't put any user identifying information into the callback. This means that when the callback arrives (e.g. message 7 in the picture above) the consumer will validate who the user is by whatever login method (if any) is being used and not via any values in the callback URL. This prevents the phishing attack from working.

Unfortunately this approach also makes an even more insidious attack possible. Let's assume the consumer doesn't put any identifying information into its callback URL. Let's further assume that the black hat goes through the entire permissioning process but swallows message 6, this is the redirect with the access token.

Let's further assume that the black hat fools the white hat into clicking on a link that sends the black hat's access token to the service provider. Assuming the white hat is logged in the black hat will think the access token belongs to the white hat user instead of to the black hat (remember, the redirect URL has no user identifying information so there is no way for the consumer to know who the token was issued for). Now when the consumer wants to interact with the service provider on behalf of the white hat they will actually use the black hat's access token.

To see how bad this could be imagine that the consumer is a service like Plaxo that handles address book synchronization and the service provider is some service with an address book like Live. In that case when the consumer wants to synch the white hat's address book with Live they will actually push all the address book information into the account the black hat set up and got the access token for. So much for the white hat user's private address book information.

To prevent this the consumer has to be able to associate an access token from the service provider with the user that the consumer had sent to get the access token in the first place.

There are a couple of ways the attack can be prevented. Probably the easiest is for the consumer to generate a cryptographically secure random number (can you say – verifier?) and stick it as a cookie on the user's browser and then stick that number into the callback URL. When the callback arrives the consumer compares the ID in the cookie with the ID in the callback and if they don't match then the access token is rejected.

If cookies aren't available (and that situation is getting increasingly less interesting) then the consumer can take the user's ID (this only works if the user has authenticated to the consumer) and encrypt it and put it on the callback URL. Then when the call back comes in the consumer checks the encrypted ID with the currently user ID and if they don't match rejects the request.

It is also possible to make things work in the case that the user is anonymous to the consumer and doesn't support cookies but that will require the user to manually copy and paste the authentication token back to the consumer.

Can't someone falsely pretend to be a consumer?

Sort of, but it won't help them.

The attack is that a black hat sets up a phishing site called blackhat.contoso.com that looks like whitehat.example.com. blackhat.contoso.com then fools a user into thinking its whitehat.example.com and redirects the user over to the service provider. What happens next depends on the relationship between the service provider and its consumers.

Let's take the most open case, the service provider allows a consumer to ask for user permission without requiring pre-registration. I actually was the PM on a project that shipped a system like this for Live and it worked fine. In that case the service provider will need to prominently display the consumer's DNS name from the callback to the user. So the user will see 'blackhat.contoso.com' and they thought they were coming from 'whitehat.example.com' and so the user will know to reject the request.

But let's say the blackhat gets fancy and instead makes the DNS in the redirect URL point to whitehat.example.com? In that case the user will do what they intended to do, give permission to whitehat, and the access token will be sent to whitehat's domain as part of the redirect. So blackhat got nothing for their efforts (other than maybe a thank you from whitehat).

Isn't this design susceptible to open redirector attacks?

The short answer is – yes.

The attack goes like this. Evil.com phishes the user into thinking that evil.com is actually stupid.com. Evil.com, in message 3 in the second picture above, then redirects the user to the service provider with a callback URL of the form: http://stupid.com/redirector?redirect=http://evil.com/dobadstuff.

As I pointed out in the previous question this attack is generally useless because the response will go to stupid.com, not evil.com. Unless, of course, stupid.com is criminally stupid. In which case the previous URL points to what's known as an open redirector. This is a web page on stupid.com's domain that accepts a URL as a query argument and will blindly redirect the user to that URL regardless of where it is.

There are at least two ways to secure the protocol as described above against open redirector attacks.

Require SSL callback URLs – Generally speaking open redirectors don't run over SSL so if the SP requires SSL callback URLs then the SP will have protected its users against most (but certainly not all) incompetent websites.

Require an out of band secret – The core of the attack is that evil.com, thanks to the open redirector, gets a copy of the authorization token which is what is known as a 'bearer token'. That is, mere possession of the token is proof that one has the right to use the token. So another fix is to require that the authorization token cannot be used on its own. Instead the SP and the consumer have to provision a shared secret out of band. In the simplest possible case (for the consumer) message 9 in the second picture requires that the shared secret be included on the request. Because evil.com won't have the secret that was provisioned for stupid.com any attempt to use the authentication token will fail. An alternative approach (this is preferable from a security perspective) is to sign the redirect in message 3 with the shared secret.

Perform Discovery – (Thanks to Allen Tom for pointing this one out) The SP can require that consumers provide a way to discover what addresses a consumer uses for callbacks. For example, the consumer, in message 3, could pass in a URL that points to the same domain as the callback URL. The SP would then do a GET on the passed in URL and get back a file that lists the callback locations the consumer supports. Obviously the SP wouldn't support redirects on that GET. Equally presumably the consumer wouldn't list in that location evil.com's address. So in theory that's that.

This all having been said I do not believe we should do anything about open redirectors. There are good reasons to want to run redirectors but it's been known for many years now that without canaries or some other form of protection these redirectors are very insecure. Running an open redirector without cross site attack protections is outrageously irresponsible. And I just don't believe we should design protocols against outrageously irresponsible behavior because no protocol can be protected against all the idiotic things an irresponsible person can do.

And, for what it's worth, Live's first delegated authentication system worked exactly as this blog post describes. You can see an overview here. And yes, we were fully aware of the open redirector problem and no, we didn't try to protect against it for exactly the reasons described in this Q&A.

Claims, Tickets and HTTP – Security protocols for services

Administrator — Fri, 15 May 2009 00:00:00 +0000

I'm writing an enterprise service. A request comes in. Do I honor the request or reject it? Answering this apparently trivial access control question has spawned whole universes of interlocking protocols. Kerberos, Shibboleth, SAML, WS-*, Liberty, OAuth, OpenID and so on. Before I can pick which protocol to use I need to define my requirements.

DISCLAIMER: Although I am an architect on .NET Services' Access Control Service nothing said in this document necessarily represents the opinions of my employer, my friends, my enemies or my teddy bears. No warranty express or implied. Your mileage may vary. Do not remove tag.

Claims – I like claims based programming for handling access control. In a claims based system a requester shows up with a request and attached to that request is a set of claims about the requester. The specific type of claim really isn't that important but typical examples include claims that the requester is person X, or is in role Y or is a member of group Z. The service then decides what to allow the requester to do based on those claims.

Validating that the claims submitted to a service are true isn't exactly trivial. Whole protocol suites exist to validate claims about identity, for example. And once identity is validated, figuring out what roles or groups the caller is in typically requires centralized logic. For example, my enterprise service probably doesn't want to figure out if someone is an employee of my company or not. I'd rather leave that determination to someone else and just get informed of the answer. So what I need is a way to let someone else validate a requester's claims and then just have my service informed of the result.

Tickets – So what I would really like is for the requester to present me with a ticket (in the Kerberos sense) that was signed by someone I trust attesting to what claims I should believe about the requester. It's up to the ticket issuer (called a Key Distribution Center in Kerberos, a STS in WS-*, an Identity Provider in SAML and so on) to figure out who the requester is and what claims I should believe about them. The ticket issuer summarizes what it knows about the requester in a signed (and/or encrypted) ticket that it gives the requester who then hands the ticket to my service. My service then validates the ticket (using shared trust with the ticket issuer, not the requester), pulls out the claims and makes its access control decision.

HTTP – The services I'm working with are native to HTTP. I say native to distinguish from "dig a tunnel through HTTP". Lots of protocols out there claim to be based on HTTP but in fact use HTTP as nothing more than a really inefficient implementation of TCP, completely ignoring the HTTP resource model, headers, etc. So I want a protocol that gives me claims and tickets that are native to HTTP. That means I want to see resources and authorization headers.

Less is more- The enemy of interoperability is optionality. The more options a protocol has the lower the probability that two parties using that protocol will successfully interoperate. As we used to say in the protocol world "the spec is done when there is nothing left to cut". So I want the protocol to be embarrassingly simple. If a protocol looks like a Chinese Menu then it isn't for me.

Reach – The services I need to work with are written in every language for every platform so I can't assume that my partners will just be able to grab some library to implement the security protocol. Sure, some components, such as a HTTP stack or hashing/encryption libraries are ubiquitous. But beyond that my partners, for the foreseeable future, will need to write things themselves. So I'm looking for a protocol so trivial that someone could whip together a basic implementation in a few hours.

When I think about the protocol I would like to have I focus on a few things. The claims format should be so trivial that it uses name/value pairs not structured formats like XML/JSON/YAML. I want tickets so small that in practice all the claims and the ticket overhead can trivially fit into a HTTP header. And I want the protocol/encryption process to be so simple that all the claims handling/encryption code can be written in 100 or 200 lines of readable code. I also want a pony.

So what do you think? Is there a protocol that meets my needs today? Is there a protocol that is close and I should just adapt it? Feel free to respond in comments or, if you are going to be at the Internet Identity Workshop 2009A in Mountain View May 18-20th we can talk there!

Two Phase Commit Need Have Nothing To Do With ACID or Any Other Kind of Transaction!

Administrator — Mon, 03 Oct 2005 00:00:00 +0000

O.k., it's pet peeve time. Otherwise intelligent people seem to get really confused whenever they hear "two phase commit". They immediately assume that if one is using two phase commits then one must be implementing an ACID or at least Atomic transaction. Nonsense! A two phase commit is nothing more than a coordination mechanism. It allows one to guarantee that a group of independent systems will either all perform or all not perform a particular action. Historically the 'action' was usually some kind of ACID or at least Atomic transaction. But there is nothing in two phase commits that requires or even implies any particular action, atomic, ACID or otherwise.

In fact, two phase commits are incredibly useful for the ultimate in non-ACID transactions – compensating transactions. In an ACID transaction if the attempted transaction fails then everything gets nicely cleaned up, no muss, no fuss. In a compensating transaction, on the other hand, if the action fails then an attempt is made to undo the action but with no ACID semantics. For example, if I book a flight with a travel agency and it turns out they couldn't book the flight they said they would then instead of just canceling the whole ticket (and losing their booking fee) they might rebook me on another flight that I may or may not prefer. The point being, that the 'failed' transaction (booking the agreed flight) is 'compensated' (by buying another flight) rather than rolled back in an ACID sense (by canceling the ticket and refunding the fees).

Now imagine that I want to book a whole trip myself involving a hotel, a rental car, a plane flight, etc. I could use a two phase commit protocol to allow me to first send reservation requests to each of the companies and then if I can't get reservation confirmations from everyone the two phase commit coordinator would automatically trigger a compensate command to each of the companies. Of course the compensation may not be what I like, i.e. the airline may charge me a re-booking fee, but them's the breaks in the real world.

The point is that two phase commit protocols are nothing more or less than a coordination mechanism and what they are coordinating may or may not be an ACID or Atomic transaction.

Rights Not Exercised are Lost – Tibco, BPEL & Rendezvous

Administrator — Wed, 04 May 2005 00:00:00 +0000

Our rights only exist to the extent that we defend them in our daily lives. The right to free speech, for example, would quickly drain away if we didn't frequently exercise it and by so doing kept the knowledge of the rights importance and the mechanisms to protect it alive and well. But too often people believe that the need to defend our rights doesn't apply to them because they are just a single individual. Who cares what a single person does? But when each individual believes their actions don't matter then they significantly reduce the work required by those who would take our rights away.

An example of this situation came up in the BPEL TC today. As I discussed previously the Tibco corporation has decided that the BPEL TC's use of the term rendezvous, even though we explicitly use it in its common English usage, is violating their trademark. To this end the Tibco corporation has demanded that the BPEL group desist in its use of the word rendezvous as the value of an attribute in BPEL. Mind you, what Tibco hasn't even attempted to do in the group is to actually demonstrate that our use is, in any way, a violation of their trademark.

To make matters worse today's discussion of the issue on the group phone call focused on the possibility of Tibco suing people. The assumption of the group was that if we didn't change the term then Tibco would start suing us. An assumption, I would add, that the Tibco representative neither encouraged nor attempted to dispel. So under the cloud of the implicit threat of a lawsuit the group voted. 11 people voted to open the issue and implicitly to change the term. 6 people voted to not open the issue and thus reject Tibco's claim. 13 people had so disassociated themselves from the proceedings that they abstained.

What hope do our freedoms have if 24 out of 30 people are willing to throw those freedoms away without any evidence of an actual infraction? What hope do our freedoms have when 24 out of 30 people hold those freedoms so cheap they would discard them at the slightest whiff of a threat? As things now stand the Tibco corporations has successfully removed the word rendezvous from the BPEL TC's dictionary without having to spend a dime on legal proceedings or providing any proof that our use is infringing.

There is still, however, some hope. We will have a vote in the group on what term to replace 'rendezvous' with. The group could still choose to not replace it at all. There is still a chance left for people to realize their mistake and stand up for all of our liberties. Our individual actions do matter. Our collective freedoms only exist to the extent that we exercise them. Let's hope that the group decides that our rights to use our own language without corporations like Tibco taking the very words from our mouths is a right worth defending.

BPEL, TIBCO and trademarking the English language

Administrator — Wed, 13 Apr 2005 00:00:00 +0000

Issue 202 in the BPEL TC is a demand from Tibco that the BPEL TC change the word 'rendezvous' that is used as the value of an attribute in the BPEL programming language because Tibco has trademarked the word 'rendezvous'. To be clear, trademarks do not apply to 'descriptive' uses and that is exactly how BPEL uses the term 'rendezvous'. So legally speaking Tibco most likely doesn't have a leg to stand on. I do have sympathy for Tibco because one is required to 'vigorously' enforce a trademark in order to keep it. So to protect their trademark they need to be seen to be defending it, hence issue 202. But now people in the group are scared that Tibco will sue them (or their employer) if we don't change the term. Hopefully the group will stand up for the freedom to use the English language and reject issue 202 on next week's call. An even better outcome would be a change in the law that would make it illegal to get a Trademark on a common English word. The fact that Tibco could trademark the word 'rendezvous' and then try to banish its use from technical efforts is insane.