Contents

1 Kicking OAuth delegation up to 11

Delegation is all about transferring permissions. At its most general a delegation expresses something like ”User X of service Y gives permission Z to user A of service B”. Today OAuth cannot express this statement because OAuth always assumes that X = A. In other words what OAuth can say is ”The user of service Y gives permission Z to their own account on service B.”

With general delegation a user of Sharepoint Online can give permission to a user of Google Docs to see the Sharepoint Online user’s documents from inside of Google Docs. Or, as explored below, a user of Yahoo calendar can give permission to a user of Live calendar to see the Yahoo calendar user’s free/busy time from inside of Live calendar. With full delegation we can finally start having truly open permissions.

2 The first step on the road to general delegation - target user

Imagine that we added one feature to OAuth, the ability in a permission request to specify who the target user is. This would enable us to make the following statement in OAuth ”The user of service Y gives permission Z to user A of service B”. But how do we identify user A? We already have a pretty good solution to how to identify users, e-mail addresses. So let’s go with that.

But now imagine the user experience. Yochi, who uses Yahoo Calendar, wants to give Leon, who uses Live Calendar, permission to see his free/busy time. Using the new ’target user’ functionality Yochi can say to Yahoo Calendar ”Hey, Yahoo Calendar, give leon@bogus.gmail.com who uses Live Calendar the right to see my free/busy time.” Yahoo calendar can now make a three legged OAuth request to Live Calendar and specify that the permission to be granted, free/busy time read capability, is to be given to the user leon@gmail.com.

But how does Live Calendar know who is giving the permission? In other words, how does Live Calendar know that Yochi is the one granting the permission to Leon? If all we add is Target User then what will have to happen is that Yochi will have to be sent to a browser pointed at Live Calendar who will then require Yochi to prove his identity. Personally I think that oauth can handle that part just fine but let’s just say we use OpenID. So now Yochi has to login to Live Calendar via OpenID to prove to Live Calendar who he is.

The good news is that all the mechanics needed to make the previous work exists. This is important because it will take a while for the machinery I describe below to be ubiquitous so this is a reasonable fall back experience. But we can do better, much better.

3 The next step on the road to general delegation - on behalf of

In the previous section we added to OAuth the ability to specify the user who is the target of a request using e-mail. What if we could also specify the identity of the user the request is on behalf of? In other words, imagine we add an ’OnBehalfOf’ field to OAuth? Now we have reached the fully generic semantics we talked about above. We can make a statement of the form ”Yahoo Calendar, on behalf of Yochi@bogus.example.org, gives permission to see his free/busy time to leon@bogus.gmail.com using the service Live Calendar.” The beauty of the previous statement is that Yochi no longer needs to login to Live calendar to prove his identity. So long as Live calendar believes that Yahoo calendar is authorized to act on behalf of Yochi the entire permissioning step can be handled with no additional UX.

But this begs the question, why should Live calendar believe that Yahoo calendar has the right to speak on Yochi’s behalf?

4 Making on behalf of actually work - discovery

There are a number of ways that Live calendar can figure out if Yahoo calendar is allowed to speak on behalf of Yochi. But a fairly straight forward way that turns out to have some really nice capabilities (which I’ll get to in the next section) is a discovery server. Imagine if Live calendar could take Yochi’s e-mail address, Yochi@bogus.example.org and make a GET request of the form https://bogus.example.org/.well-known/finger-service?user=Yochi@bogus.example.org&service=URN:Services:calendar and get back the URL for Yochi’s calendar service? Notice that as directories go this is a fairly tame one. Queries consist of two fields (a user e-mail and a service ID) and the response is one or more URIs. In this case if the URI returned to the query points at Yahoo calendar than Live Calendar knows that Yahoo Calendar really is authorized to make statements on Yochi’s behalf, at least in regards to calendaring.

5 Another nice feature of discovery

The calendar scenario is predicated on Yochi being able to specify that Leon’s calendar is at Live Calendar. But how did Yochi know that? Did he ask Leon where he keeps his calendar and then type in some URL pointing at Live Calendar into Yahoo Calendar? Did he pick from a drop down? Ideally Yochi could just say to Yahoo calendar ”Hey, my friend Leon’s e-mail address is leon@bogus.gmail.com, give him permission to see my free/busy time” and then Yahoo calendar would handle the rest. Thanks to the discovery service that’s possible.

Yahoo calendar can go to https://bogus.gmail.com/.well-known-finger-service?user=leon@bogus.gmail.com&service=URN:Services:calendar and get back the URL for Leon’s calendar service, in this case, Live calendar. And then Yahoo can make an on-behalf of request and set up the permission. So Yochi didn’t need to know where Leon’s calendar service was at. He just needed to know Leon’s e-mail address.

6 Conclusion

The goal was to enable generic delegation so that any user of any service can give permissions to any user of any other service. To make this happen we needed to introduce two key features. First was on behalf of (and the related target user). Doing this required little more than adding two fields to OAuth requests along with the idea that users are identified by e-mail addresses. Second was discovery. By enabling a simple discovery server (two strings in, a list of URIs out) we both enable users to find other user’s services as well as allow services to figure out which services are allowed to act on behalf of which other services.

These two features give us the foundation for open permissions on the Internet.

A Some odds and ends

My goal with the previous article was to provide a very high level overview of how full delegation could work in OAuth. In this appendix I provide a similar high level overview of a number of ancillary features needed to fill out the experience.

B How exactly did Live calendar provision with Yahoo calendar?

In most OAuth contexts there is an assumption that two services have provisioned with each other out of band and that’s why they can securely talk to each other. But requiring everyone to provision out of band with everyone else is not a recipe for a scalable web. What I think we want is the ability for any two arbitrary services to be able to provision a relationship dynamically. There are two parts to provisioning from a technical perspective. One part is establishing a secure communication channel. The other part is establishing a shared application protocol.

B.1 Establishing a secure communication channel

Establishing a secure communication channel generally means creating a mutually authenticated connection, that is, a connection where both ends have authenticated each other. Public key cryptography would be a great technology to use here were it not that most of the languages/platforms used on the Internet do not have good public key support. So instead I propose that we use SSL (ironically built on public key cryptography but in a sufficiently low level way that there exist high quality wide spread support) to establish a shared symmetric key between two services. This key can then be used to sign things like request/access tokens (thus authenticating the sender) over a SSL connection (thus authenticated the receiver).

Establishing the shared symmetric key requires two round trips. For example. let’s say that Yahoo Calendar wants to provision with Live Calendar. Note that this provisioning would only need to happen once. The symmetric key, once established, could be used with all on behalf of requests between the services independent of the users involved. In other words the symmetric key is provisioned between the services directly.

1. Yahoo calendar (after using discovery to find the Live calendar provisioning endpoint) would send a HTTPS request to Live calendar’s provisioning endpoint including a cryptographically secure random number.
2. Live calendar (after using discovery to confirm the Yahoo calendar provisioning endpoint) would send a HTTPS request to Yahoo calendar’s provisioning endpoint including both the cryptographically secure random number and the symmetric key that is to be used between the services.

That’s it. The two services have now securely established a shared key that can be used to sign tokens and prove identity. This same mechanism can also be used to generate new keys when it’s time to do a key rollover.

B.2 A shared application protocol

As for the second part, the shared application protocol, the Internet is full of those. Are we talking about a file or photo sharing service? WebDAV. Are we talking about a content management system? Delta-V. calendar? CalDAV. You have a problem? DAV has a protocol. :)

C How did Yochi’s discovery server come to list Yahoo calendar as Yochi’s calendar service?

In the scenario Yochi’s discovery service is at bogus.example.com but Yochi’s calendaring service is at Yahoo. How did Yahoo tell bogus.example.com to list Yahoo Calendar as Yochi’s discovery service? Making this work is really just an application of bog standard OAuth. After setting up his Yahoo Calendar, the Yahoo Calendar service would redirect Yochi with a standard OAuth delegation request to Yochi’s discovery service to set itself as Yochi’s calendaring service location. Yochi would confirm to his discovery service that he agrees (again, this is bog standard OAuth) and the change would be made.

D And why was Live calendar allowed to discover who Yochi’s calendar service is anyway?

It’s not a good idea to have discovery services sharing information about user’s services to just anyone. So in our scenario the discovery service should have demanded authentication from Live. But it seems doubtful that Yochi would have thought to give Live permission to see his discovery data. What is more likely is that Yochi set up his discovery service to allow anyone in his address book to see his service locations and Leon is in his address book. So what should happen is that Live should make an ’on behalf of’ request to Yochi’s discovery server in Leon’s name. But how does Yochi’s discovery server know that Live Calendar is allowed to act on behalf of Leon?

The answer is that Yochi’s discovery service will make a request to Leon’s discovery service on behalf of Yochi to find out Leon’s calendar service and see if it’s Live Calendar. Leon’s discovery service can easily validate that Yochi’s discovery service is allowed to act on Yochi’s behalf by matching Yochi’s e-mail address to the discovery service location (e.g. they are both bogus.example.org) so it knows everything is o.k. Since Yochi is in Leon’s address book, Leon’s discovery service will respond to the request confirming that Live is allowed to act on behalf of Leon. And now the scenario is complete. Yochi’s discovery service knows that Live can act on behalf of Leon (at least in this context) and since Leon is in Yochi’s address book the request for calendar location information from Live is positively responded to.

E What if Leon wanted to ask Yochi for permission to see his free/busy time?

This article has worked on the scenario where Yochi wants to inform Leon that he has been granted a permission. But a more general case is that Leon realizes he needs access to Yochi’s free/busy time. This just requires adding a command to OAuth ”asking for permission” whose semantics are ”User X of service Y asks for permission Z from user A of service B”. But all the details of discovery, permissioning, etc. are the same as above. Once user A reviews the request and approves it then the process of notifying User X that they have received the permission is exactly as given above.

F Where can I go to learn more? (Read: I’m having trouble sleeping)

I’ve tried my best to keep this article focused on the big picture and in something reasonably approximate to English. But I’ve written a series of articles going into the gory details of how all this can be made to work. Those articles are:

Open Permissions Matter for an Open Web

An argument for why open permissions matter and an exploration of the components needed to make them happen.

The Outline of a Profile for Granting Permissions Using OAuth WRAP

Provides a sequence diagram and supporting material explaining how the basic protocol exchanges would work.

Thoughts on Building a Finger Service

What I call a discovery service in this article was more traditionally called a finger service. This article explores key issues in designing a dirt simple finger service.

Using OAuth WRAP and Finger for Ad-Hoc User Authentication

This is where I first introduce how to do ad-hoc provisioning (e.g. using two request/response pairs to establish a shared symmetric key). But I do it in the context of showing how OAuth can be used as a replacement for OpenID v1.

Thoughts on Updating Finger Services

All the gory details on how services can update user’s discovery services.

Contents

The OpenID community has worked long and hard to make ad-hoc logins possible on the web. Part of that process has been experiments with a number of different technologies and approaches. Below I make my own proposal for how to handle ad-hoc logins on the Internet using OAuth WRAP and my own spin on Finger. I offer this up as food for thought.

Contents

1 Disclaimer

My employer has nothing to do with anything in this article. They didn’t review it, authorize it or influence it. It’s my ideas and my ideas alone. So blame me.

I have a deep interest in identity issues mostly as part of my fervent desire to live in an Open Web and that web doesn’t exist today. So I ask the reader to please take these ideas in the spirit that they are offered, as a fellow traveler trying to find a successful path to a web where users decide who they want to interact with.

2 Logging into the Foo service - a scenario

Joe wants to use the Foo service. To do this Joe needs to log into the Foo service. Joe’s identity provider is example.com where Joe is known as joe@example.com. The Foo service and example.com have no previous relationship. Joe tells the Foo service that his email is joe@example.com (note: the scenario works just as well with just Joe’s domain so there is no need to expose one’s identity). The Foo service then uses finger (my thoughts on which I have explored here) to obtain example.com’s symmetric key negotiation service and login service endpoints. The Foo service uses the key negotiation service to negotiate a symmetric key with example.com. Then the Foo service uses a profile of OAuth WRAP to forward Joe to the login service asking for proof that the user really is joe@example.com. Example.com validates Joe’s identity and then forwards Joe’s web browser back to the Foo service with a security token attesting to Joe’s identity.

3 My thoughts on requirements

3.1 No federation, no registration, it’s truly ad-hoc

We need an approach to authenticating users across services that doesn’t require the services to have any pre-existing relationship. The services must not need to register with each other, federate or any other ’off line’ magic in order to successfully authenticate users to each other.

3.2 No public key encryption beyond SSL/TLS, at least not initially

The use of public key encryption has obvious applications to this scenario. But I believe we have to start simple with solutions that do not require any form of PKI beyond SSL/TLS which I believe should be mandatory requirements of the protocol. Yes, in the future, we definitely should extend to PKI because it has some very nice advantages but I believe that the base functionality shouldn’t use anything more than SSL/TLS with some HMAC thrown in for validation of security tokens. Note however that support for SSL/TLS is mandatory and a critical component of the security of the key negotiation algorithm defined below.

4 An example

When Joe first navigates to the Foo service he is prompted to login. Ideally Joe should just type in his identity provider’s domain name (example.com) but realistically most users won’t ’get’ that (at least not initially) and will probably just type in their e-mail addresses, in this case, joe@example.com.

The Foo service has never had an interaction with example.com before so the first thing it does is do a lookup on example.com’s finger service to find the location of example.com’s key negotiation endpoint. This process consists of sending a POST request to https://example.com/.well-known/finger-service?service=example.com with a scope (a la OAuth WRAP) of URN:SomeStandardsOrgId:KeyNegotiationService. The response would contain a URL such as https://example.com/keynegotiationservice.

The Foo service would then send a request to establish a symmetric key to https://example.com/keynegotiationservice. The request would contain Foo’s key negotiation service URL (https://foo.com/mykeygenerator), a request ID and a cryptographically secure randomly generated number that I’ll call proof1.

Example.com would then double check the key negotiation service location by using a finger request on foo.com to find foo.com’s key negotiation endpoint. Once it had validated that the URL it received in the key negotiation request matched the value retrieved from finger then example.com would issue a POST to that URL with the request ID, proof1 and a symmetric key value that is to be used to sign security tokens between example.com and the Foo service.

At this point there is now a symmetric key that both Example.com and the Foo service (represented by the domain foo.com) have agreed to use with each other.

Now the Foo service will perform a second finger lookup but this time look for URN:SomeStandardsOrgId:LoginService. This time the Foo service wants to know where to send Joe in order to log him in.

Foo service will then forward Joe’s browser to the URI returned in the previous finger request along with an OAuth WRAP style permission request of ”Please log this person in and tell me who the heck they are.”

Example.com will then log Joe in and ask Joe if he wants to tell Foo service who he is. Joe will say yes and Example.com will redirect Joe back to Foo service with a security token, HMAC’d with the shared key, asserting Joe’s identity.

And now we’re done. Joe logged into Foo service using his identity provider example.com even though Foo service and example.com had never had any previous interactions with each other.

Only the colored links involve standardization. The blue links (3/3R, 5/5R and 7/7R) use finger whose design I have discussed elsewhere. The green links (4/4R and 6/6R) are the symmetric key negotiation algorithm I introduce in this article. The red links (2R, 8, 9R and 10) are a minor profile of OAuth WRAP that I discuss in this article.

4.1 What if Joe’s identity provider isn’t the same as his domain name?

This is actually not an uncommon situation. For example, users establish Facebook accounts with e-mail addresses that Facebook doesn’t own yet Facebook acts very much as an identity provider. Some services, like Google and Live are identity providers where users can either choose to use a domain for their e-mail controlled by Google or Live (e.g. gmail.com or live.com) or they can use an existing e-mail address. This can sometimes create confusion. If Joe has a Live account he created with the e-mail address joe@example.com and Joe wants to login to the Foo service then if he says his e-mail is joe@example.com the Foo service will go to example.com instead of live.com who is Joe’s identity provider.

This is a really sticky problem and I’m pretty convinced that it’s not interesting to solve. The reason identity providers like Live or Google let users use existing e-mail addresses from domains not owned by identity provider was as a convenience. That convenience is proving more and more troublesome to the point where the message needs to go out - login using your identity provider. So if Joe wants to be known as joe@example.com then example.com better be his identity provider. Otherwise he needs to get another e-mail address.

I know many people don’t find this to be a satisfying answer but I believe that bending ourselves over backwards to solve a weird pointer problem just isn’t worth the effort.

4.2 Why do we need proof1 in messages 4 and 6? Shouldn’t the request ID be enough?

As long as the request ID contains a cryptographically secure random number of sufficient length then yes it is enough. But I felt like calling out the requirement for the secure value as its own value in the exchange in order to hammer home how important it is from a security perspective. The security of the key establishment hinges on the use of TLS/SSL and the exchange of an unguessable secret. If either is compromised then the key establishment protocol is not secure.

4.3 How did Example.com know what security token format to send proof of Joe’s identity in to the Foo service or what claims to use?

My assumption is that we will define profiles for common situations like this. So there will be a profile called ’login’ and that profile will specify things like what security token format to use and what claims can be placed in that security token.

4.4 Wouldn’t it be easier for the Foo service to just make one directory lookup request against example.com instead of two?

This is really more of an issue for the finger server article but my opinion is that with HTTP/1.1 pipelining (and in this case the POST’s semantics are idempotent so pipelining is fine) I don’t see any reason to over optimize. Besides these requests should be reasonably rare.

4.5 Why is Example.com using finger in 5/5R to find Foo’s key negotiation service URL when that value was already provided in message 4?

In theory any request that can be validated against a domain over TLS/SSL should be ’trusted’ as having properly come from that domain. But in practice paranoia is a healthy thing. Let’s say an attacker has managed to take over some small part of example.com and is using that to launch key attacks. By checking the key negotiation service location in message 4 against the value returned by the finger service in 5R, example.com gives itself an extra level of protection.

4.6 How do keys expire and get replaced?

As part of the key exchange one expects that an expiration date will be associated with the key. To keep things running smoothly it will be necessary to be able to roll keys over without a ’gap’ where no key is in place. This means that at a minimum the two parties to a key negotiation need to handle having at least two keys active at the same time.

For example, let’s say a key has been negotiated and will expire in a few days. One of the parties may decide it is time to create a new key so that it can be established before the old key expires. For this scheme to work at least two keys have to be active at the same time, the old key that is about to expire and the newly established key.

In general a new key is established any time the key negotiation exchange is made. So in theory an unbounded number of keys could be put into play. In practice however each side is likely to have some upper limit to how many keys they want in play at any one time. This limit can be enforced either by refusing to add new keys if there already exist a maximum number of unexpired keys or dropping off an existing key (e.g. no longer honoring it) if the maximum number of supported keys has been reached. Which approach isn’t as important as making sure both ends of the conversation understand what has happened. My own preference is for each side to just support a maximum number of keys and to refuse to create new ones if that maximum is filled with unexpired keys. I think the failure scenarios for that situation are easier for each side to understand.

4.7 What happens if a key gets compromised or lost?

My own thinking is that the key negotiation protocol should have a message exchange type of ”Delete all keys we have negotiated” along with a human description of what the heck happened and some contact information since a follow up is probably necessary. This exchange would occur using the same two step pattern as establishing a key so as to prevent attacks.

4.8 Aren’t there race conditions between two services trying to establish keys?

You betcha. Imagine a case where a user is logging into one of example.com’s services using a Foo service identity and vice versa. This could easily result in two different keys getting established between the same services. But as long as we mandate that each side be able to handle a reasonable number of keys (say 10 just to pick some integer) then it really shouldn’t matter.

4.9 Does Foo service repeat this entire process with everyone from example.com who wants to login?

Heck no. The negotiated key is good for all communications between foo.com and example.com

4.10 Why do we need to discover the location of the key negotiation or login services? Why not just hard code their locations under /.well-known?

Somewhere I can hear Mark Nottingham groaning. Strictly speaking hard coding the locations under /.well-known is at least logically consistent. But in general hard coding makes for fragile systems. It limits implementers flexibility around where services should be hosted. So it seems like a good idea to hard code as few things as possible and let everything else be dynamic. Hence the desire to redirect through a finger server.

Those folks of a certain age will remember the finger command/protocol which allowed one to look up information about a person based just on their login identifier. This command was extremely useful even if it had some troubling security and privacy implications. Efforts are underway to create a Web Finger but for reasons I’ve previously discussed I think the underlying technologies for those efforts are sub-optimal. So in this article I propose what I think is a much simpler approach. My motivation for caring is that I think having a finger service will make permissioning systems much more useful (see here and here).

Contents

1 Disclaimer

I don’t always remember to put on a disclaimer because this is my blog and my opinions and by default the assumption should be that anything I say here represents my views and only my views. But just in case, this article (like all the articles on my blog) represents my views and only my views.

2 Scenario - discovering a friend’s calendar service using a local application

Martok wants to invite Dax to a meeting but to do so Martok needs to see Dax’s free/busy time which is available at Dax’s calendar service. But Martok doesn’t know which calendar service Dax uses. So Martok uses the finger functionality from inside of his local calendaring client to take Dax’s e-mail address and use it to discover Dax’s finger service. Martok then authenticates to Dax’s finger service and queries for Dax’s calendar. Dax’s finger service has been instructed to share details such as Dax’s calendar service location with anyone who is part of Dax’s personal address book. Since Martok is in Dax’s personal address book he is allowed to see where Dax’s calendar service is kept.

Note: This scenario is strictly about programmatic interaction with the finger service. I’ll leave UX issues for another day.

3 My personal opinions on requirements

3.1 The finger service should just point at other services

Once upon a time finger directly returned key information about the user such as their office location or status message. But in the new world order with social networks all over the place it seems more reasonable to think about finger as a catalog of pointers to more specialized services. In other words one shouldn’t go to Finger to find out someone’s status. One should rather go to Finger to find out what service or services a user uses to publish their status (Messenger? Facebook? Twitter?). The same thing with other services such as blogs (personal or work?), calendars, preferred social network, etc.

3.2 The finger service needs to support authentication and authorization to control who gets to see what

Because of the privacy and security implications of finger it’s necessary to authenticate who is requesting information (even if the authentication is just ’anonymous’) and to require the requester to specify exactly what information they want. This allows the finger service to reason about who is making the request and what they want and so decide what is appropriate to share.

It’s tempting to argue that it isn’t necessary to know what the requester wants since who they are should specify what they can see. But I believe that as a rule it’s not a good idea to over share.

3.3 The finger service should just be a service, not a document

I have previously explained my concerns about the XRD/LRDD approach to finger. So I won’t repeat myself here other than to say - I believe finger is a service not a document. So I expect that people will issue queries to the finger service and get back responses, not just do global GETs.

3.4 The finger service should start with trivial queries and get complex later

At its heart the finger service is a database with ACLs on top. So one could imagine specifying that the interface for the finger service should be something like OData. I actually expect that eventually finger services will sport interfaces like OData because they will make potentially interesting query and data navigation capabilities possible.

But in the short run I think we need to focus on easy things first. If all we do initially is just create an interface where someone can submit an identifier of a service and get back what, if anything, the finger service is willing to say about that the user’s usage of that service then we will have done a good thing for the world.

3.5 The finger service should at least support using e-mail addresses as user identifiers

A perennial argument about how to identify users is what kind of identifier should be used? Whatever that identifier is it has to be something that enables a finger client to go from that identifier to the finger service that handles that identifier. Similarly the identifier needs to be something that is easy for users to remember and exchange. Something that is easily transcribable and typeable so users can just type a friend’s identifier into an application or service and away they go. To date there is exactly one user identifier that has proven robust across all of these requirements - e-mail addresses. So email addresses are a minimum required user identifier. This means that the finger service has to define how to find the finger service for a particular individual by starting off with nothing more than their e-mail address.

3.6 The finger service should use HTTP, in preference to DNS, to go from e-mail to finger service

Going from an e-mail address to the finger service that owns that address essentially requires figuring out how to use the DNS name in the e-mail address to find the finger service. One obvious way to do this would be DNS SRV records. We could add a new SRV type for finger services and allow discovery that way. The downside to this approach is that it requires the ability to manipulate DNS records on the server side and the ability to make DNS queries on the client side. Neither is a common operation for most programmers or users. So basing discovering on DNS is likely to slow adoption. Nothing stops us from adding an extension based on DNS but I personally don’t think that it is the right place to start.

Instead I think we should build on top of RFC 5785 that defines a location for hosting ’well known’ services via HTTP. So if someone has the e-mail address joe@example.com they could try to find the finger service for Joe at http://example.com/.well-known/finger-service.

There are some pretty obvious problems with this approach. For example, what happens if the owner of one’s e-mail domain doesn’t support the finger service? Is one supposed to change one’s e-mail address just to get to a service provider who does support the finger service? The answer, I believe, is, unfortunately, yes. It’s the nature of hierarchical trust systems that there are choke points, in this case, the domain owner. We could look for a web of trust system but previous experience with systems like GPG teach that while these systems can be powerful they are hard for users to understand so I don’t feel they are a good starting point.

Nevertheless the RFC 5785 approach does offer flexibility. For example, let’s say I don’t want to host my own finger service but I do own goland.org. I can easily set up a redirect rule in my domain to redirect requests for http://goland.org/.well-known/finger-service to where ever my finger service actually rests. I would posit that more people know how to set up redirect rules then know how to configure SRV records.

3.7 The finger service should use SSL/TLS when dealing with non-anonymous queries/responses

Any finger service that needs to support authorization/authentication (and one hopes that is the majority) must support SSL/TLS. The OAuth 1.0 signature/encryption experience has taught that adding message integrity/confidentiality as a layer on top of protocols like HTTP is so difficult as to be impractical for even sophisticated developers. So rather than re-invent the wheel we’ll require using the one that’s already there, SSL/TLS.

Note: The reader is well advised to review articles such as this one that explain some fundamental flaws in how SSL is used operationally. The way certs are used today enables abuse and I think this is much more threatening in the long run then functional flaws like the renegotiation attack for which a fix is likely to be available sooner rather than later.

4 Martok gets Dax’s free/busy time - An example

In this example Martok’s calendar client is going to start with Dax’s e-mail address and use it to go to the reserved URL for the finger service and ask for Dax’s calendar location. Embedded in the request is a security token authenticating Martok to Dax’s finger service. For right now I’ll just wave my hands and say that Dax’s finger service is federated with Martok’s Identity Provider and so recognizes the HMAC/signature/etc. on the security token. In a later article I’ll explain how we can build on top of the finger service to enable ad-hoc authentication of users without federation.

Dax’s finger service will validate Martok’s security token and return the location of Dax’s calendar service. Martok’s calendar client will then get a security token for Dax’s calendar service and use that to access Dax’s free/busy time. Again, I’m going to assume that Martok’s Identity Provider is also federated with Dax’s calendar service.

Note: To keep the picture from getting too big I have collapsed the services and their associated authentication servers into a single box.

This picture is essentially the STS/Kerberos two step repeated over and over again. Martok’s client gets an access token from the identity provider which is provided to the service’s authentication server who exchanges it for another security token that is then presented to the service. All the exchanges in red are bog standard OAuth WRAP calls. The only thing needed to be added to support finger is 4/4R in blue. My guess is that even 4/4R will be an OAuth WRAP profile extension where the wrap_scope will probably be a URI indicating the user and service that is being sought. Or maybe we can add another field. In any case it’s a pretty obvious variation.

4.1 Where did the refresh token in message 1 come from?

I’m assuming that Martok’s calendar client is using OAuth WRAP section 6.3 to get a refresh token to allow it to talk to Martok’s identity provider. I intend to go more into this scenario in a future article.

4.2 How can the access token from 1R be used both in message 2 and message 5?

Both requests require the client to authenticate itself to the same service, Martok’s identity provider, so the same access token can be used. The only problem is if enough time passed between message 2 and 5 that the access token (which is typically short lived) expired. In that case the client would need to repeat request 1 to get a new access token. This is all standard OAuth WRAP behavior.

4.3 Well then, why couldn’t the access token from 3R be used with request 6?

Because access tokens are service specific. The access token returned in 3R was only good with the finger service so it couldn’t be used with the calendar service. Therefore the client had to execute request 5 to get an access token for the calendar service.

4.4 Wait, remind me again, how did Martok’s calendar client know where Dax’s finger service was?

The scenario assumes that Martok, either directly or via an address book, gave the client Dax’s e-mail address. Let’s assume that Dax’s e-mail address is dax@example.com The calendar client pulled the domain name out of the e-mail address, in this case example.com, and then created a request of the form http://example.com/.well-known/finger-service. Note that I just made up ’finger-service’, the actual value will be whatever goes into the standard. The interesting question is how did the request identify Dax?

One way is in the URI itself. We could specify https://example.com/.well-known/finger-service?localname=dax. We could even handle cases where a finger service is handling an address from a domain other than its own via https://example.com/.well-known/finger-service?email=dax@federation.com. Alternatively Dax’s identifier could go into the request body, possibly into the wrap_scope element.

4.5 Why do we bother with the previously mentioned STS/Kerberos two step anyway?

The answer is scalability. It turns out that requiring services to handle authentication directly is expensive. Life works better if there is a centralized authentication service who handles the expensive authentication actions and then hands out a short lived token (typically good for a few hours) that can then be used with services without further ado. This is why OAuth WRAP uses the refresh and access tokens.

4.6 How did Martok get permission to access Dax’s finger service or free/busy time?

For the example we just assume that Dax said something like ”anyone in my address book can see my calendar location and my free/busy time” and Martok was in Dax’s address book. Martok’s identity was validated to the finger and calendar services by the security token issued by Martok’s identity provider. But in the more general case I’m proposing that we extend OAuth WRAP to enable individuals to ask each other for permissions. See here and here for more details.

4.7 How did Martok’s identity provider know how to issue security tokens with formats and claims supported by Dax’s finger service and calendar service?

My guess is that if the finger service as proposed here takes off then there will be a defined standard for a default security token format along with a default identity claim that can be used in any situation where one entity needs to prove its ’identity’ (in this case, an e-mail address) to another. See the previous question for why the finger service and calendar service honored the identity assertion from Martok’s identity provider.

4.8 Couldn’t Dax’s finger server tell Dax’s calendar server that Martok is allowed to have access to free/busy time?

This is a feature I’m interested in mostly because over time individuals trying to manage their security in a coherent way are going to get a really serious headache. How can anyone remember which permissions they gave to which services for which people? What’s needed is a centralized place where users can manage their permissions across many services. The finger server could either be that service or be in communication with that service. In that case rather than Dax telling the calendaring service directly ”Give Martok access”, she could instead tell this to her centralized permission service who would then be discovered by Martok using the finger service. Martok would then ask that central permissioning service for a token to give to Dax’s calendar service to prove he had a right to access her free/busy time.

But I think it’s premature to try and create this kind of infrastructure right now. To be blunt, until it hurts nobody is going to spend time and money fixing it. So while some enterprises are just now barely trying to get started on issues like managing security policy across multiple internal services (and believe me, that hurts way worse than a single user managing their permissions) I suspect users effectively addressing this issue is some way into the future.

In fact my strong suspicion is that the way things will really work is that users are going to synch at least the e-mails in their address books across services and then specify permissions based on the address book’s structure or tags. ”Anyone marked as family can edit X” or ”Anyone marked as business can see Y”, etc.

It’s tempting to argue that we shouldn’t design a system we know is problematic at the start but synchronized permission management is an unsolved problem mostly because we aren’t sure as to the most effective way to present permissions to users (this is also known as a the policy problem).

4.9 Couldn’t Dax have multiple calendar services?

Absolutely. My guess is that services like calendars will have standard finger schemas that can say things like ”I’m the work calendar” or ”I’m the second personal calendar” or whatever. It will be up to Martok and his calendar client to decide how many of those services he should try to synch free/busy time for.

In a previous article I talked about adding a profile to OAuth WRAP that would enable users to ask for or grant permissions to each other. In this article I show that an OAuth WRAP profile to handle granting permissions only needs two request/response pairs. I then show that an OAuth WRAP profile to handle asking for permissions only needs one additional exchange.

Contents

1 Granting Permission - The free/busy time example

In this example the user joe@google.com, who uses the Yahoo Calendar service, wants to give permission to mike@example.com, who uses the Live calendar service, to see Joe’s free/busy time. Mike will then login to Live calendar service, accept Joe’s permission grant and then schedule a meeting with Joe where Joe’s free/busy time will be displayed to Mike.

In looking at the sequence diagram it’s worth noting that the entire proposal comes down to creating a profile for OAuth WRAP that introduces request/response pairs 2/2R and 3/3R. That’s it.

7/7R are already defined by OAuth WRAP, 8/8R are defined by other protocols like CalDAV and everything else occurs outside of any standard context. So while a lot is going on the amount we have to add to enable interoperability is quite small. The sequence diagram does, however, bring up some interesting issues that I explore below.

1.1 How did Joe know to tell Yahoo that Mike keeps his calendar at Live?

One can imagine that Joe went into his Yahoo address book and clicked on a button saying "Give Mike permission to see my free/busy time". Yahoo knows that Mike’s e-mail address is mike@example.com but how does Yahoo figure out where Mike keeps his calendar? I don’t think it will ever make sense to ask a customer where their friend’s calendar is because the customer might either not know or have the wrong answer. For privacy reasons Yahoo in many cases can’t tell the customer if their choice is right or wrong. So my suspicion is that Yahoo will need to find out for itself where the calendar is for the associated e-mail address.

In the long term there will exist discovery services that given an identifier like an e-mail address can return information such as where the owner’s calendar(s) is(are). I hope to discuss my ideas around that problem in another article. But in the short term my guess is that services will just have to ask their partners. In some cases, where there is very deep trust (or very long legal agreements and enough money to sue if things go bad) services will just bulk upload lists of who they host calendars for to their partners and update those lists on a regular basis. In other cases a service will have to send a protocol request to their partners asking "Do you host X’s calendar?" This request/response pair is shown in 2/2R in the diagram.

This means that a service like Yahoo who is probably partnered with a large number of other services will either have to check its local database of bulk uploads and/or make queries to partner services in order to figure out where a user’s calendar is. This won’t scale but it’s probably o.k. in the short term until the discovery service infrastructure is up and running.

1.2 What if Mike has multiple calendar services?

We can easily imagine that Mike has a personal calendar that he keeps at Live and a work calendar that is run from his business. One can also imagine that both Live and Mike’s business are federated with Yahoo. So when Yahoo goes out looking for where Mike’s calendar is they are likely to find not one, but two calendars. In fact it could be even more complicated. Let’s say that before deciding that Live was his primary personal calendar service Mike created accounts with a few other calendar services. He has since abandoned those calendars but he used the same e-mail address, mike@example.com, to create all the calendar accounts. If Yahoo is federated/registered with those calendaring services then Yahoo could find itself having numerous potential calendaring services who all say they represent mike@example.com and at least two of them are even right!

One suspects that the only sane way to handle this for now is for Yahoo to just find all the calendar services who have an entry for Mike and send the permission request to all of them. Most likely Mike will probably accept Joe’s request for his Live calendar and reject Joe’s request for his work calendar. But just as likely Joe might accept both requests and that’s probably a good thing. In the end Joe needs to know when Mike is busy, if Mike hasn’t federated his two calendars together then the only way to have an accurate picture of Mike’s activities is to get free/busy time from both calendars. Yahoo would then pull the data from both and show Joe a unified free/busy time view for Mike.

1.3 How did Live know the requests came from Yahoo?

In the sequence diagram Yahoo sends two requests to Live, one to find out if Live has Mike’s calendar (2/2R) and another to notify Live that Joe has granted Mike permission to see Joe’s free/busy time (3/3R). In the short run my guess is that services like Live will require services that want to talk to it to register and as part of that registration the requesting service, in this case Yahoo, will get what OAuth WRAP calls a client id and a client secret. Yahoo will include those values in its requests and so authenticate itself to Live.

In the long run I’d actually like to see an ad-hoc system that lets two services exchange requests without a previous relationship. In a future blog post I hope to explain how this could be made to work.

1.4 How did Live know what permission Yahoo was asking for?

My guess is that requests 2/2R and 3/3R will use the OAuth WRAP request/response format which is just HTML Forms. So we would expect that request 3 will include some indicator of what permission or permissions are being asked for. This is nothing new in and of itself, OAuth handles this all the time today. What’s unfortunate is that the way OAuth typically handles this is that every service posts its own unique list of permissions it handles and anyone wanting to work with that service has to work with that service’s permissions. This makes economies of scale hard to come by. Also the permissioning part of the game is the easy part, all things considered. The hard work comes in request/response pair 8/8R where the free/busy time is retrieved.

My hope is that folks in communities like calendaring, document sharing, etc. will get together and create packages of both claims for use with OAuth WRAP as well as details on what protocols to use along with those permissions. But one step at a time.

1.5 How did Live know that joe@google.com was the one who asked Yahoo to make the request?

The real question is probably more subtle - how did Yahoo know its user was the legitimate owner of joe@google.com? It turns out that there are a non-trivial number of ways that Yahoo could have validated Joe’s identify. But the two typical approaches are:

E-mail Validation

Joe could have created an account with Yahoo using the e-mail address ’joe@google.com’ and Yahoo e-mailed a confirmation message to that address that Joe then had to click on and then log in again. This provides reasonable proof that the user owns the e-mail address ’joe@google.com’.

WS-Federation, SAML-P or OpenID

Yahoo could have a trusted federation with Google and so accept Google’s attestations of Joe’s identity.

My guess is that initially Live will require that Yahoo disclose the mechanism used to authenticate the user’s identity. Live will then decide how to proceed based on context and the authentication mechanism. Yahoo’s disclosure could be as simple as a field in the permission notification request that says "auth_type = EmailValidation" or what have you. Live would then trust that Yahoo was being honest about the value it put in the auth_type claim.

There are some edge case scenarios where a service that is giving permission might not know the public identity of the person granting the permission. I have some ideas on how to solve that problem but they are fairly complex so I’ll save them for another day.

1.6 How does Joe remove permission for Mike to see his free/busy time?

In message 1R Yahoo let Joe know that it was able to deliver the notification request, although Yahoo won’t tell Joe where the request was sent. But Joe won’t ever know if Mike actually accepted the request since, to be blunt, it’s none of Joe’s business. Joe can offer the permission but that’s it. But what is Joe’s business is if Joe changes his mind and wants to withdraw permission he needs a way to do that.

Thankfully removing permissions can be handled completely outside of the protocol. Joe needs to let Yahoo know that he no longer wants Mike to be able to see his free/busy time and at that point Yahoo will remove Mike’s permission from its permission store. It’s worth remembering that the way OAuth WRAP works every time a particular user gives a particular permission to another particular user a refresh token is issued. That token is unique to the combination of Source User/Source Service/Destination User/Destination Service/Permission. In our case that is joe@google.com/Yahoo Calendaring Service/mike@example.com/Live Calendaring Service/See Free Busy time. So the next time that Live comes with the refresh token (request 7) to get a new access token (7R) the request will be denied since Joe had Yahoo remove the permission.

2 Asking for permission - extending the free/busy time example

In this version of the example Mike asks for permission to see Joe’s free busy time. By way of reference, in the previous example Joe pre-emptively granted Mike permission, Mike hadn’t asked. The scenario is that Mike creates a meeting in the Live calendar service and invites Joe. Live sees that Mike doesn’t have Joe’s free/busy time and prompts Mike if he would like to ask Joe to see his free/busy time. Mike responds in the affirmative. Yahoo queries the services it knows to see if anyone owns Joe’s calendar and gets a positive response from Yahoo. So Live asks Yahoo to forward a free/busy time view request to Joe from Mike. Mike logs in, sees the request and approves it. At that point the rest of the flow is essentially identical to the previous example. Yahoo notifies Live that Joe has granted the permission and Live uses that permission to show Joe’s free/busy time to Mike.

In comparing this example to the previous example only one new request/response pair is added, 4/4R. All the other request/response pairs that deal with anything that needs to be standardized are already defined in the previous example. And even 4/4R is a variant on 7/7R.

A variant on this scenario would be to have Mike both request Joe’s free/busy time while simultaneously granting Joe the right to see Mike’s free/busy time. But I suspect these should be modeled as two separate interactions.

2.1 How does Live know that the notification in 7 is a response to the request in 4?

Strictly speaking correlation isn’t necessary since Live should be able to recognize that it asked for specific permission from Yahoo for a specific user and if it later gets that permission then matching the request with the response shouldn’t be a big deal. One can imagine Live keeping a table where the index is on a set of columns that specify what service the request went to, what user at the server it was targeted at, what permission was asked for and what local user made the request. Correlating 7 with 4 then becomes a simple task of looking up the appropriate entry in the table. If there is an entry then this is a response to a previous permission request and if there isn’t an entry then this is the previous example.

But if we really want we can just include a correlation ID from Live in request 4 that Yahoo is expected to include in 7 because 7 was generated in response to 4. If 7 was generated on its own (e.g. the previous example) then no correlation ID would be present.

Contents

Disclaimer

This article represents my personal opinions and only my personal opinions. Nothing in this article should be construed as representing my current employers opinions, actions, plans, or what have you.

1 Bring in the scenarios!

1.1 Notification that permission has been granted

Jane works at Big Corp international where she uses Exchange to maintain her work calendar. Jane wants to give her spouse, Lucy, permission to see her free/busy time so that Lucy can schedule events with Jane that won’t interfere with Jane’s work schedule. Jane sets up a permission for Lucy in her calendar granting Lucy permission to see Jane’s free/busy time. Lucy, who uses Yahoo Calendaring for her personal calendar, is notified by Yahoo that Jane has granted the permission and asked if she accepts the permission. Lucy accepts the permission and Jane’s free/busy time is displayed in Lucy’s Yahoo Calendar.

1.2 Asking for permission

Morgan is a vendor hired by Istvan’s company to help with a project. Morgan uses Google Docs to edit and manage documents while Istvan uses Sharepoint Online. To get the project done Morgan needs access to certain documents in Istvan’s Sharepoint. Morgan uses a permission request dialog in Google Docs to send a request to Sharepoint Online asking Istvan for access to Istvan’s Sharepoint. Sharepoint Online receives the request and shows it to Istvan who approves granting Morgan read/write access to a particular sub-directory in Istvan’s Sharepoint. Sharepoint Online then notifies Google Docs that Istvan has granted Morgan read/write permissions to a directory in Istvan’s Sharepoint. Google Docs notifies Morgan and displays the folder from Istvan’s Sharepoint as part of Morgan’s Google Docs workspace.

2 Breaking down the scenarios

Below I break the scenarios into what I think are the ’key’ actions. My suspicion is that if we can define reasonable ways to achieve each of these actions then the rest of the questions around how to implement the scenarios will more or less answer themselves.

2.1 Notification that permission has been granted

Identify the principal

In order to grant a permission Jane had to let Exchange Online know ’who’ Lucy is so that Exchange could transmit the permission to Lucy. What is the pointer used to identify Lucy to Exchange Online?

Describe the permission

What format did Exchange Online use to tell Lucy about the permission granted by Jane? What information is needed in the format?

Discover where to deliver the permission notification

How did Exchange Online figure out where to deliver the permission notification to?

Deliver the permission notification

What are the details of how Exchange Online delivered the permission notification, isn’t this just a spammers dream?

Utilize the permission

Assuming that Lucy accepts the permission how does she go about taking advantage of it?

2.2 Asking for permission

Identify the principal

How did Aahan tell Facebook that he wanted to ask for permission from Aanjay at Flicker? What is the pointer that Aahan used to identify Aanjay to Facebook?

Describe the permission

How does Facebook tell Aanjay about her father’s permission request? What is the format? What information is in the format?

Discover where to deliver the permission request

How does Facebook know where to deliver the permission request?

Deliver the permission request

What are the details of how Facebook delivered the permission request, isn’t this just a spammers dream?

My assumption is that if the permission request is granted then notification of this fact will be delivered in the manner specified in the previous section.

3 Addressing the scenario components

3.1 Identify the principal

The obvious answer would seem to be e-mail addresses. Jane types in lucy@yahoo.com (or more likely selects Lucy’s entry in Jane’s personal address book which then resolves to lucy@yahoo.com). Unfortunately e-mail addresses are lousy identifiers for a couple of reasons.

Late bound identifiers

Are identifiers that are resolved to a principal at time of use and are not guaranteed to always bind to the same principal. In other words, Jane’s spouse Lucy may have control over lucy@yahoo.com today, but who knows who might own lucy@yahoo.com tomorrow?

Non transferable

If Lucy decides that she no longer wants to use Yahoo and would like to move to another identity provider or even be her own identity provider there is no well defined way to achieve this goal with an e-mail address. Once the address is abandoned, that’s it. There’s no forwarding mechanism.

Not uniquely bound to an identity provider

It’s tempting to argue that when trying to interact with the identifier lucy@yahoo.com it makes the most sense to contact yahoo.com. But what to then make of folks like Live or Google who will let users create account with other identity provider’s e-mail names? For example, Lucy can create a Google account using her Yahoo address. So when typing in lucy@yahoo.com to give a calendaring permission does Jane mean Lucy’s calendar with Yahoo or Lucy’s calendar with Google?

These are all very serious problems but solving them will require introducing a new identity infrastructure. There are ways to make this new infrastructure work over the old one but the issues are complex and I think we should solve the easy problems before the hard ones.

So I’m going to assume that principals are identified by e-mail. Once we have that more or less working we can move on to the more difficult identity issues.

3.2 Discovery where to deliver the permission notification/request

So who gets notified that a permission has been granted? I suspect a key requirement is that we don’t mandate a single notification point for all services. In other words if I want to notify someone about a calendaring permission I should be able to do that at a different location than say an IM permission. That way services can be well factored and spread out. But this also means that there needs to be some mechanism to go from the principal identifier (an e-mail address) to the right address to notify for a particular service.

There is already a proposed solution for exactly this problem that we could, in theory, build on top of - Link-based Resource Descriptor Discovery (LRDD). But besides supporting no less than three different discovery mechanisms and throwing in XRD I think LRDD has is backwards. It assumes that one publishes all the data to be known about oneself. Besides wasting space this makes versioning and extensibility a nightmare. What if one supports four different types of calendaring protocol with various versions for each one? LRDD requires publishing the whole lot.

It’s tempting to argue that LRDD is easier to implement than a service since with LRDD one can, in theory, just publish a static file. But given the type and amount of data likely to be in a LRDD retrieved discovery file, the ’static file’ argument is, in my mind, a fiction. The data in a LRDD file will be dynamic and it will be maintained so it’s a service. Might as well treat it as one.

LRDD, besides describing what’s in the discovery file (e.g. the file containing the pointers to various services) also describes how to find the discovery file for a particular user when starting with an e-mail address. Specifically LRDD uses Web Host Metadata. But this is just another static file (using the XRD format that is also used in LRDD) that hangs off a well known address. Instead of using a request/response service (”here is the e-mail I’m looking for”, ”here is the address to go to”) It uses a static file that includes a templating language which one downloads and then locally feeds the e-mail address into the template which then spits out a URL.

So the LRDD process of going from an e-mail address to a service URL involves two steps. First one retrieves the Web Host Metadata that contains the pseudo-regular expression that one then downloads and ’runs’ locally to translate the e-mail address to a LRDD discovery file location. Then one retrieves the LRDD discovery file and from there finds out the service address.

Personally I think having a service at a well known location that one can send a request of the form ”I want to talk to the calendaring permission server for user X” and get back a response would be oodles easier to implement and reason about and involve one less hop.

But in any case, at least there is some existing work to start from.

3.3 Deliver the permission notification/request

In theory delivering the request or permission notification is easy enough, just POST some bit of XML or JSON and call it a day. But this is going to be a spammer/security nightmare. We need some authentication. But in this case we need to authenticate the sender, which is the service sending the notification and not the user who granted the permission. This simplifies things a bit and more accurately represents who the responsible parties are. (Note: For you WS-* heads, this is explicitly not an ’act as’ scenario)

For services with long standing relationship HMAC’s can be used to validate the POSTs by pre-arranging a shared secret. For ad-hoc scenarios I suspect we’ll want to use digital signatures and leverage the previous discovery mechanism to find the public key for a service. That way when a service receives a message it can validate the signature by checking the source service’s public key.

Another check against spamming/security attacks is to make sure to only process requests or permission notifications who are identified as coming from people the user ’trusts’, typically people in their address book. Yes, this leaves open the question of how to handle requests that aren’t from folks in the address book but most social networking systems already deal with this by asking users if they wish to receive requests or permissions from people they don’t have listed.

3.4 Utilize the permission

In the case of a permission notice the service receiving the notice will need the right information to use the permission on the user’s behalf. This is a classic OAuth use case so my guess is that we should just use OAuth access tokens in the permission notice and use the OAuth access pattern to exchange the access token for a request token (or whatever they are called this week) which is then passed into the actual service.

3.5 Describe the permission

This leaves the question (which I moved out of order because I actually think it’s the least interesting question) of what the actual permission notification/request looks like. My guess is that we should, again, rely on OAuth. It has a very simple format that can be easily adapted for the purposes of describing anything and it provides mechanisms for sending requests either over SSL or without. Again, no need to get fancy. The whole thing then becomes just another OAuth profile.

4 Conclusion

I think we have all the pieces to create a really powerful ad-hoc sharing infrastructure that can allow a user of any random kind of service to ask for or grant permissions to any user of any other service. This is what a real, open, social web is all about. So let’s get cracking!

In that case partition tolerance (which, as I explain below, ends up meaning tolerance of machine failure) is a requirement. So in designing frameworks for the data centers I work with the CAP theorem makes me choose between exactly two choices - do I want consistency or availability?

My belief is that for the vast majority of developers, at least for the immediate future, they need to choose consistency.

Um... what's the CAP theorem?

The CAP theorem is explained and proven in this theorem paper which is reasonably approachable. There are plenty of articles on-line that summarize the CAP theorem so I'm not going to write another one. I do however want to point out that the terms used in the CAP theorem, consistency, availability and partition tolerance don't mean what the plain meaning of the words imply.

Consistency is closest to its normal meaning but as the theorem paper points out it might be easier to think of this as meaning atomic and consistent in the AC part of ACID sense.

Availability, as I explore later, doesn't mean availability of the entire system but rather availability of a particular piece of information to be read or written to.

Partition tolerance is a tiny bit tricky. It's plain meaning, e.g. dealing with what happens when machines can't talk to each other, is part of the definition. But it also encompasses what happens if a machine fails. After all, if machine A is trying to talk to machine B and machine A isn't getting a response it's irrelevant if the response didn't come because machine B failed or because there is a network partition. The message didn't get where it was supposed to, therefore the communication failure, from the perspective of the CAP theorem, is modeled as a network partition.

The CAP theorem says of the previous three system qualities, consistency, availability and partition tolerance, we only get to choose two. (Wait, did I just summarize the CAP theorem? D'oh!) Therefore when designing distributed systems I have three choices, consistent/available, consistent/partition tolerant or available/partition tolerant. I explore all three choice below.

Consistent and Available - Not an option

In an ideal world I would like all my service's data to be consistent and available. But CAP says I only get that if I'm willing to essentially fail if there is a network partition and as previously discussed a network partition also includes machine failure.

And to be fair the consistent/available option is actually pretty common. Anyone who is running a single box that hosts their database is choosing this option. So long as the box is up their data is consistent and available but if it (or its network tap) goes down then that's that until the box gets fixed.

But as I mentioned above I come into this situation with a dependency on data centers filled with commodity machines that tend to fail on a pretty frequent basis. So wishing away machine failures (or even network failures which, although rarer, do happen not infrequently) is a non-starter.

So I have no choice, whatever design I use, it must be partition tolerant (read: keep working in the face of machine failure). So choosing consistency and availability over partition tolerance isn't a choice available to me.

Consistent and Partition Tolerant - Easy to program to

I'm primarily in the development platform business. I build platforms that other people use to build their software. So I spend a lot of time worrying about abstractions that my customers can easily understand and live with. The model most programmers are most familiar with is one in which the world is 'consistent'. By which I mean that when one wants to change system state one can do so and either all the changes happen or they don't. Furthermore when someone comes along to read values they will see the changes that have been made. This is a world that is pretty easy to reason about.

But if I want to offer consistency and if, as I have previously argued, I must have partition tolerance, then CAP says I have to give up availability. Which might seem nuts. Who the heck wants a system that isn't available? But remember, availability is not about the global state of the system, it's about pieces of state that have to be mutually consistent.

Imagine you are building a website for your car rental company. You want to host the website in the cloud to save money and reduce time to market. You come to me looking for storage infrastructure and I say "Hey, look, 99.99% of the time when your customer comes to the website they will be able to access their data, put in an order for a rental car, see what cars they have rented, etc. but 0.01% of the time the customer request will fail and btw, typically that failure will resolve itself within a minute or two."

To most businesses this is a fine trade off. Programming and maintaining programs that expect consistency is an order of magnitude less work than dealing with the lack of consistency (see the next section). So a small number of failures that are quickly resolved is probably an acceptable trade off.

Available and Partition Tolerant - Takes a licking and keeps on ticking

Still, some companies, most famously Amazon's Dynamo, take availability very seriously. They don't ever want a customer coming to their website and told 'sorry, we can't help you right now, try again later.' They have done the math and figured out that even rare failures, due to the enormous number of customers Amazon deals with, were costing them real money. So Amazon was willing to deal with the implementation headaches of reducing consistency in return for getting higher availability.

Imagine we are using the previous system which chooses consistency over availability. Let's say that Andres wants to rent a car. He comes to the car rental site. The front end machine tries to access Andres's rental records which are kept on machine Alpha (in reality this would probably be a group of machines using a quorum protocol with an elected master). But Alpha isn't available (i.e. the master has died and the system is in processing of elevating another member of the quorum to master or enough machines have died or been partitioned so that quorum is lost). So the website has no choice but to say "please try again later" until Alpha (or really the quorum) can be brought back online.

Now let's look at a world with a lower level of consistency. Andres comes to the website and the front end machine tries to get to machine Alpha and fails. But rather than sending Andres away the front end machine looks for another back end machine, let's call it Beta, and asks it to handle Andres's rental records. Beta agrees. So Andres goes through the rental process and rents a car. All this information is recorded on machine Beta.

In the meantime it turns out that Beta failed before Alpha came back up so the information that Beta had about Andres is currently offline. Andres navigates back to the website to check on something about his order. The front end machines goes looking for machines who know about Andres and finds Alpha who is now back up. Much to his surprise Andres is now shown that he has no rental order! After all, Beta never told Alpha about the order and Beta is currently down. We have a data inconsistency.

Andres, frustrated by this, puts in a second order and leaves. Meanwhile Beta comes back up and finds Alpha. Now there is some confusion. Both Beta and Alpha have orders from Andres for a car. Did Andres mean to rent two cars? Is the newer order a replacement for the older order? What should the system do?

All of these problems are solvable. It just takes very careful thought about all the possible failure states and code that can identify and resolve those failure states. The process of taking inconsistent data and making it consistent over time as failed systems come back on-line and share what they know is called 'eventual consistency'.

Eventual consistency is an incredibly powerful mechanism for making services more resilient but it isn't free. Modeling and dealing with the potential problems are non-trivial. Much like the inappropriate optimism around optimistic concurrency I suspect that in practice most implementers would do well to stay away from eventual consistency frameworks. At least until they can be reduced to well understood design patterns (a la Amazon's shopping cart example).

There doesn’t appear to be a decent solution for this problem on the mac. VNC is a joke. It will just take my 24 inch iMac screen and shrink it down to the laptop’s screen size. And yes I have played around with smart zoom but it’s really painful.

Isn’t there an equivalent for the mac to Microsoft’s outstanding Remote Desktop Connection application and RDP protocol?

For what it’s worth I signed up to be notified when AquaConnect releases their mac remote desktop product which is based on RDP but they aren’t even announcing dates.

Any ideas or am I just out of luck?

In fact what I specifically need to do is:

1. Lobby the Windows Azure Table Storage team to add undelete for tables so if I accidentally blow away one of my tables I have some hope (oh and ACL’s would be nice too)
2. Be very careful about how I update my schemas
3. Implement a command journal (and be clear about their limitations)
4. If time permits implement tombstoning
5. If I’m feeling really wacko implement my own versioning system on top of the table store (or just backups if I’m feeling only slightly wacko)
6. Put into place a realistic plan to take advantage of all these features while keeping in mind the limitations of these techniques.

The links in the previous text are to the other articles in this series that I wrote for my blog. Those articles are:

• Do I need to backup/journal my Windows Azure Table Store?
• The Limits of Command Journals
• Techniques to Ease Recovering from Self Inflicted Data Corruption
• Thoughts on implementing a command journal
• Tombstoning on top of Windows Azure Table Store
• The limits of recovering from application logic failures
• Implementing Versioning in Windows Azure Table Store

In a previous article I argued that I needed some kind of journaling/backup for my Windows Azure Tables in order to handle my own screw ups. In this article I re-examine the value of versioning for recovering from self inflicted data corruption. Discuss backups as a possible substitute for versioning. Look at what versioning might look like if added as a native feature of Windows Azure Table Store and finish up by proposing a design that would let me implement versioning on top of Windows Azure Table Store.

This article is part of a series. Click here to see summary and complete list of articles in the series.

Contents

1 The value of versioning

The value of versioning in recovering from application errors has already been covered here and here. But to summarize - when it hits the fan versioning can help one figure out if the original damage has been compounded by subsequent changes. Furthermore version, by providing the outcome of a command let’s one examine what happened in the past with less baggage than needed to understand the past then the ’replay’ issues that plague command journals.

Versioning is also useful as a last ditch ’go back in time’ mechanism where if the damage is just too great to repair at least the system can provide the option of turning back the clock to some better state. Although one shouldn’t overstate the utility of this feature. In non-trivial cases there will be a variety of side effects of ’turning back the clock’ that will be hard to control and the clock can’t go too far back or issues with schema changes, functionality changes, etc. come into play. Many of the same issues with replaying command journals apply to using versioning as an emergency escape hatch to the past.

So while versioning is useful, I suspect that command journals and tombstones in the average case probably provide the most bang for the buck. My real hope is that systems like Windows Azure Table Store will offer versioning as a feature so the cost and complexity of taking advantage of versioning will go way down.

2 What about backups?

As discussed below implementing versioning on top of the Windows Azure Table Store, while not brain surgery, isn’t trivial either. A much simpler technique would be to regularly backup tables. This can be done in the background without having to interfere with normal operations. So it’s less risky.

Backups work using snapshots. At regular intervals the table is read in (typically with a filter that ignores values that haven’t changed since the last snapshot) and a snapshot created. Unfortunately snapshots miss things. If a value is changed multiple times between snapshots then the intermediate values will not be recorded.

This leads to situations where if a buggy command is given between snapshots and then the buggy value is overwritten just before the snapshot I have no way of knowing what the original value was unless I can replay the command (which is tricky and assumes that the value produced by the bug is predictable). This makes it more or less impossible to handle the put syndrome since I can’t see if the same value was written twice or a new value written.

For similar reasons backups are not useful when dealing with the etag syndrome since it’s at best just luck if the snapshot happens to have captured the correct system state at the time the command was executed.

Also backups don’t deal at all well with deletes. Unless one copies the entire table during every snapshot (a rather expensive proposition) then any deleted records will be missed. So if one is going to implement delta based snapshots (e.g. just copying things changed since last snapshot) then one also needs to implement tombstones and backup the tombstone table as well.

If a transaction is in progress during a snapshot then only the parts of the transaction that occurred before the snapshot will be captured, those coming after will be missed until the next snapshot. So restoring from the most recent snapshot means restoring the system to an inconsistent state. While inconsistency happens anyway in loosely coupled systems its one thing for a user to issue a command that fails in a bad way, something the user is generally told about. It’s another thing for the system at some point to just ’shift state’ to some previous, inconsistent, point and users are then told to pick up the mess.

Still, for all of that, at least backups offer some hope of turning back the clock in the case of hopeless data corruption so perhaps they do have some value.

3 Imagining a versioned Windows Azure Table Store

Versioning tends to come in two flavors, linear and non-linear. My belief is that Windows Azure Table Store only needs linear versioning. My reasoning is that if one looks at the table store using the lens of the CAP Theorem then one notices that Windows Azure Table Store focuses on availability and consistency. If one is willing to give up partition tolerance (which Azure Table Store has) then most of the use cases for non-linear versioning go away. It is possible in a consistent system to enforce an order, even without locking, thanks to optimistic concurrency which the table store supports.

So if the table store supported linear versioning then the experience would be that every write would cause a new version of a particular row to come into existence, I’ll call that the tip version.

All existing store commands would work exactly as they do now but would only apply to the tip version and in the case of POST and PUT would create new tip versions. The delete command would create a tombstone entry stating that the row was deleted. The tombstone entry would be invisible to all the existing Windows Azure Table commands.

I don’t think that check-in/check-out semantics are appropriate to a highly distributed system like the table store so the commands available to a versioning aware client would actually be quite limited. I would add a way to specify a version in the URL of a row (say with a query parameter) as well as a query to include versions in the output of a table query. Finally I would support the ability to destroy (as in delete without trace) a row. I don’t know that much more than that is really needed in terms of interacting with older versions.

4 In place versioning on top of the table store

[Disclaimer: The following is more of a mental exercise. I haven’t had time to actually mock this up and make sure all the details are right.]

Right now Windows Azure Table Store doesn’t offering versioning so I’ve given some thought to how I might implement versioning myself on top of the table store. The services I work on that use the table store tend to be high read and low write. So I want an approach to adding versioning to the table store that places the cost of versioning more on writes than reads. I also want an approach that is more or less guaranteed to produce consistent output. That is, I don’t want to end up in a situation where the state of my production tables and my version history are out of whack. The whole point of introducing versioning is that it’s correct and complete so I can reason about certain things that would otherwise be hard to do. If I can’t get consistency in my version store I might as well use backups which at least are simpler to implement. Thankfully the table store provides the features to meet all of my requirements including consistency.

The approach I would use is in place versioning. That is most current version of a row (referred to as the tip) and its previous versions all live in the same partition in the same table. This is the opposite of the approach I used with tombstones because in the case of tombstones consistency wasn’t a problem.

In the in place versioning approach the tip version of any row will have whatever partition key/row key it is supposed to have plus the prefix ”tip” on the row key. This means that anytime I want to interact with the tip version of a row I just generate the expected partition key/row key and add in ”tip” as a prefix on the row key. This makes reads fast.

Every column I’m versioning will contain a version ID which is a monotonically increasing integer. The first time I create the ”tip” version of a row (e.g. when the row is first created) I will give it the version number 0. When updating a row I will copy the old value and give it the prefix ”old”. Then I’ll update the tip version and increment its version number. The key to consistency with an in place versioning approach is that it’s possible to both create the old version and update the tip atomically. The table store’s entity group transaction mechanism is guaranteed to be atomic and so can be used to solve exactly this problem.

To version enable a table store table I need either to build a proxy or a library. My guess is that I would use a library to save the processing and network time of a proxy but what is really nice about a proxy is that I can use the proxy as a lock down mechanism. I can make sure nobody but the proxy has the key to the table so if someone doesn’t go through the proxy then they don’t get access to the data. That alone will prevent tons of bugs. By having a single proxy I can also more easily control issues like versioning of the proxy code which deals with a whole other set of bugs. But proxies do demand both a processing and latency cost so I have to consider that in deciding between proxies and libraries.

The following goes through the standard methods in their non-version aware form and explains how their behavior would change if one was using a version aware library/proxy to interact with the table store using an in place versioning approach.

GET

If the query contains a filter that specifies a rowkey then prefix the rowkey value(s) with ”tip”. In all cases add the filter argument (if it doesn’t already exist) of ”rowkey gt ’old”’. This will filter out everything but tip versions of rows since ’tip’ comes after ’old’.

DELETE

A GET is needed to retrieve the current ’tip’ version. If none exists then the request should fail since there is nothing to delete. If the tip version does exist then create an entity group transaction that includes creating a new row to act as the tombstone with a column ’tombstone’ set to true as well as a delete command for the current tip that includes the etag from the GET in an if-match header.

PUT

First retrieve the existing versions ’tip’ (using an etag if one was provided in an if-match or equivalent header). If there isn’t one then the resource doesn’t exist or has been deleted and so the request should fail. If the ’tip’ version exists then an entity group transaction is needed to update the tip version as previously described but use if-match with the etag retrieved from the original GET.

MERGE

The logic is the same as PUT for all intents and purposes. It’s just that values not specified in the MERGE request have to be retrieved from the soon to be replaced ’tip’ version to create the ’old’ prefixed copy.

POST

Check to see if a tip version exists. If so, then fail. If not then check to see if there is a tombstone. If so then issue the POST request with the version number set to an increment of the number in the tombstone. If there is no tombstone then the version number is 0 and row key will have ’tip’ added as a prefix.

Entity group transaction

In essence just glue together the instructions for the individual methods mentioned above and apply to the contents of the entity group transaction. Entity group transactions even support if-match headers.