APP and Dare, the sitting duck
Friday June 15th 2007, 12:00 am
Filed under: SOA/Web/Etc.

So poor Dare made the apparently unforgivable mistake of questioning anything about APP. First he suggested that maybe APP doesn't solve all the world's problems. Then he clarified that GData isn't APP. And then after a particularly appalling article by Mr. Bray that was so rude that I refuse to contribute to its popularity by linking to it, Date finally tried to explain that Microsoft isn't trying to destroy APP. I'm going to ignore all the heat because I suspect my handful of readers have already read Dare's articles and the various responses. Instead I'll try to explain what's actually going on at Live. I know what's going on because my job for little over the last year has been to work with Live groups designing our platform strategy. So I know where the bodies are buried and in many cases helped to bury them.


Live's goals in the protocol space

Most of the services in Live land follow a very similar design pattern, what I originally called S3C which stood for Structured data, with some kind of Schema (in the general sense, I don't mean XML Schema), with some kind of Search and usually manipulated with operations that look rather CRUD like. So it seemed fairly natural to figure out how to unify access to those services with a single protocol. The first place we went was APP. This was a business decision. Live is in the services business, not the protocol or development tool business. I am not paid to give a damn what browser someone talks to Live with, what language our partners/customers develop their software in, what operating system they run, etc. The only thing I am paid to care about is that we get as many people as possible writing as much software as possible that interacts with Live services.

So for us the whole protocol issue is just a barrier to entry. We don't make any money from that barrier. And no, we don't care about lock in. At least the people who give me orders (hey George!) have a good understanding that the days of lock in are long over. The future clearly belongs to connecting services together. That is, nobody is going to have all of their data, services, etc. at Live. It won't happen. We could be the absolute best at everything we do and we still won't own all of a user's data and services. So for us to succeed we have to convince users to keep some of their data/services with us and then make it brain dead easy to connect the data/services they keep with us to all the other data/services they keep in lots of other places.

In other words, it's all about interoperability and the easier it is to interoperate the more successful we will be. So in our dream world there would exist a protocol that would meet the S3C pattern. A popular protocol. A widely support supported. A protocol we could just adopt and get on to the business that can make us money – building cool services.

So with this in mind we first went to APP. It's the hottest thing around. Yahoo, Google, etc. everyone loves it. And as Dare pointed out in his last article Microsoft has adopted it and will continue to adopt it where it makes sense. There was only one problem – we couldn't make APP work in any sane way for our scenarios. In fact, after looking around for a bit, we couldn't find any protocol that really did what we needed.

Because my boss hated the name S3C we renamed the spec Web3S and that's the name we published it under. The very first section of the spec explains our requirements. I also published a FAQ that explains the design rationale for Web3S. And sure enough, the very first question, 2.1, explains why we didn't use ATOM.

Below I try to respond to some of the points made about Dare's article (at least the polite ones).

Hierarchy

The general conclusion of the anti-Darians is to use 'gloving'. The idea that you put a link in the ATOM feed to the actual object. This isn't a bad idea if the goal was to publish notices about data. E.g. if I wanted to have a feed that published information about changes to my address book then having a link to the actual address book data in the APP entries is fine and dandy. But if the goal is to directly manipulate the address book's contents then having to first download the feed, pull out the URLs for the entries and then retrieve each and every one of those URLs in separate requests in order to pull together all the address book data is unacceptable from both an implementation simplicity and performance perspective. We need a way where by someone can get all the content in the address book at once. Also, each of our contacts, for example, are actually quite large trees. So the problem recurses. We need a way to get all the data in one contact at a go without having to necessarily pull down the entire address book. At the next level we need a way to get all the phone numbers for a single contact without having to download the entire contact and so on. What this all boils down to is that we need a protocol that natively understands and can interact with the hierarchy of our data. That isn't what APP does. That doesn't mean APP is wrong or bad. It just means that APP isn't optimized to provide access at any arbitrary point in a tree of data.

Merging

Use optimistic concurrency

Replacement based semantics are a problem because of versioning issues. We are constantly changing our schemas in backwards compatible ways so we need clients to know how to deal with data they don't recognize.

APP's approach to this problem is to have the client download all the content, change the stuff they understand and then upload all the content including stuff they don't understand. I believe that one of the most important innovations in HTTP was explicitly distinguishing between safe and unsafe methods. GET is safe, one can execute GETs against any server for any reasons and not be held responsible for any side effects. PUT, on the other hand, is unsafe. The consequences of a PUT is the caller's direct, personal responsibility. But that responsibility goes out the door as soon as one introduces a paradigm where clients are expected to blindly upload content they don't understand. In that case there is no responsibility. I believe this approach goes too far in bluring the distinction between safe and unsafe methods.

On a practical level though the 'download then upload what you don't understand' approach is complicated. To make it work at all one has to use optimistic concurrency. For example, let's say I just want to change the first name of a contact and I want to use last update wins semantics. E.g. I don't want to use optimistic concurrency. But when I download the contact I get a first name and a last name. I don't care about the last name. I just want to change the first name. But since I don't have merge semantics I am forced to upload the entire record including both first name and last name. If someone changed the last name on the contact after I downloaded but before I uploaded I don't want to lose that change since I only want to change the first name. So I am forced to get an etag and then do an if-match and if the if-match fails then I have to download again and try again with a new etag. Besides creating race conditions I have to take on a whole bunch of extra complexity when all I wanted in the first place was just to do a 'last update wins' update of the first name. In a merge scenario the previous is trivial. I would just upload the first name.

Use PATCH

A number of folks seem to agree that merge makes sense but they suggested that instead of using PUT we should use PATCH. Currently we use PUT with a specific content type (application/Web3S+xml). If you execute a PUT against a Web3S resources with that specific content-type then we will interpret the content using merge semantics. In other words by default PUT has replacement semantics unless you use our specific content-type on a Web3S resource. Should we use PATCH? I don't think so but I'm flexible on the topic. If Web3S proves to matter then it will inevitably end up at a standards body and if that standards body says use PATCH not PUT then we'll use PATCH, not PUT.

Why not just modify APP?

We considered this option but the changes needed to make APP work for our scenarios were so fundamental that it wasn't clear if the resulting protocol would still be APP. The core of ATOM is the feed/entry model. But that model is what causes us our problems. If we change the data model are we still dealing with the same protocol? I also have to admit that I was deathly afraid of the political implications of Microsoft messing around with APP. I suspect Mr. Bray's comments would be taken as a pleasant walk in the park compared to the kind of pummeling Microsoft would receive if it touched one hair on APP's head.

Conclusion

Inventing our own protocol wasn't an easy choice. It raised the very barrier to entry we are trying to avoid. But we couldn't figure out how to use APP without putting an unacceptable implementation and performance burden on both our customers and ourselves. So we felt we had no other option. But Web3S is a tactical not a strategic choice. If it turns out there is a better option then we'll adopt it.



23 Comments so far

I believe Bill’s response and my own cover the technical issues you raise above.

What I am more interested in hearing about is why you didn’t approach the WG with these issues when you ran into them. The spec was obviously still in the works when you were working on WebS3 and if you believed you had found real weaknesses with APP, which the ensuing discussion has shown that you didn’t, then why not bring attention to them before we shipped?

Comment by Joe Gregorio 06.16.07 @ 3:45 pm

I have read both your blog article on the topic as well as Bill’s in addition to a number of others and in the article above I summarized what I understood to be the proposed solutions to our problems and explained why I didn’t feel those proposals were sufficient. I’m not sure what more I can add.

As for approaching the TC, I actually did talk with folks in the APP community and they gave answers much like those we received as part of the responses to Dare’s post. As I explain above we do not believe those answers actually solved our problems. Nevertheless we should have gone to the TC. This is a mistake I will remedy forthwith.

BTW, a critical point to keep in mind in this discussion is that Web3S hasn’t ‘shipped’. There does not exist a Web3S service anywhere in the world that I’m aware of. The point of making the Web3S specs available before we have an implementation is to say – PLEASE GIVE US FEEDBACK!

Also, as I repeatedly explain both to folks inside of Microsoft and to folks outside of Microsoft “There cannot be only one”. Meaning that Live as a whole is not going to have some universal protocol to solve all of its problems. Each Live service will have to evaluate its own community and choose one or more protocols depending on who it is targeting. Internet developers are not a homogeneous mass so we must be able to segment and target the right protocols to the right communities. So regardless of what happens with Web3S we will adopt any and all protocols that will get us a sufficiently large audience. That’s why we spent so much time trying to figure out how to make APP work as we felt it would give us a ready made audience.

But in the end we failed to figure out how to adapt APP to our needs without changing it so fundamentally that it would no longer be APP. And equally unfortunately none of the suggestions made so far have, in our opinion, remedied those failures.

Hopefully a discussion in the TC will lead us either to understand why we are wrong in our analysis or perhaps the TC will decide that, in fact, APP doesn’t meet our needs and perhaps decide to discuss what changes could be introduced to make it a better fit. But in either case the first stop is the TC.

Comment by Administrator 06.16.07 @ 9:03 pm

Good for you for sticking your neck out there, Yaron!

Comment by M. David Peterson 06.17.07 @ 1:21 pm

“”"As for approaching the TC…”"”

What’s a TC?

Comment by Joe Gregorio 06.17.07 @ 2:07 pm

Ack… sorry. TC stands for Technical Committee. I spent the last few years hanging out in OASIS where what the IETF calls a working group is called a TC. I apologize for the nomenclature screw up.

Comment by Administrator 06.17.07 @ 3:48 pm

[...] Yaron posted more helpful clarification about the REST stuff that Microsoft Live is working on. Definitely an [...]

Pingback by snellspace.com » Blog Archive » Still Silly 06.17.07 @ 7:38 pm

“”"I apologize for the nomenclature screw up.”"”

Ah, that’s OK, and it’s better than the alternative explanation: which is that you didn’t even know which venue the APP was being standardized under.

Comment by Joe Gregorio 06.18.07 @ 6:04 am

Little chance of me not knowing where APP is being standardized. I spent way too many years in the IETF for that.

Comment by Administrator 06.18.07 @ 7:53 am

Yaron,

In my linked blog entry I try to understand how the hierarchies in Web3S work relative to graphs. I’m confused by the structural identity of an EII (path based), instead of canonical ID or URI based.

Is it accurate that the articles/article/author example that Dare posted wouldn’t maintain author identify if the same author published more than one article?

In my post I quotes pieces of the FAQ that I drew that conclusion from, please explain that I’m wrong.

Thanks,
John Heintz

Comment by John D. Heintz 06.18.07 @ 4:00 pm

Thanks for the background, and I’m happy to see another substantial REST service proposed. I would like to see a discussion within the WG about your issues; they are both generic and important. I think there are some straightforward answers to them (content can contain trees, and PATCH exists). I am genuinely interested in learning about why these wouldn’t meet your needs, if indeed they wouldn’t, or if you felt that doing so would simply make your cost/benefit ratio untenable.

Comment by John Panzer 06.18.07 @ 5:03 pm

[...] on Microsoft Live as Google and others have been doing for the last couple of years. The group according to Yaron Goland from Microsoft required a protocol for structured data and search and looked to ATOM Publishing [...]

Pingback by » Microsoft needs REST | Newton’s Theory | ZDNet.com 06.19.07 @ 5:49 am

John (Heintz), I have your article on my list of blog articles to give meaningful responses to. I first need to finish the series of posts I’m writing for the APP mailing list and then an internal work assignment I have to finish and then I’ll be writing responses to your blog, Tim’s, etc. Sorry for the delay but your article asks some good questions and I want to give good answers.

Comment by Administrator 06.19.07 @ 8:50 pm

John (Panzer) as I mentioned in my response to John Heintz I am neck deep in writing a series of mails to the APP mailing list that explain in gory detail exactly what the issues are and why we didn’t think we could resolve them. I hope to have those mails sent out tomorrow.

Comment by Administrator 06.19.07 @ 8:53 pm

Yaron — Looking forward to seeing your emails on atom-protocol. I feel your pain w.r.t. time crunches :).

Comment by John Panzer 06.20.07 @ 5:57 am

[...] Windows Live.” There was no mention of UPDATE. However, the spec was published in order to solicit feedback, and I’ll be interested to see what comes of [...]

Pingback by RESTful Live Contacts for Internet-scale social networking « Jon Udell 06.21.07 @ 5:54 am

Yaron, thanks for responding to my comments (see Yaron’s comment here).

I’m still thinking about your words and looking through the spec.

Also, you may want to package that up as a post to your own blog (so more people will see it).

Comment by John D. Heintz 06.27.07 @ 9:26 am

Yaron,
My latest blog post is a response to your comments.

Here is my summary:
Web3S doesn’t naturally support RESTful primary resource identifiers for EIIs. It can, but only with optional elements and server-specific guarantees about identifiers. A generic Web3S client could never “know” what canonical and primary resources were identified.

Thanks for listening, John.

Comment by John D. Heintz 06.28.07 @ 12:21 pm

Two comments are here :

http://sberyozkin.blogspot.com/2007/06/some-comments-on-web3s.html

I like the idea about merge and somewhat confused about the utility of exposing the fine-grained resources like “lastname”

Comment by Sergey Beryozkin 07.03.07 @ 2:38 am

Sergey – Allow anonymous postings! I had found your blog post earlier and was trying to respond but you don’t allow people to just enter their name and e-mail address.

In answer to your question allowing SIIs to be addressed is useless in the case you identified which is why my current position is that the ‘core’ Web3S spec should not define a way to address SIIs.

But as an optional feature being able to address SIIs is useful because it allows you to use a more appropriate content-type. For example, if someone wants to store an XML document as a SII they have to do it by embedding it into another XML document. This typically involves nastiness like CDATA although we will be providing a better solution. But if you can address the SII directly then you can just upload the XML doc directly as an Application/XML file.

We are also likely to add binary support along side SII support so again, direct addressing means you can pull down a JPEG file as a binary stream using an SII address instead of a base 64 encoded stream in a XML or JSON file.

Comment by Administrator 07.03.07 @ 9:00 pm

Thanks, I fixed it, doing first blogging steps…
Updating contained SIIs directly does look like a neat idea. In context of updates it’s clearer why it makes sense to model them as individual resources, as the scoping is clear from their URIs…Can contained SIIs be characterized as write-only resources ?

Comment by Sergey Beryozkin 07.05.07 @ 4:26 am

Sergey – but wouldn’t you want to be able to get them directly? For example, if I do an update to a SII that makes the SII into a JPEG then wouldn’t I later want to be able to do a read and get back an Image/JPEG and not some XML encoded nightmare?

An implementer can also use the deferred content feature to make sure that the XML encoded glop doesn’t show up in normal requests.

Comment by Administrator 07.10.07 @ 8:39 pm

So, is this blog entry still relevant given the announcements on Feb 28th, 2008?

i.e.: “Microsoft is making a large investment in unifying our developer platform protocols for services on the open, standards-based Atom format (RFC 4287) and the Atom Publishing Protocol (RFC 5023).”
(http://dev.live.com/blogs/devlive/archive/2008/02/27/213.aspx)

Has Web3S disappeared? An appropriate blog post would be much appreciated Yaron.

cheers
Jamie

Comment by Jamie T 02.28.08 @ 1:17 pm

[...] far as when an influential team within Microsoft were probably going to reject AtomPub and instead use their own proprietary alternative) and back in March 2008 I noted with interest an announcement by George Moore on the Live Services [...]

Pingback by SSIS Junkie : OData gunning for ubiquity across Microsoft products 12.21.09 @ 10:51 am



Leave a comment
Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>



7 − = four