[5/26/2009 - Updated with Q&A on open redirectors]

[6/2/2009 - Updated with a note from Allen Tom on another way to prevent open redirector attacks]

To set the stage for the conversation it's useful to first review the newly proposed three legged OAuth message exchange pattern. The picture below is based on OAuth Core 1.0 Rev A (Draft 3):

There are several things that concern me about the current OAuth pattern:

State – This pattern forces both the consumer and service provider to store someone else's state for a potentially non-trivial period of time. That is always a messy proposition at best. The service provider is forced to associate a call back with a request token and the consumer is forced to associate a verifier with a request token.

Performance – The service provider better implement messages 2/3 really quickly because the user is going to have to swallow all the latency of messages 1-4 before anything useful happens.

Complexity – Having 2 extra round trips means extra code, extra testing and extra chances for things to go wrong.

What if we instead use a pattern like:

Now the only long term secret is the access token and only the consumer has to record it. The request/response pairs go down to four and the performance issue with requesting the request token is gone.

I suspect there is a good reason that OAuth felt it had to have the request token but I honestly am not clear what that reason is and if its important enough to justify the complications it introduces. Hopefully the article will set me on the path to finding out.

Q&A

Isn't the newly proposed pattern susceptible to phishing?

Absolutely. The short answer is that the consumer will need to associate a value with the callback URL that lets the consumer make sure that the user coming back on the call back URL with an authorization token is the same user the consumer sent out. This can be trivially handled via random numbers or encrypted ID. See below for details.

The attack is that a black hat user gets to message 2 (the redirect) in the previous chart and then doesn't actually execute the redirect but instead takes the call back URL and publishes it on a web page. The black hat convinces a white hat user to click on the link in a context where the white hat user expects to grant permission to the service provider. The white hat user grants permission and is redirected back to the call back URL specified by the consumer originally for the black hat user with the access token.

In theory the attack would be if the consumer somehow got confused and thought that the access token returned by the white hat user belonged to the black hat instead because the callback URL that was used was the one that the consumer had originally issued to the black hat.

It might seem trivial to prevent this attack – don't put any user identifying information into the callback. This means that when the callback arrives (e.g. message 7 in the picture above) the consumer will validate who the user is by whatever login method (if any) is being used and not via any values in the callback URL. This prevents the phishing attack from working.

Unfortunately this approach also makes an even more insidious attack possible. Let's assume the consumer doesn't put any identifying information into its callback URL. Let's further assume that the black hat goes through the entire permissioning process but swallows message 6, this is the redirect with the access token.

Let's further assume that the black hat fools the white hat into clicking on a link that sends the black hat's access token to the service provider. Assuming the white hat is logged in the black hat will think the access token belongs to the white hat user instead of to the black hat (remember, the redirect URL has no user identifying information so there is no way for the consumer to know who the token was issued for). Now when the consumer wants to interact with the service provider on behalf of the white hat they will actually use the black hat's access token.

To see how bad this could be imagine that the consumer is a service like Plaxo that handles address book synchronization and the service provider is some service with an address book like Live. In that case when the consumer wants to synch the white hat's address book with Live they will actually push all the address book information into the account the black hat set up and got the access token for. So much for the white hat user's private address book information.

To prevent this the consumer has to be able to associate an access token from the service provider with the user that the consumer had sent to get the access token in the first place.

There are a couple of ways the attack can be prevented. Probably the easiest is for the consumer to generate a cryptographically secure random number (can you say – verifier?) and stick it as a cookie on the user's browser and then stick that number into the callback URL. When the callback arrives the consumer compares the ID in the cookie with the ID in the callback and if they don't match then the access token is rejected.

If cookies aren't available (and that situation is getting increasingly less interesting) then the consumer can take the user's ID (this only works if the user has authenticated to the consumer) and encrypt it and put it on the callback URL. Then when the call back comes in the consumer checks the encrypted ID with the currently user ID and if they don't match rejects the request.

It is also possible to make things work in the case that the user is anonymous to the consumer and doesn't support cookies but that will require the user to manually copy and paste the authentication token back to the consumer.

Can't someone falsely pretend to be a consumer?

Sort of, but it won't help them.

The attack is that a black hat sets up a phishing site called blackhat.contoso.com that looks like whitehat.example.com. blackhat.contoso.com then fools a user into thinking its whitehat.example.com and redirects the user over to the service provider. What happens next depends on the relationship between the service provider and its consumers.

Let's take the most open case, the service provider allows a consumer to ask for user permission without requiring pre-registration. I actually was the PM on a project that shipped a system like this for Live and it worked fine. In that case the service provider will need to prominently display the consumer's DNS name from the callback to the user. So the user will see 'blackhat.contoso.com' and they thought they were coming from 'whitehat.example.com' and so the user will know to reject the request.

But let's say the blackhat gets fancy and instead makes the DNS in the redirect URL point to whitehat.example.com? In that case the user will do what they intended to do, give permission to whitehat, and the access token will be sent to whitehat's domain as part of the redirect. So blackhat got nothing for their efforts (other than maybe a thank you from whitehat).

Isn't this design susceptible to open redirector attacks?

The short answer is – yes.

The attack goes like this. Evil.com phishes the user into thinking that evil.com is actually stupid.com. Evil.com, in message 3 in the second picture above, then redirects the user to the service provider with a callback URL of the form: http://stupid.com/redirector?redirect=http://evil.com/dobadstuff.

As I pointed out in the previous question this attack is generally useless because the response will go to stupid.com, not evil.com. Unless, of course, stupid.com is criminally stupid. In which case the previous URL points to what's known as an open redirector. This is a web page on stupid.com's domain that accepts a URL as a query argument and will blindly redirect the user to that URL regardless of where it is.

There are at least two ways to secure the protocol as described above against open redirector attacks.

Require SSL callback URLs – Generally speaking open redirectors don't run over SSL so if the SP requires SSL callback URLs then the SP will have protected its users against most (but certainly not all) incompetent websites.

Require an out of band secret – The core of the attack is that evil.com, thanks to the open redirector, gets a copy of the authorization token which is what is known as a 'bearer token'. That is, mere possession of the token is proof that one has the right to use the token. So another fix is to require that the authorization token cannot be used on its own. Instead the SP and the consumer have to provision a shared secret out of band. In the simplest possible case (for the consumer) message 9 in the second picture requires that the shared secret be included on the request. Because evil.com won't have the secret that was provisioned for stupid.com any attempt to use the authentication token will fail. An alternative approach (this is preferable from a security perspective) is to sign the redirect in message 3 with the shared secret.

Perform Discovery – (Thanks to Allen Tom for pointing this one out) The SP can require that consumers provide a way to discover what addresses a consumer uses for callbacks. For example, the consumer, in message 3, could pass in a URL that points to the same domain as the callback URL. The SP would then do a GET on the passed in URL and get back a file that lists the callback locations the consumer supports. Obviously the SP wouldn't support redirects on that GET. Equally presumably the consumer wouldn't list in that location evil.com's address. So in theory that's that.

This all having been said I do not believe we should do anything about open redirectors. There are good reasons to want to run redirectors but it's been known for many years now that without canaries or some other form of protection these redirectors are very insecure. Running an open redirector without cross site attack protections is outrageously irresponsible. And I just don't believe we should design protocols against outrageously irresponsible behavior because no protocol can be protected against all the idiotic things an irresponsible person can do.

And, for what it's worth, Live's first delegated authentication system worked exactly as this blog post describes. You can see an overview here. And yes, we were fully aware of the open redirector problem and no, we didn't try to protect against it for exactly the reasons described in this Q&A.

DISCLAIMER: Although I am an architect on .NET Services' Access Control Service nothing said in this document necessarily represents the opinions of my employer, my friends, my enemies or my teddy bears. No warranty express or implied. Your mileage may vary. Do not remove tag.

Claims – I like claims based programming for handling access control. In a claims based system a requester shows up with a request and attached to that request is a set of claims about the requester. The specific type of claim really isn't that important but typical examples include claims that the requester is person X, or is in role Y or is a member of group Z. The service then decides what to allow the requester to do based on those claims.

Validating that the claims submitted to a service are true isn't exactly trivial. Whole protocol suites exist to validate claims about identity, for example. And once identity is validated, figuring out what roles or groups the caller is in typically requires centralized logic. For example, my enterprise service probably doesn't want to figure out if someone is an employee of my company or not. I'd rather leave that determination to someone else and just get informed of the answer. So what I need is a way to let someone else validate a requester's claims and then just have my service informed of the result.

Tickets – So what I would really like is for the requester to present me with a ticket (in the Kerberos sense) that was signed by someone I trust attesting to what claims I should believe about the requester. It's up to the ticket issuer (called a Key Distribution Center in Kerberos, a STS in WS-*, an Identity Provider in SAML and so on) to figure out who the requester is and what claims I should believe about them. The ticket issuer summarizes what it knows about the requester in a signed (and/or encrypted) ticket that it gives the requester who then hands the ticket to my service. My service then validates the ticket (using shared trust with the ticket issuer, not the requester), pulls out the claims and makes its access control decision.

HTTP – The services I'm working with are native to HTTP. I say native to distinguish from "dig a tunnel through HTTP". Lots of protocols out there claim to be based on HTTP but in fact use HTTP as nothing more than a really inefficient implementation of TCP, completely ignoring the HTTP resource model, headers, etc. So I want a protocol that gives me claims and tickets that are native to HTTP. That means I want to see resources and authorization headers.

Less is more- The enemy of interoperability is optionality. The more options a protocol has the lower the probability that two parties using that protocol will successfully interoperate. As we used to say in the protocol world "the spec is done when there is nothing left to cut". So I want the protocol to be embarrassingly simple. If a protocol looks like a Chinese Menu then it isn't for me.

Reach – The services I need to work with are written in every language for every platform so I can't assume that my partners will just be able to grab some library to implement the security protocol. Sure, some components, such as a HTTP stack or hashing/encryption libraries are ubiquitous. But beyond that my partners, for the foreseeable future, will need to write things themselves. So I'm looking for a protocol so trivial that someone could whip together a basic implementation in a few hours.

When I think about the protocol I would like to have I focus on a few things. The claims format should be so trivial that it uses name/value pairs not structured formats like XML/JSON/YAML. I want tickets so small that in practice all the claims and the ticket overhead can trivially fit into a HTTP header. And I want the protocol/encryption process to be so simple that all the claims handling/encryption code can be written in 100 or 200 lines of readable code. I also want a pony.

So what do you think? Is there a protocol that meets my needs today? Is there a protocol that is close and I should just adapt it? Feel free to respond in comments or, if you are going to be at the Internet Identity Workshop 2009A in Mountain View May 18-20th we can talk there!

In fact, two phase commits are incredibly useful for the ultimate in non-ACID transactions – compensating transactions. In an ACID transaction if the attempted transaction fails then everything gets nicely cleaned up, no muss, no fuss. In a compensating transaction, on the other hand, if the action fails then an attempt is made to undo the action but with no ACID semantics. For example, if I book a flight with a travel agency and it turns out they couldn't book the flight they said they would then instead of just canceling the whole ticket (and losing their booking fee) they might rebook me on another flight that I may or may not prefer. The point being, that the 'failed' transaction (booking the agreed flight) is 'compensated' (by buying another flight) rather than rolled back in an ACID sense (by canceling the ticket and refunding the fees).

Now imagine that I want to book a whole trip myself involving a hotel, a rental car, a plane flight, etc. I could use a two phase commit protocol to allow me to first send reservation requests to each of the companies and then if I can't get reservation confirmations from everyone the two phase commit coordinator would automatically trigger a compensate command to each of the companies. Of course the compensation may not be what I like, i.e. the airline may charge me a re-booking fee, but them's the breaks in the real world.

The point is that two phase commit protocols are nothing more or less than a coordination mechanism and what they are coordinating may or may not be an ACID or Atomic transaction.

An example of this situation came up in the BPEL TC today. As I discussed previously the Tibco corporation has decided that the BPEL TC's use of the term rendezvous, even though we explicitly use it in its common English usage, is violating their trademark. To this end the Tibco corporation has demanded that the BPEL group desist in its use of the word rendezvous as the value of an attribute in BPEL. Mind you, what Tibco hasn't even attempted to do in the group is to actually demonstrate that our use is, in any way, a violation of their trademark.

To make matters worse today's discussion of the issue on the group phone call focused on the possibility of Tibco suing people. The assumption of the group was that if we didn't change the term then Tibco would start suing us. An assumption, I would add, that the Tibco representative neither encouraged nor attempted to dispel. So under the cloud of the implicit threat of a lawsuit the group voted. 11 people voted to open the issue and implicitly to change the term. 6 people voted to not open the issue and thus reject Tibco's claim. 13 people had so disassociated themselves from the proceedings that they abstained.

What hope do our freedoms have if 24 out of 30 people are willing to throw those freedoms away without any evidence of an actual infraction? What hope do our freedoms have when 24 out of 30 people hold those freedoms so cheap they would discard them at the slightest whiff of a threat? As things now stand the Tibco corporations has successfully removed the word rendezvous from the BPEL TC's dictionary without having to spend a dime on legal proceedings or providing any proof that our use is infringing.

There is still, however, some hope. We will have a vote in the group on what term to replace 'rendezvous' with. The group could still choose to not replace it at all. There is still a chance left for people to realize their mistake and stand up for all of our liberties. Our individual actions do matter. Our collective freedoms only exist to the extent that we exercise them. Let's hope that the group decides that our rights to use our own language without corporations like Tibco taking the very words from our mouths is a right worth defending.

Unfortunately it's easy to get extensibility design wrong. Typically such design errors result from assuming oneself or others to be perfect and in that assumption one fails to provide for sufficient extensibility. A rather subtle example of this problem recently came up in the Web Services – Business Process Execution Language Technical Committee (WS-BPEL TC).

The BPEL TC is trying to agree on how to enable extensions to BPEL and its execution environment. The working assumption of the TC is that extensions will have IDs and those IDs will be URIs and those URIs will be included in each BPEL file that makes use of the identified extension. Each URI can then be tagged with a mustUnderstand attribute that indicates if the BPEL engine must support that extension or not. When processing a BPEL program file a BPEL engine will check the list of extensions and if it doesn't recognize any extension marked "must understand" then the engine will refuse to run the file.

Where things get interesting is a proposal to overload the extension ID, which is a URI, to also have it be the XML namespace for any XML attributes or elements associated with that extension. The attraction of doing this is that a BPEL engine can collect together all the XML attributes/elements in a BPEL program instance and determine if those elements/attributes come from the BPEL namespace or one of the extension namespaces. If the answer is neither then the element/attribute must be illegal.

The benefit of this trick is that it can automatically catch certain typos. If someone misspells a namespace name then this trick would automatically catch it. Of course the trick is useless if someone gets the namespace right but misspells an element or attribute local name as the only information automatically available to the BPEL engine about an extension as a result of this trick is just its namespace. This trick also can't catch when elements or attributes are used with the wrong syntax or in the wrong locations, again, that information is not encoded in the extension declaration. In fact the value of this trick is so minimal one wonders why it's worth having at all? Still, I've seen variants of this design pattern attempted so many times that I think it's worth cataloging what the problems are.

In theory there is nothing wrong with the trick, so long as all extensions are perfectly designed for every possible use. In other words this design assumes extension writers are infallible.

For example, let's say that Company A comes up with two extension XML elements to BPEL, <EXT1> and <EXT2>. It puts these two elements into the http://companya.com/ext namespace which, according to the previous design, must also be the extension ID for any BPEL process that uses these two elements.

Now along comes Company B. Company B has invented its own new elements but it turns out that company A's <EXT1> extension element would perfectly complete Company B's extension set. However Company A's other extension element, <EXT2>, doesn't fit in at all so Company B doesn't want to have to support it. Naively enough Company B creates a new extension called http://Bcompany.com/bpel which includes all the new Company B extensions as well as implicitly referencing the <EXT1> extension element from Company A's namespace.

Of course as soon as a programmer creates a BPEL program that uses <a:EXT1 xmlns:a="http://companya.com/ext"/> but only declares the extension http://Bcompany.com/bpel, the BPEL engine will throw an error. It will say "your program is in error because you use an element from the namespace http://companya.com/ext but you don't declare that as an extension."

Company B can work around this problem by telling everyone who uses Company B's extension to make sure to include an extension declaration for http://companya.com/ext but to mark it as "Must Understand = NO". Now the automatic validator won't fail on the element from Company A's namespace but by making the extension declaration optional engines won't actually have to support all of Company A's namespace, only the <EXT1> element that the Company B extension implicitly requires.

An immediate complication of this trick is that it reduces the utility of the typo prevention system. Notice what happens if a programmer using Company B's extension accidentally types in <EXT2> instead of <EXT1>. Since <EXT2> is from the same namespace as <EXT1> the automatic validator will think it's legitimate. In a funny way the more re-use extensions make of other people's work the less valuable the typo prevention mechanism is because the more extraneous namespaces will be declared and so the larger the room for error.

Also note how complicated the automatic validation logic will have to be. A really simple validator would go through the optional extensions, identify any that aren't supported and delete any elements or attributes in the BPEL file from the unsupported extension's namespace. But that won't be possible because extensions like Company B's may have implicit requirements for specific elements from other namespaces. So in this case there would have to be some kind of exception list that an engine that supported Company B's extension but didn't support Company A's exception would have to have that told it that it was o.k. to delete all elements and attributes from Company A's namespace except for <EXT1>.

But wait, the fun's not over yet! Throwing in an optional reference to http://companya.com/ext can cause additional complications because an extension doesn't just define new attributes and elements, it can also define new behavioral requirements for the BPEL engine. For example, Company A's extension might include the implicit requirement that any BPEL engine that supports the extension must use weak cryptography for web service communications in order to make Company A's extension exportable. So when someone just wanting to use Company B's extension, which neither needs nor wants the weak cryptography limitation, is forced to include http://companya.com/ext as an optional extension (to subvert the 'typo' protection) and if the BPEL engine that the program is running on actually supports Company A's extension then they will find their program inadvertently only using weak cryptography. This is a great example of the law of unintended consequences.

To dig out of this mess Company B will have to explicitly define as part of its mandatory semantics that if Company A's extension is used but is marked as optional then the weak cryptography requirement required by Company A's extension must be ignored.

To review, the problems with overloading the extension ID to also be the namespace name are:

1. Makes Re-Use Painful – One can't just declare a single extension and be done. Rather, one has to declare the extension and then declare extensions for each and every namespace that the first extension draws elements or attributes from. In addition the extension designer has to review the implied semantics of all the extensions that it re-uses elements or attributes from and determine which of their implicit engine semantics to de-activate.
2. Doesn't Actually Catch Many Typos – Because the typo system only works at the namespace level it already can't catch errors in typing element or attribute local names and because of the requirement to throw in namespace declarations for each and every namespace one borrows so much as one attribute or element from the typo validator will become even less useful.
   

The core of all of these problems is the underlying assumption that people who design extensions are infallible. That extension designers will exactly nail the semantics, attributes and elements that everyone else will want to re-use so there will never be a need to borrow just part of the attributes or elements in an extension or to override any of the extension's implicit engine semantics.. A humbler approach admits to both the possibility and indeed probability of mistakes in the granularity of extensions and so doesn't overload extension IDs with namespace functionality.

The answer is devastatingly simple – money.

Customers love technologies that have the word standard, no matter how disingenuously used, applied to them. Customer reactions to the word 'standard' are so strong that it causes vendors who are dread enemies to get together in back rooms and cook up something they can slap the 'standard' label on. Sure the technologies are typical examples of design by committee, sure they are badly thought through, have no real world experience and more or less fail to work as soon as they are tried in the real world. And of course they are never actually standardized in any useful sense of the term. But hey, the word 'standards' is applied to them, so they must be good!

You'd think that eventually customers would clue in. That they would learn the sort of things I talk about in my buyer's guide to standards and refuse to touch the stuff. But quite the opposite, everyone, everywhere, customers, analysts, journalists, etc. declare these messes to be the greatest thing since sliced bread. Everyone says "well sure, we know it has problems but 'they' will fix it." Personally, I'm still waiting for 'they' to show up.

Customer interest in standards is understandable. A true standard, that is, a specification that is based on long experience that has been through extensive review with clear intellectual property rights (IPR) and solid change processes provide customers with investment protection. It makes sure that no single vendor or even small group of vendors can screw a customer over. As long as the customer sticks to the standard they can mix and match vendors at will. Standards are good stuff. So who can blame customers for being excited when one is available and for preferring standardized technologies to non-standardized ones?

But the cruel reality is that real standards come at the end of a technology's innovation period, not the beginning. By the time a real standard shows up the technology isn't new or inventive. This is inherent to the process. Standards come after a technology is fully understood. Premature standardization results in badly constructed, difficult to use, non interoperable, non portable messes. But so long as the word 'standard' can be said next to these toxic spills, at least with a more or less straight face, customers will lap it up.

In the end premature standards fail. They collapse under their own ill considered, untested, badly understood incoherent designs. But who is hurt? Certainly not the vendors. They will just sell the next snake oil. The ones who get really hurt are the customers who are now stuck with sludge. The ultimate irony is that customers will end up paying the vendors tons of cash to figure out how to get the old snake oil to work with the new.

So who can blame the vendors? If customers won't get educated enough to protect themselves you can't blame the vendors for taking advantage of customer ignorance. Capitalism only works when both the buyer and the seller are vigorous in defending their own interests. If buyers let down their side of the bargain no one should be surprised if the sellers take advantage. Caveat emptor indeed.

The Background

WebDAV is a very cool technology that I provided a nice, short, summary of in an article I wrote a while back. But, to quickly recap the article, WebDAV is a HTTP based protocol for accessing data. It defines how to surf the data's hierarchy, how to add/edit/copy/move/delete both folders and files in that hierarchy and how to set/get properties belonging to folders/files in the hierarchy. It works great for accessing e-mail, file systems, directories, databases, whatever. It's a standard, it's widely adopted and it's very nifty.

The Problem

WebDAV has a weakness that has been causing me some problems as I promote its use – Search.

The DASL effort, to add search to WebDAV, has been going on for some time now and the latest draft, dated January of this year, is keeping hope alive. But there is a technique that was used to outstanding effect with WebDAV that I think could really help DASL get a lot of community interest and support – ride on someone else's coat tails. In the case of WebDAV the coat tails we rode on were XML's itself. WebDAV was the first protocol I'm aware of to use XML and this got us tons of free interest and publicity. Thankfully XML added real value to WebDAV so the standards committee can be proud of its pioneering efforts.

The Pitch

Today there are new coat tails that I think DASL could ride on – XPATH/XQUERY. After more years than anyone cares to count it looks like XPATH/XQUERY will finally finish sometime this decade. Meanwhile everyone is already investing in XQUERY. At last count there were 29 or so implementations, several of which are open source. JSR 225, which will standardize an API for XQUERY, was founded by Oracle and IBM and is supported by BEA and Sun. BEA, Microsoft and Oracle all have XPATH/XQUERY implementations. The XPATH/XQUERY band wagon is clearly one worth being on and one that could really help pump up interest in DASL. And, much as with XML's contribution to WebDAV, XPATH/XQUERY has a ton of real value to add to DASL.

I realize that XPATH/XQUERY is significantly more complex than DASL's current basic query, but given the enormous number of existing implementations , including open source ones, I do not believe that the complexity of XPATH/XQUERY need be a burden on the WebDAV community. Equally importantly, XPATH/XQUERY has already figured out a lot of very hard problems regarding how to search XML. By using XPATH/XQUERY, the DASL community would avail itself of the enormous experience built right into XPATH/XQUERY.

The Details

What I am suggesting is that DASL make one big change, replace its basic grammar with some profile of XPATH 2.0. This, of course, begs the question of exactly what a XPATH 2.0 expression submitted in a WebDAV SEARCH method would be querying. After all, XPATH expects to be applied against something, what is the 'something' that is would be searching?

I propose that what XPATH be applied against is a representation of the WebDAV resource space as a XML infoset. A XML infoset is essentially an abstract representation of the XML object model, it consists of a hierarchy whose entries are elements, attributes, text, etc. It allows one to model XML data independent of the actual XML serialization.

So what my proposal boils down to is creating a virtual XML document, represented by the infoset, whose contents are the names, properties and URIs of the resources within a WebDAV search space. The XPATH would then be applied against that virtual document. A XML serialization of the virtual document, something which would never be created in the real world and is only included here to make it easier for the reader to visualize what I'm talking about, could look like:

<root xmlns="dav:infoset">
   <resource>
      <name>http://example.com/</name>
      <properties>
         <displayName>example.com root</displayName>
         <resourcetype><collection/></resourcetype>
      </properties>
      <children>
         <name>http://example.com/index.html</name>
      </children>
   </resource>
   <resource>
      <name>http://example.com/index.html</name>
      <properties>
         <getContentType>text/html</getContentType>
         <resourcetype/>
      </properties>
      <representation>
         <properties>
            <getcontentType>text/html</getcontentType>
            <getcontentLanguage>en</getcontentLanguage>
         </properties>
         <bodyURI>http://example.com/index.html;lang=en</bodyURI>
      </representation>
      <representation>
         <properties>
            <getcontentType>text/html</getcontentType>
            <getcontentLanguage>de</getcontentLanguage>
         </properties>
         <bodyURI>http://example.com/index.html;lang=de</bodyURI>
      </representation>
   </resource>
</root>

Queries would be executed against this infoset as if itreally existed. In reality the WebDAV server would take theincoming query and translate it into a form that matched how thesystem actually stored information. It bears repeating that theinfoset is a conceptual not an actual entity. I'm notsuggesting that WebDAV servers go create a huge XML document withall their data in it and then run XQUERY against it.

Queries that wanted to query both the values of the name/property space as well as the contents of actual resources could use the document-uri function. This function returns the contents of a resource it is pointed at and makes it available for querying. See the Q&A in the appendix for a discussion of why I choose this approach.

At some point full XQUERY support would be needed but I would remind the reader of an old design rule – The spec is done where there is nothing left to cut. A standardized DASL specification with the current query language negotiation features and a 'basic grammar' based on XPATH 2.0 would be very useful and well worth standardizing. One could then have a later extension spec that added in full XQUERY support as a new pluggable query language option.

The Conclusion

As the Web is finally, slowly, moving from a read only to a truly collaborative environment WebDAV is more and more finding an audience who realize that WebDAV solves real problems. Unfortunately search is still a chink in WebDAV's armor. But DASL can turn this problem into an opportunity if it adopts XPATH as its basic grammar and so allows itself to leverage the excitement and investment available via the XPATH/XQUERY community.

Appendix – Some thoughts on the WebDAV Infoset

Below is a RelaxNG compact schema that describes what an infoset for the WebDAV resource space could look like:

namespace dav = "dav:infoset"
start = element dav:root { Contents* }
Contents = Collection | Resource

Resource = element dav:resource { Name+, Properties, Representation* }
Name = element dav:name { xsd:anyURI }
Properties = element dav:properties { RTNoCol & AnyButRT* }
RTNoCol = element dav:resourcetype { (element * - dav:collection { any* })* }
Representation = element dav:representation { RepProperties, BodyUri+ }
RepProperties = element dav:properties { anyElement* }
BodyUri = element dav:bodyURI { xsd:anyURI }

Collection = element dav:resource { Name+, ColProperties, Representation*, Children }
ColProperties = element dav:properties { TypeCol & AnyButRT* }
TypeCol = element dav:resourcetype { element dav:collection { anyElement* } }
Children = element dav:children { Name+ }

AnyButRT = element * - dav:resourcetype { any* }

any = anyAttribute | text | anyElement
anyAttribute = attribute * { text }
anyElement = element * { (anyAttribute | text | anyElement)* }

Q&A about the Infoset Idea

Q. Why is the infoset so flat?
A. My original design was hierarchical where resource elements could be contained inside of collections. But the HTTP URL hierarchy is just one particular view on a WebDAV space. There are lots of other possible views, as the bindings spec enables. So rather than get into the question of 'whose view' is being seen I though it easier to just make the whole thing flat. I also suspect that the flat hierarchy will make it easier for implementers to map between queries and implementation. But I could be completely wrong in which case perhaps the structure should be hierarchical.

Q. How do I search on the content of a resource?
A. XPATH includes a document-uri function call whose response is the data recorded at that URI which themselves become part of the data available to XPATH to examine. So if one wants to search on the contents of the body one just uses the document-uri function call. Originally I had thought of just including the content of the resources directly into the infoset model but this is probably a bad idea. As Michael Rowley pointed out to me, it makes simple queries about names and properties more complex as one has to explicitly exclude the body content. Perhaps even more importantly it will probably lead to people making really expensive queries. Someone looking for a resource named foo.htm is quite likely to do a search on "//foo.htm". The results of this query on an infoset that directly includes the contents of each and every resource is left as an exercise for the reader.

Q. Why is the bodyURI element's value different than the name element's?
A. A single resource can have multiple representations in various languages, encodings, etc. In that case we need unique URIs for each and every representation so it can be uniquely addressed. Keep in mind that the URI can be completely bogus. For example, the URI could be "data:id239428390432" which is some unique identifier. The point is that the URI is only meant to be used in the document-uri XQUERY function although it is good hygiene to use a URI like 'data' if the URI is not normally resolvable. I suspect, btw, that in the majority of cases the bodyURI will never actually be generated and instead will only be implicitly used in a query.

Q. Isn't the Children element unnecessary?
A. In theory one can reverse engineer the contents of collections by searching through the names in the infoset. E.g. all the children of foo should have names that match foo/*. But specs like bindings, which enables one to make arbitrary URIs members of collections or ordering, which allows one to specify an ordering for the children of a collection, illustrate the need for an explicit membership list.

Q. What if a resource has multiple names?
A. That's why the name element is allowed to show up more than once.

Q. Why did you use properties in the representation element instead of just using HTTP headers?
A. I have an agenda to see WebDAV made available via web services. It would take no great effort to turn WebDAV methods into WSDL operations that could then be mapped to SOAP. So this motivates me to stay away from explicit HTTP headers. Since WebDAV already provides mappings of the major HTTP headers into the WebDAV property space, e.g. getcontenttype, I decided to just leverage that existing mapping.

Yaron's rules of standardization:

• The technology must be very old and very well understood
  • 20 years is a good rule of thumb
• Everyone must implement the technologies in essentially the same way
  • A good rule of thumb is, how hard would it be to build a bi-directional proxy between the various players implementation of the technology?
• Standardizing the technology must provide greater advantage to the software writing community then keeping the technology incompatible
  • Even in the open source world standards can fail if there isn't enough advantage in it, just look at the fights over RSS.

Regardless, many companies are still afraid and want to stop Open Source. Unlike previous commoditizers who were companies, with lawyers and marketing departments who could step in if unfair things were done there is no similar protector for Open Source. So it is very tempting to create legal barriers to Open Source rather than just competing with it.

One of the easiest ways to create those legal barriers is through patent licensing terms. After all patents are legal monopolies and one is allowed to apply a legal monopoly in any way one wants, including ways that stop or hinder Open Source. A hot area for patents these days are the patent licenses being offered for Web Services protocols. Software companies have made it clear that they won't implement a Web Service protocol if it isn't royalty free, in other words, they won't implement it if they have to pay someone money. However a simple technique allows one to write a royalty free license in such a way that it will not hurt a typical commercial software company but will hurt Open Source. This technique is called 'a non-transferable license'.

Imagine that Company A files for a bunch of patents for its new Web Services Protocol WS-FooBar. Company B files for a WS-FooBar royalty free patent license with Company A which Company A grants. Generally speaking the way non-transferable licenses work is that Company B is free to implement WS-FooBar and sell product to its customers which includes the WS-FooBar code. The customers don't have to get their own WS-FooBar licenses so long as they use Company B's binaries. Where things get interesting is if Company B decides to make its WS-FooBar program Open Source.

Now company C enters the picture. Company C is creating an internal program based on company B's source code. However, because Company C is using Company B's source code and not its binaries it will be necessary for Company C to file for a WS-FooBar royalty free patent license with Company A.

This doesn't sound too bad until you think about what it actually takes to get a license. Let's say that the average piece of non-trivial Open Source software supports 5 "open standards" technologies and that each technology was authored by 5 companies. That means just to use one piece of software Company C will have to file 25 license applications.[1] Imagine some engineer at Company C downloading some Open Source and then having to go to their legal department and saying "um… I need you to approve us filing these 25 licenses". Keep in mind that this process will have to be repeated for each and every Open Source program whose source code is being used since most non-transferable licenses are granted on a per-application basis.

Non-transferable licenses are actually a knock out blow to GPL . Section 6 of GPL requires that anyone who downloads GPL source code is free to change/use it in any way they want without having to ask anyone's permission. Company B's source code can't meet this requirement because of its dependence on the WS-FooBar patent license. In other words, because Company C has to ask Company A's permission to use Company B's source code it isn't possible for Company B's source code to be licensed under the GPL. Therefore any source code that involves a non-transferable patent license cannot be implemented under a GPL license.

Previously I had argued that customers needed to demand Royalty Free/Reciprocal (RFR) licenses. Now apparently the demand has to be extended to Royalty Free, Reciprocal and Transferable (RFRT).

In the end it will be up to customers to decide if they want to benefit from Open Source. If they believe Open Source is in their interest then they have to refuse to use Web Service protocols that are not RFRT.

[1] For example, if SOAP & WSDL had been licensed under non-transferable licenses then anyone using software that including SOAP and WSDL would have to file 26 licenses.