apparently.me.uk

New Android API: StrictMode

via android-developers.blogspot.com

Nice commentary on an awesome new feature of Android Gingerbread. Simple in concept, but powerful and useful. (Which, I have to say, is true of most of Brad's projects.)

How broken is Google's accounts system? Let me count the ways

Google has a number of products that could reasonably be described as "social" products. Many Google products that you wouldn't normally think of as social have interactions with other users, such as sharing a document on Google Docs.

Unfortunately, my attempts to use these products is almost always confounded by the fact that Google's account system is utterly broken.

The root of the problem is that not all Google accounts are created equal. There are at least four different classes of Google account; most folks these days have a GMail account, which is an account that has a GMail address associated with it. There are also non-gmail personal accounts (which I will henceforth refer to as "non-GMail accounts" which can use any email address in any domain. Then getting still more obscure there's the "Google Apps For Your Domain" (or "Dasher") type of account which allows you to use GMail for your email but use your own email domain; this is popular with small companies and non-profits as well as nerds like me who have vanity domains. Finally, there's the special kind of account that Google employees have which have email addresses in the google.com domain.

Not all Google applications will work with all of these account types. Google Reader works with a GMail account and a non-GMail account but not with a Dasher account. (I can't speak for Google employee accounts, of course.) Google Buzz only works with GMail accounts. Android's Google accounts integration supports all of these account classes but Android Market only supports GMail accounts, so you need to use GMail if you want to buy apps.

So in order to work around all of this insanity, anyone that has a Dasher account inevitably ends up also maintaining either a GMail or a non-GMail account. I had a Google Account before GMail and before Dasher so my Google Account is my personal email address on my own domain. When I started using Google Apps For Your Domain I created a Dasher account matching the email address I'd been using for years, which was of course the same address I used to make my Google Account in the first place.

So the second way in which Google accounts are broken is that there is namespace ambiguity between non-GMail and Dasher accounts. I currently have two accounts at Google with the same email address, and it's a lottery what will happen when I try to use any Google product.

Some Google properties seem to use the email address itself as an effective user PK, which means I see the same stuff whatever user I log in as. Google Latitude is an example of this. Other properties try to get me to choose which account I want to use when I access them for the first time, with hilarious consequences.

Services which handle their own sessions rather than using the .google.com session cookies often choose which account to use based on what password I enter, and I got Google App Engine's web UI into a real state by using my Dasher credentials on its non-Dasher UI and now my app dashboard is one account but my account verification (by SMS) is on the other account, so I have to log in as a different account to create an app as I do to manage it.

Many properties only support my non-GMail account, which is fine by me since that skips over the issue of my Dasher account entirely. But as I noted earlier, there are a few Google properties that absolutely require a GMail account, such as Buzz. Furthermore, while Android supports all of the classes of account its Google Accounts integration will only allow one account for each unique email address, so if I have my Dasher account added (which is where my contacts, IM and calendar are) I can't add my non-GMail account (which is where my Google Maps "My Maps" are, amongst other things.)

Last week I tried to work around both of these issues by giving in and adding GMail to my non-GMail account, thus giving it an address in the gmail.com domain. This changed my primary account email address across the whole of Google, which resolved the ambiguity with my Dasher account, but promptly broke everything that uses my email address as the key for my account data. One example of this is Google Latitude: on my phone I use my Dasher account to get my calendar to work, but the Google Latitude Location History web UI does not support Dasher accounts. It worked before by accident, because my data was keyed on my email address. Now that my non-Dasher account had a different email address, I was unable to use Google Latitude Location History.

I've now removed GMail from my non-Dasher account, returning it to its status as a non-GMail account with a conflicting email address. However, now my account has an "other username" associated with it, which matches what used to be before @gmail.com in my no-longer-valid GMail address. However, a couple Google properties erroneously assume that this username must be a valid GMail account to which email can be sent, ignoring the real email address associated with my account.

All of this is moderately annoying when I'm trying to use Google applications on my own. It's infuriating when it comes to interacting with others, since Google's properties almost always use email address as the primary identifier by which one user begins an interaction with another. I logged into my Dasher account's Google Docs recently and saw a dozen files people had shared with me which I didn't notice because I use Google Docs in my non-GMail account. I'm following several people multiple times in Google's social graph because they too have multiple accounts, sometimes with the same email address on more than one.

In the three years or so I've been complaining about this to every Googler who will listen (sorry!) I've often been told that they're working to fix it, but that doesn't make it any less infuriating when for the umpteenth time I'm trying to figure out what account I need to use to do what and how I can trick each new Google property into doing the right thing.

It's really hard to take any of Google's social products seriously when they've failed to get their user accounts, which is the foundation of all social interactions, to work in a sane way.

Important Changes in OpenID Connect

As someone who is very familiar with the capabilities and plumbing of OpenID 1.1 and OpenID 2.0, my first question about OpenID Connect was around what has changed. Here are four important things that have changed in OpenID Connect in terms of capabilities and usage model, ignoring the specific wire protocols.

Identifier, Profile Page and User Input are decoupled

The first major change is that in OpenID Connect the only supported flow is what OAuth 2.0 called the "identifier_select" flow: you enter something that identifies your server (returning to the OpenID 1.1 vernacular) and the server tells the client (formerly known as RP, or before that consumer) your persistent identifier. The persistent identifier can then be mapped to some details about the user including the user's human-friendly profile page.

This decoupling has some important implications. The first is that the profile page no longer hosts any discovery information and therefore it can quite happily be served in cleartext. I'll touch on some other related implications in subsequent sections.

Delegation is no longer supported

"Delegation" in OpenID terms creates an alias identifer: In OpenID 1.1 I could make any page I control an alias for any OpenID identifier in spite of the provider of that identifier. OpenID 2.0 made delegation detectable by the provider, and subsequently many providers (as was predicted by many folks at the time of the decision) intentionally or accidentally broke this capability.

This mode of delegation was largely a tactic to get adoption: it was easy for us nerds to just add some HTML to our personal home pages and get an OpenID identifier to play with but not actually need to run our own OpenID provider to do so. I don't think anyone seriously thought that this would be a real use-case in the long term, since "normal people" don't have personal websites whose HTML they can hack at.

OpenID Connect gives you three choices: use an identifier issued by an existing provider, use a provider that provides white-label OpenID services under your domain, or run your own provider. The first and second of these options are reasonable for your average user and the last is for us nerds.

One OpenID Server per Domain

In OpenID 1.1 and OpenID 2.0, each unique URL can potentially have its own discovery information pointing at a different OpenID Provider endpoint. In OpenID Connect there is only one OpenID Server per domain. When you sign in, what is considered for discovery is a bare domain rather than a full URL.

However, as currently written an OpenID Server can actually make assertions about identifiers in any domain as long as the identifier itself marks the relevant OpenID Server as authoritative; if I'm outsourcing my domain's OpenID services to GMail there's no reason why Google couldn't set things up such that I can "Sign in with Google" but actually ultimately authenticate as my vanity domain.

Provider Buttons are Baked In

The use of user-opaque identifier URIs and the identifier_select flow means that the "Sign in with specific provider" buttons are here to stay under OpenID Connect. In principle the process can be kicked off by the user entering the domain of a provider, but that user experience has never been shown to be successful under OpenID 2.0; instead, identifier_select was used to implement one-click signin to popular providers.

Although it is in theory possible to provide an "enter your email address" user experience, since discovery only considers your domain all you're doing here is telling the client to use your email domain's OpenID provider; ultimately you might end up identified as a completely different user if you're already signed in and cookied at the provider as someone else, so this is at best a smoke-and-mirrors trick.

As Facebook is increasingly the most popular identity provider, and Facebook doesn't issue email addresses, it is becoming increasingly the case that asking for a user's email address to begin OpenID discovery won't be useful anyway, since a user is more likely to actually want to sign in with Facebook than whatever arbitrary provider they happen to use for email, especially since email is for old people.

Does anyone actually care about Atom Activity Streams?

When the Activity Streams project kicked off as an informal gathering of people at the Six Apart office over a year ago, we observed the plight of services like FriendFeed and Plaxo Pulse that were trying to aggregate activity from disparate sources into a single stream but had to implement custom code for each services. This means that each new service has an uphill struggle to gain enough adoption to be considered worth the implementation effort by the aggregators, and means separate integration work for each site an aggregator supports. We perceived this as the main use-case to address at the time. If I want to host my own content on my own domain then I'm completely hosed under this model.

With this in mind the first draft of AtomActivity was produced, which observed that these aggregators are built on feeds and concluded that therefore we should add some annotations to these feeds to provide the information that was previously being hard-coded on a provider-by-provider basis by the aggregators. The theory was that publishers would just add a couple new tags to their feeds and be done, and then publishers could consume a feed from an unfamiliar source and handle it with the same fidelity — or at least, a much closer approximation of the fidelity — as a feed from a popular service.

Part of the design is a progressive enhancement model where a compliant activity streams consumer will actually consider any mostly-valid Atom entry as a somewhat-barren-but-still-usable activity, and feed publishers can progressively add additional tags to their feeds as desired to achieve higher fidelity handling.

A small trickle of publishers started to produce Activity Streams in a manner that supports the kind of ad-hoc association that this specification was designed to afford, but on the whole the Atom serialization of Activity Streams has largely been used for tightly-coupled implementations between specific partners. At the time of writing I know of not a single implementation that will take an arbitrary Atom feed containing the Activity extensions and process it, except my own Activity Streams Tester tool which hardly counts... and after over a year to bake that's pretty pitiful adoption.

It seems to me that the world isn't yet ready for ad-hoc activity syndication. The prevailing pattern is that consumers want to target specific publishers and publishers want a legal relationship (that is, a terms of service agreement) with their consumers. In this world, none of the motivations that led us to an Atom extension apply. We end up with all of the drawbacks of using Atom and none of the benefits. Parsing the Atom serialization is an order of magnitude more complicated than the JSON serialization, and the Atom serialization sections are becoming an albatross in the spec due to the sheer complexity of handling all of the cases that support progressive enhancement.

I wrote at length a couple months back about how a simple JSON model is evidently more appropriate for tightly-coupled integrations, but at that time I remained hopeful for the ad-hoc use-case. I think it might now be time to throw in the towel on the ad-hoc integration ability and return to simplicity.

Are there any consumers out there that are ready to implement and ship consumption of arbitrary Atom Activity Streams feeds? If so, please make yourself known. If not, I think it might be time to commit Atom Activity Streams to an archive (as a reference for those who have existing tightly-coupled integrations) and recast the spec as simply the JSON serialization. The resulting spec will be shorter, more straightforward and, I think, more useful for solving the problems we have today rather than problems we thought we'd have a year ago.

Let me know!

Why XAuth is fundamentally wrong

Right now Google is pimping this weird spec from Meebo called XAuth. This should not be confused with Twitter's xAuth, which is a different beast entirely. Google is currently spinning this as a "Facebook vs. the rest of the web" issue, but I'm not going to get caught up in all of the marketing bullshit. The technology should speak for itself.

It seems that the main goal being publicized for XAuth is to allow one site to declare that the user wants to possibly "share" content on that site so that another content-publishing site can use this information to show only the share buttons that the user wants to see. This is a sensible goal: the proliferation of "Share this!" buttons is becoming pretty annoying.

However, the approach that they've taken is flawed in a number of ways. Here are my top three:

• It's designed around a single point of failure. It relies on xauth.org continuing to host the chunks of JavaScript that the specification requires and it continuing to behave in a trustworthy manner. At this point I don't think I really need to go into any more detail about why a single point of failure is a showstopper for any Internet-wide protocol.
• It requires strong coupling between sites. Given the stated goals, I expected to be able to use this protocol to answer the question "What sites does this user use for sharing?". Instead, it requires the caller (or "retriever", to use their terms) to explicitly name the domains it wishes to test, making the question more like "Does the user use Digg, StumbleApon or Facebook?". If you're going to have a strong coupling like this then you might as well just go ask the sites you're interested in directly; there is no need for the single point of failure that is fundamental to this protocol.
• It allows sites to self-declare that they are used by the current user. Assuming that this approach ever gained any traction — which I honestly hope it does not — there's nothing to stop any site from saving "user uses this site!" state immediately when the user loads any page from its domain, without any user intervention. For this to be useful at all to an end-user, the end-user must be in control.

So in summary, this protocol is architecturally flawed, doesn't effectively solve the problem it claims to address, and is not resilient to abuse by bad players. I agree that this is a good problem to solve, but this is not the right solution.

I'm optimistic that XAuth will get folks thinking about this problem space and get us closer to the right answer, but we're not there yet.

A little Type goes a long way

TypeKit is a service that uses CSS font embedding techniques to bring commercial fonts to the web. It's been a long time coming, but it's nice to finally be able to make use of a large library of typefaces of various styles on the web.

Sew Weekly uses two fonts from TypeKit which serve to make this design look interesting and different despite it being simple in most other ways. WebWorkerDaily is also using a touch of TypeKit. It's amazing how much difference a fresh typeface can make to an otherwise-pretty-standard design.

Of course, TypeKit is bothersome in that it creates a point of failure for any site that uses it. While fallback fonts can be listed to use where TypeKit fails, if TypeKit goes away completely any site using it will permanently revert to the web's traditional, mundane fonts.

TypeKit is doing well at smoothing the jagged edges while browser support for embedded fonts shakes out and stabilizes, but hopefully in the long term a service like this won't be necessary and font foundries will be able to issue web fonts directly such that the sites using them can host their own type assets much as they host their own image assets today.

I took the opportunity today to sprinkle my own blog with some TypeKit magic as part of moving away from my aging and creaky old design that followed me over here from LiveJournal. My new design is just standard TypePad fare, but at least now I have working comments!

The Role of JSON Activity Streams

The Activity Streams project, which is hopefully nearing completion of its first version, is build apon the Atom syndication format. This design decision was made in order to afford relatively easy adoption by those who already publish feeds, in much the same way as Microformats like XFN are relatively easy to adopt for those who are already publishing HTML. This has had mixed success, since most of the implementations so far have been fresh implementations or feed-like APIs that make use of the format.

One thing that has been raised relatively often during the development of this first iteration of Activity Streams is the possibility of a JSON serialization. JSON is, after all, often easier to process due to the fact that its data model matches better the data structures offered in most popular programming languages. This discussion causes some contention between members of the Activity Streams community, perhaps due to the existing investment in Atom as the base serialization format or concerns about the fact that JSON has not yet proven itself as a good format for extensible data models.

At this point I'd like to go on record as supporting an effort to define a JSON serialization. A while ago I published a very early draft of what JSON Activity Streams might look like, and that spurred some discussion about the merits of this vs the existing AtomActivity specification of which I'm a co-author.

The important thing to realise is that Atom Activity Streams and JSON Activity Streams both serve distinct needs. A goal of Atom Activity Streams is to allow ad-hoc integration of arbitrary implementers, by building on the infrastructure that Atom and RSS have already established. I can add any blog I like — along with just about any other non-blog content — to my feed reader of choice and it will, with some caveats, work. Activity Streams aims to add some additional markup to bring that kind of interoperability to the world of activity publishing, as RSS and Atom did for content publishing, while remaining compatible with existing implementations and infrastructure where it makes sense.

As a pertinent example, making use of Atom allowed Activity Streams to benefit automatically from PubSubHubbub, which allows activity notifications to be transmitted in near-realtime between the producer and any interested consumers without any additional specification work. Recently Google launched Buzz which demonstrates how these two technologies can be combined while still allowing for the use of traditional content feed consumers.

JSON Activity Streams, on the other hand, are likely to serve more as an integration format between two specific implementers. As an analogy, consider that OpenID supports ad-hoc association between any pair of compliant consumers and providers, while OAuth is designed for specific integrations between two strongly-coupled services. The benefit of defining JSON Activity Streams is not to allow ad-hoc federation but rather to allow code re-use and knowledge re-use when activity information appears within an API, much as I can re-use the same OAuth library when I make specific integrations with the APIs of both Twitter and TypePad but I still need, to a certain extent, to do some custom work for each.

While it is definitely preferable to allow ad-hoc integrations rather than require tightly-coupled integrations, there is a trade-off between the complexity that ad-hoc integration requires in order to support decentralized extensibility and interop with a variety of implementations vs. the relative simplicity of integrating with one external service and doing that well. The friction of a tightly-coupled integration can still be reduced by a shared understanding of protocols that promotes the ability to share as much code as possible between different integrations and make use of battle-tested third-party libraries to handle the hairy details.

So what do I see as the future of JSON Activity Streams? I expect that its use will largely be limited to being used as a common vocabulary for otherwise-proprietary APIs, probably requiring an existing business relationship authenticated by either OAuth or one of its potential successors. Meanwhile, Atom Activity Streams remains a useful way to add Activity Streams semantics to an existing, publically-available Atom feed to allow progressive enhancement that achieves more useful presentation in activity-aware clients while remaining usable for already-deployed software.

With all that said, the immediate focus I share with the other Activity Streams authors and the community is publishing a solid spec for the Atom serialization and its associated serialization that covers a large number of use-cases observed in the wild today. I feel it's more important to focus on one generally-useful approach first rather than expanding the scope so far that we'll never finish.

My JSON Activity Streams draft will remain available to document my currently-favored approach: to more directly model the properties of activity objects and to simplify the processing model compared to that required for Atom Activity Streams. I have a rough implementation of this format on my personal site, but it's by no means comprehensive. It is my hope to eventually use the abstract data structures defined for JSON as a canonical representation of the Activity Streams data model, in terms of which the other serializations will be defined. For now, the activity schema is defined in terms of the Atom serialization and I think this is a good place to start for the first iteration.

First Draft of Atom Cross-posting Extension

I wrote up a first draft of an Atom extension for declaring cross-posting duplicates. It defines both a way for the primary version to declare the duplicates and a way for the duplicates to declare the primary version.

In practice it's very unlikely that the duplicates will declare the primary version, because most of the time they don't know they're duplicates and even if they did they probably wouldn't want to admit to being "just a copy". But I hope that publishers that create cross-post duplicates will see the benefit in declaring these in the feed to improve the usability of the feed when it's consumed into a multi-feed aggregator such as FriendFeed or MT Action Streams.

An Atom extension for declaring cross-post dupes

It's becoming increasingly common for content-publishing applications to include a feature where they'll duplicate (in some sense) the content a user creates on other services such as Twitter or Facebook.

Unfortunately, this has the unfortunate side-effect that multi-feed aggregators cannot easily detect this and often end up showing the same content more than once.

However, I think we can go some way towards a technical solution to this problem without trying to boil the ocean and stop people cross-posting: have the publisher that's creating the duplicate content declare that it has done so in its feeds.

What does this look like? It feels like this just takes one very simple extension element with the same attributes as the in-reply-to element introduced by Atom Threading Extensions: a ref attribute giving the id of the duplicate entry, and a type,href pair linking to a representation of the duplicate entry. For example:

<crosspost:dupe
    ref="http://twitter.com/apparentlymart/statuses/3641424947"
    href="http://twitter.com/apparentlymart/statuses/3641424947"
    type="text/html"
/>

This alone isn't enough to do the de-duping, since we can't trust publishers not to lie about what's a duplicate, but in an application such as FriendFeed or MT Action Streams where a user has configured a list of feeds to import it is easier to assume that all of the referenced feeds are trustworthy in the context of that user: if I've got both my notes blog and Twitter both added to MT Action Streams and the notes blog declares a Twitter entry from my account as a duplicate it's fair to assume that it is indeed a duplicate.

This is not a complete solution, since it is possible that I've cross-posted to both Twitter and Facebook and you consume those two feeds but not the "origin" feed; however, I think this is a step in the right direction and solves the immediate problem at hand. It would be nice if the services that tend to receive these duplicates would extend their APIs such that publishers can declare that they're posting a dupe and so the receiving service can create a reverse-dupe element, but that's not something we can bootstrap so easily today.

I'm interested to see if any providers who offer the functionality to duplicate their content on Twitter and/or Facebook would be willing to work on this. It ought to be a reasonably easy, tightly-scoped specification and should not be a burden for implementers as long as they know how to form an Atom id (or RSS equivalent) for the services they publish to based on a service-local id returned from the API.

A Protocol for Batch HTTP Requests

There's been murmerings of discussion about the possibility of doing batch HTTP requests for some time now. Some folks maintain that it isn't necessary for one reason or another, while several popular web services provide batch request mechanisms that are tailored to their specific API but are not generally applicable.

A while back I wrote up a draft spec for a general-purpose batch request protocol, loosely based on James Snell's proposals. I've discussed this with folks at various conferences and people seemed generally receptive to the idea. I notice that since then he's posted HTTP Multipart Batched Request Format, which is also based on his initial thoughts but he went in a different direction to me. I regret that I didn't post about my draft sooner so that we could have potentially worked together on this.

My approach is to mimick the behavior of an HTTP proxy server. Although in practice I expect many implementations won't be acting as proxies in the traditional sense, it was my expectation that this would make it easier to adapt existing implementations that already know how to deal with proxies.

Simon Wistow wrote a Perl implementation of this spec (now living on my Github) and there's a Python library implementation based on httplib2 as well as a standalone proxy server written in Python using Twisted done by some of my Python-speaking colleagues, both of which will hopefully be released soon.

As with many of these things, the main win in having a standard here is that it should cease to be necessary to write a separate implementation for each new web service. Of course, until more than a couple folks adopt a standard it's just another proprietary request format, so I hope to have some discussion about this to figure out where we can meet in the middle of the various proposals and come out with something that multiple services would be happy to support.