There's been a big shift in how people use the web that caught up with Healthcare.gov and sister sites yesterday. You can build the most beautiful and "scalable" website for web visits, make it open source, put the code up on GitHub, talk about how innovative it is, then watch it crumble under the server strain of people trying to actually do something through your site.
Healthcare.gov's real challenge wasn't to build an alternative to a commercial CMS (content management system), it was to build an application that can handle event-oriented human behavior - for that you need the best systems engineering, not "10,000 authenticated users through GitHub" for your content delivery, as one of the Healthcare.gov contractors highlighted in this Atlantic profile of the project by Alex Howard.
Before the application process bogged down yesterday, Healthcare.gov got lots of nice gov tech insider buzz for its open source nature. But the project still had contractors on board, and based on how the service behaved on opening day of the Affordable Care Act, it could have stood a lot more testing of what people actually wanted to do with it. Kind of like Mitt Romney's Orca system on election day last November.
The Healthcare.gov site loaded fine, but trying to apply through it was kinda like buying first-time Comic-Con badges online.
Open source has changed the technology landscape for the better, underpinning many of our favorite startups. However, simply invoking it like a protection spell is no replacement for the architectural skill and planning required to pull off the systems needed for a successful Healthcare.gov launch. Health and Human Services, which managed the project, needed a little more "Puppet vs. Salt" and a little less "open" in its vernacular.
Adapting to a web where people are participants, not viewers, is the lesson we're all learning. Web infrastructure needs to support people, not publishing.
The reaction to failures of Healthcare.gov under heavy load won't work if the discussion is about how other services fail - it has to be about building infrastructure that's designed for peak interactivity and not for views.
Choice quotes from the Atlantic profile:
Bryan Sivak, CTO at Health and Human Services: "Instead of [running] farms of application servers to handle massive load, you're basically slimming down to two. ... The way it's being built matters."
Dave Cole from HSS contractor Development Seed: "You're just talking about content. There just needs to be one server. We're going to have two, with one for backup. That's a deduction of 30 servers."
Maybe there was a lot more infrastructure work going on behind the scenes, but the project leads' obsessive focus on the content framework is telling.
Healthcare.gov's scaling challenge was never about delivering content like a really popular website, it was the peak activity challenge that Twitter faces on a regular basis. Taking interaction-based scaling challenges seriously is why Twitter is stable now and wasn't in 2009 - those are the issues HHS should have been talking about.
Few updates after a bit of Twitter fun on these issues today:
Not faulting Alex's reporting in any way here - I believe if the HHS team was really focused on the infrastructure for supporting a signup rush at the time of the Atlantic article, that dedication would have shown up in the story. The omission of that kind of discussion (read the article - the project team seems to have an almost flippant approach to back-end server architecture). I also googled around looking for commentary on that front from earlier in the life of the project.
I didn't do a detailed investigation, this is an opinion blog piece not investigative journalism. As I said above, it's quite possible there was more going on - but the fact the site had so much persistent trouble as an actual application (while it functioned fine as what we call in the biz a "brochure site") means whatever was done fell dangerously short.
Finally, if an important initiative like Healthcare.gov is going to get 2.8 million views in a day, I want everyone who wants to apply through that site to do so smoothly. My ding on "open government buzzwords" is that it's really easy to do "innovative" things with government technology and get headlines, without actually delivering for constituents.
Another update from Twitter conversation:
Alex speculates the devs and designers who built the content framework aren't to blame here.
Fair enough. I think it's fairly clear from the above that I blamed HSS and a culture of thinking that web properties are publishing applications and not designing them for interaction. It's really time to stop talking about a "front-end" and a "back-end" for any kind of website. If it doesn't scale for interaction, it doesn't scale. Twitter's infrastructure challenge isn't displaying millions of tweets, it's keeping all of them threaded in real-time.
Open source content frameworks are nice (hey, Twitter released Bootstrap!), but HHS separated that issue from the kind of services needed to effectively scale the application process. It's like building a really shiny muscle car and then giving it a weak 2-liter engine. Fully integrated applications with content delivery and scalable interaction design are really, really hard. And that's where buzzwords fall short.
Sept. 7 update:
On Saturday, I wrote about these issues on GovFresh, "The openwashing of Healthcare.gov" and cited a Reuters article that laid the project on CGI Inc., a giant federal contractor.
Today, the Wall Street Journal quoted an HHS spokeswoman and IT experts regarding flaws in the system. The article mentions CGI and also says Experian had a contract around identity verification. Based on the analyses I've read, it seems like there could be timeouts or critical delays between security question submittal and verification, which would indicate architecture issues again, not an Experian issue per se.
On a recent segment of HuffPost Live, NationBuilder VP of Community Adriel Hampton participated in a virtual roundtable discussion on Rep. Steve Pearce's (R-N.M.) proposed legislation to create a "virtual Congress." Rep. Pearce believes that allowing members of Congress to spend the majority of their time in their home districts will enable them to be more connected to the needs of the people they are representing.Read more
Today I stumbled upon a bunch of the recent news about Groupon firing CEO Andrew Mason. I went looking for my own anti-Groupon blog post from right before its IPO. Tracking it down made me finally back up my two Posterous blogs (sorry to see Posterous go, always liked it better than Tumblr personally). Anyway, here's the Groupon piece, from Jan. 15, 2011.Read more
Ottawa, Public Sector Engage 2012, November 27, 2012:Read more
The last few weeks at NationBuilder have been personally very rewarding for me. In January, we publicly released our government edition, and we've already seen really exciting adoption rates. I'm loving the discussion our organizers are having with government folks around the country about how we can be of service.
As a hard-core Gov 2.0 advocate, one of the big things I want to see change in government technology is efficiency. Traditional cloud vendors and open data advocates have really helped turn the tide over the last couple years towards helping governments realize huge cost savings through SaaS technologies. Here are NationBuilder we're ready for that shift.
Read more about NationBuilder's new government edition here. My site is currently sporting our free government-focused theme, Civitas, and you can see our Pershing Forest theme here. Email government [at] nationbuilder.com for more information.
Social trust is a popular online currency in modern political campaigns. Now a new service, Candidate Check, is bringing the mature industry of background checks and opposition research into the political web game. Founded by David Doud, a two-time candidate from Seattle frustrated by how easy it is for charismatic candidates to sweep up votes irregardless of their credentials and backgrounds, Candidate Check turns the traditional background check on its head by asking candidates to pay for the service and post a "Candidate Trusted" seal on their websites linking back to a independent report from the service.Read more
David Campos, my former colleague at the San Francisco City Attorney's Office and current SF District 9 Supervisor, is running for re-election. He represents the Mission, Bernal Heights, St. Mary's Park and Portola neighborhoods. David's also using my company's NationBuilder software for his race. Check out davidcampossf.com for more info.
For the past few years, my primary advocacy focus has been on increasing government efficiency and transparency through technology. One of the areas of focus for me and other government-reform-through-technology advocates has been "open data", or increased access to government information, in machine readable and structured formats where possible. Other prominent open data advocates include Craig Newmark, founder of Craigslist and the Craigslist Foundation, and Tim O'Reilly, a tech publisher who has popularized the notion of "government as a platform" for economic growth and innovation.
Over the past several months, I've been supporting a California initiative that would incorporate a modest "open data standard" into the California Public Records Act.
The bill is backed by the California Newspaper Publishers Association, the SF Tech Dems (a group I co-founded last year), the California Faculty Association, California Teachers Association, Common Cause, the American Federation of State, County, and Municipal Employees and many other prominent good government and employee groups across California and the United States. The most important provision of the bill, the California Senate 2011-2012 session's SB 1002, by Sen. Leland Yee, would ensure that when available, records would be produced in electronic form searchable by free software. The changes the bill proposes to California law are key to ending practices of posting records in unsearchable formats that are not indexed by search engines and are of low value for resuse and research.
Regretfully, the lobbing group League of California Cities has made gutting or killing this modest and needed reform one of its top priorities. The League, in a muddled screed nearly as long as the simple open data bill, makes false assertions about its language and intent, and attacks the aims of open data advocates: "It appears that the bill sponsors are more focused on imposing a new mandate requiring public agencies to create new data files and formats on request, to facilitate the creation of commercial, information-based products and services at public expense." In fact, open government advocates are very forward about our goals, which include reuse of existing government data as an economic engine. It is befuddling and troubling that the California League of Cities would object to beneficial reuse of public records and adjacent economic growth. (Ironically, the League's effort to kill this important economic growth and government transparency bill is funded by your money and mine in the form of its public funding for Capitol lobbying on behalf of city executives.)
SB 1002 is the product of open and public discussion by open government advocates around the state, nation and world. The League of California Cities has participated only through misleading lobbying and efforts to mute open data advocates.
Adriel Hampton is an entrepreneur, private investigator and journalist. He is a founding employee of NationBuilder, founder of Gov 2.0 Radio, advisory board member of LegiNation Inc., and co-founder of the SF Tech Dems. Before joining NationBuilder, he worked for the San Francisco City Attorney's Office for six years, where he developed the office's social media practice, and as an editor and writer at the San Francisco Examiner, ANG Newspapers and the Lodi News-Sentinel.