$634 million for Healthcare.gov website

Rick Moran
That's what we paid to a private contractor to design and build this turkey?

It shouldn't surprise you that the website was slightly overbudget - more than $500 million, according to Digital Trends. Some pretty good snark from them: "We, the taxpayers, seem to have forked up more than $634 million of the federal purse to build the digital equivalent of a rock."

The site itself, which apparently underwent major code renovations over the weekend, still rejects user logins, fails to load drop-down menus and other crucial components for users that successfully gain entrance, and otherwise prevents uninsured Americans in the 36 states it serves from purchasing healthcare at competitive rates - Healthcare.gov's primary purpose. The site is so busted that, as of a couple days ago, the number of people that successfully purchased healthcare through it was in the "single digits," according to the Washington Post.

The reason for this nationwide headache apparently stems from poorly written code, which buckled under the heavy influx of traffic that its engineers and administrators should have seen coming. But the fact that Healthcare.gov can't do the one job it was built to do isn't the most infuriating part of this debacle - it's that we, the taxpayers, seem to have forked up more than $634 million of the federal purse to build the digital equivalent of a rock.

The exact cost to build Healthcare.gov, according to U.S. government records, appears to have been $634,320,919, which we paid to a company you probably never heard of: CGI Federal.  The company originally won the contract back in 2011, but at that time, the cost was expected to run "up to" $93.7 million - still a chunk of change, but nothing near where it ended up.

Given the complicated nature of federal contracts, it's difficult to make a direct comparison between the cost to develop Healthcare.gov and the amount of money spent building private online businesses. But for the sake of putting the monstrous amount of money into perspective, here are a few figures to chew on: Facebook, which received its first investment in June 2004, operated for a full six years before surpassing the $600 million mark in June 2010. Twitter, created in 2006, managed to get by with only $360.17 million in total funding until a $400 million boost in 2011. Instagram ginned up just $57.5 million in funding before Facebook bought it for (a staggering) $1 billion last year. And LinkedIn and Spotify, meanwhile, have only raised, respectively, $200 million and $288 million.

Meanwhile, Slate's David Auerbach has a somewhat technical post on what went wrong with the site:

The Redditors picking apart the client code have found some genuine issues with it, but healthcare.gov's biggest problems are most likely not in the front-end code of the site's Web pages, but in the back-end, server-side code that handles-or doesn't handle-the registration process, which no one can see. Consequently, I would be skeptical of any outside claim to have identified the problem with the site. Bugs rarely manifest in obvious forms, often cascading and metamorphizing into seemingly different issues entirely, and one visible bug usually masks others.

There are a few clues, however. The site's front end (the actual Web pages and bits of script) doesn't look too bad, but it is not coping well with whatever scaling issues the back end (account storage, database lookups, etc.) is having. I tried to sign up for the federal marketplace six days after rollout. The site claimed to be working, but after I started the registration process, I sat on a "Please Wait" page for 10 minutes before being redirected to an error page:

"Sorry, we can't find that page on HealthCare.gov."

Except that wasn't the problem, since the error message immediately below read:

"Error from: https%3A//www.healthcare.gov/oberr.cgi%3Fstatus%253D500%2520errmsg%253DErrEngineDown%23signUpStepOne."

To translate, that's an Oracle database complaining that it can't do a signup because its "engine" server is down. So you can see Web pages with text and pictures, but the actual meat-and-potatoes account signup "engine" of the site was offline. A good site would have translated that error into a more helpful error message, such as "The system is temporarily down," or "President Obama personally apologizes to you for this engine failure." But it didn't.

This failure points to the fundamental cause of the larger failure, which is the end-to-end process. That is, the front-end static website and the back-end servers (and possibly some dynamic components of the Web pages) were developed by two different contractors. Coordination between them appears to have been nonexistent, or else front-end architect Development Seed never would have given this interview to the Atlantic a few months back, in which they embrace open-source and envision a new world of government agencies sharing code with one another. (It didn't work out, apparently.)

I'm no expert but it sounds like this is a problem that is going to persist for weeks. And how much harder is it going to be fixing it if they keep the site open during reparis?

The longer it takes to iron out problems, the more likely it is that few consumers will brave the disaster to sign up for health insurance. That is the worst possible news for the administration who are desperately counting on millions of people signing up before the January 1 deadline. If they fall short, insurance companies may flee the exchanges if they can't make a profit.

There have been more expensive government boondoggles over the years. But as a metaphor for government incompentence, this example takes the cake.

H/T: Weasel Zippers


That's what we paid to a private contractor to design and build this turkey?

It shouldn't surprise you that the website was slightly overbudget - more than $500 million, according to Digital Trends. Some pretty good snark from them: "We, the taxpayers, seem to have forked up more than $634 million of the federal purse to build the digital equivalent of a rock."

The site itself, which apparently underwent major code renovations over the weekend, still rejects user logins, fails to load drop-down menus and other crucial components for users that successfully gain entrance, and otherwise prevents uninsured Americans in the 36 states it serves from purchasing healthcare at competitive rates - Healthcare.gov's primary purpose. The site is so busted that, as of a couple days ago, the number of people that successfully purchased healthcare through it was in the "single digits," according to the Washington Post.

The reason for this nationwide headache apparently stems from poorly written code, which buckled under the heavy influx of traffic that its engineers and administrators should have seen coming. But the fact that Healthcare.gov can't do the one job it was built to do isn't the most infuriating part of this debacle - it's that we, the taxpayers, seem to have forked up more than $634 million of the federal purse to build the digital equivalent of a rock.

The exact cost to build Healthcare.gov, according to U.S. government records, appears to have been $634,320,919, which we paid to a company you probably never heard of: CGI Federal.  The company originally won the contract back in 2011, but at that time, the cost was expected to run "up to" $93.7 million - still a chunk of change, but nothing near where it ended up.

Given the complicated nature of federal contracts, it's difficult to make a direct comparison between the cost to develop Healthcare.gov and the amount of money spent building private online businesses. But for the sake of putting the monstrous amount of money into perspective, here are a few figures to chew on: Facebook, which received its first investment in June 2004, operated for a full six years before surpassing the $600 million mark in June 2010. Twitter, created in 2006, managed to get by with only $360.17 million in total funding until a $400 million boost in 2011. Instagram ginned up just $57.5 million in funding before Facebook bought it for (a staggering) $1 billion last year. And LinkedIn and Spotify, meanwhile, have only raised, respectively, $200 million and $288 million.

Meanwhile, Slate's David Auerbach has a somewhat technical post on what went wrong with the site:

The Redditors picking apart the client code have found some genuine issues with it, but healthcare.gov's biggest problems are most likely not in the front-end code of the site's Web pages, but in the back-end, server-side code that handles-or doesn't handle-the registration process, which no one can see. Consequently, I would be skeptical of any outside claim to have identified the problem with the site. Bugs rarely manifest in obvious forms, often cascading and metamorphizing into seemingly different issues entirely, and one visible bug usually masks others.

There are a few clues, however. The site's front end (the actual Web pages and bits of script) doesn't look too bad, but it is not coping well with whatever scaling issues the back end (account storage, database lookups, etc.) is having. I tried to sign up for the federal marketplace six days after rollout. The site claimed to be working, but after I started the registration process, I sat on a "Please Wait" page for 10 minutes before being redirected to an error page:

"Sorry, we can't find that page on HealthCare.gov."

Except that wasn't the problem, since the error message immediately below read:

"Error from: https%3A//www.healthcare.gov/oberr.cgi%3Fstatus%253D500%2520errmsg%253DErrEngineDown%23signUpStepOne."

To translate, that's an Oracle database complaining that it can't do a signup because its "engine" server is down. So you can see Web pages with text and pictures, but the actual meat-and-potatoes account signup "engine" of the site was offline. A good site would have translated that error into a more helpful error message, such as "The system is temporarily down," or "President Obama personally apologizes to you for this engine failure." But it didn't.

This failure points to the fundamental cause of the larger failure, which is the end-to-end process. That is, the front-end static website and the back-end servers (and possibly some dynamic components of the Web pages) were developed by two different contractors. Coordination between them appears to have been nonexistent, or else front-end architect Development Seed never would have given this interview to the Atlantic a few months back, in which they embrace open-source and envision a new world of government agencies sharing code with one another. (It didn't work out, apparently.)

I'm no expert but it sounds like this is a problem that is going to persist for weeks. And how much harder is it going to be fixing it if they keep the site open during reparis?

The longer it takes to iron out problems, the more likely it is that few consumers will brave the disaster to sign up for health insurance. That is the worst possible news for the administration who are desperately counting on millions of people signing up before the January 1 deadline. If they fall short, insurance companies may flee the exchanges if they can't make a profit.

There have been more expensive government boondoggles over the years. But as a metaphor for government incompentence, this example takes the cake.

H/T: Weasel Zippers