As an object lesson in how not to launch a website intended for millions to conduct personal business, the federal government’s HealthCare.gov fiasco takes some beating. But could critical, data-based thinking, aided by modeling and simulation, have avoided such a result?
In the weeks that have followed its debut, it’s become clear that a website that was intended for use by millions of Americans nationwide could—when launched—probably handle only 500 concurrent users at a time. Throw in what appears to be a lack of testing, poor design, inadequate bandwidth, slow-to-load graphics-rich pages, and a population of 317 million people, and failure was probably inevitable.
HealthCare.gov had received more than 19 million website visitors in its first month. But just 27,000 people were able to successfully select an insurance plan in the entire month of October, a figure that the government said climbs to 106,000 when 14 state websites are included.
Recriminations have focused on a raft of apparent failures. Poor oversight, poor project management, rushed testing, split responsibilities, and poor IT system design: each has been pointed at as a causal factor, and all were seemingly present.
What hasn’t been asked so much is what could have been done to model and simulate the working of the site, so as to highlight its shortcomings in advance. In other words, what role can be played by fact-based, dispassionate analysis?
Put another way, if we can all make multiple successful visits to online retailers such as Amazon.com in the run-up to the holiday season, why can’t we successfully make a one-off visit to a flagship website run by the government of the wealthiest nation on the planet?
The Challenge in Statistical Terms
Talk to experts, and it’s clear that it’s precisely this latter question that cuts to the heart of the issue. Breaking new ground, planners had few facts, figures and statistics to guide them.
“The tricky bit about dealing with web-based services is that the arrival of pieces of work for the server is random: it’s a Poisson statistical distribution, with a standard deviation that equals its mean, and is ‘memoryless’,” says Prof. David Woodruff, of the graduate school of management at the University of California at Davis.
In other words, if the Poisson distribution suggests that the expected interval between the arrival of pieces of work for a server is 12 seconds, and a request hasn’t arrived after 15 seconds, then the expected interval doesn’t change—work is still 12 seconds away. Likewise, two pieces of work arriving within three seconds has no impact on the expected arrival of the next piece of work: it’s still 12 seconds away.
“The key point is that it’s a very, very random process,” Woodruff says. “There’s very little information that will help you to estimate the rate at which people will want to arrive at the site.”
Successfully reaching the HealthCare.gov site, though, was only the start of many users’ problems. In many cases, citizens would fill in requests for information, only to see screens go blank, or freeze.
Analyzing the Customer’s Workflow
But common sense also suggested taking the time to think-through what users would be actually trying to do, says Andreas Grabner, a technology strategist at Compuware, an application performance management vendor.
After which, comes usability testing. Again, say experts, dispassionate fact-based analysis can help.
“It’s always best to use third parties for usability testing: developers can get too close to the project,” says Martin Dempsey, client director at technology change consultancy Certeco. “Understand how long it takes a customer to get through each page, and where they get confused, and where they slow down. And in most cases, patterns emerge very quickly, pointing you to problems that can be fixed. Remember, it’s not about how well the technology works—it’s how easy it is to use, and how easy it is to complete the desired transaction.”
Another smart idea, says Compuware’s Grabner, might have been to simulate the surge in demand at launch, both in terms of total demand on the core system, and the distribution of that demand across states. State populations are known, and data is available on the percentage of those populations with online access, he points out.
“Lots of companies simulate the holiday shopping season; it’s certainly good practice,” Grabner notes.
So to what extent did those behind HealthCare.gov follow these best practices in modeling, design and testing? Answers, as yet, are unclear. On December 1, the Health and Human Services Department announced its HealthCare.gov team had eradicated more than 400 software bugs, introduced hardware, networking and database upgrades and had seen improvements in website responses, system stability and capacity, and a reduction in errors. It’s a safe bet, however, that finger-pointing and blame shifting will continue in the capital.
The good news—if that’s what it is—is that government IT officials and their contractors still have plenty of opportunity to put these best practices to good use when building HealthCare.gov.
For as recently as November 19, Henry Chao, deputy chief information officer at the Centers for Medicare and Medicaid Services, owner of the HealthCare.gov website, reportedly stunned a House of Representatives subcommittee with an acknowledgement that 30 to 40 percent of the HealthCare.gov portal’s accounting and payment systems have yet to be built.
Malcolm Wheatley, a freelance writer and contributing editor at Data Informed, is old enough to remember analyzing punched card datasets in batch mode, using SPSS on mainframes. He lives in Devon, England, and can be reached at email@example.com.