More excellent reporting from the Washington Post about development of HealthCare.gov before October 1:

For that day and the next, CGI staff huddled with government officials in the semicircular conference room at the headquarters of the federal Centers for Medicare and Medicaid Services (CMS), the agency overseeing the project. They combed through 15 pages of spreadsheets they had brought, which spelled out the company’s level of confidence — high, medium or low — that individual components would be ready.

By the time HealthCare.gov launched 5¹/₂ weeks later, many of those predictions proved wrong, according to internal documents obtained by The Washington Post and officials familiar with the project.

A final “pre-flight checklist” before the Web site’s Oct. 1 opening, compiled a week before by CMS, shows that 41 of 91 separate functions that CGI was responsible for finishing by the launch were still not working.

And a spreadsheet produced by CGI, dated the day of the launch, shows that the company acknowledged about 30 defects on features scheduled to have been working already, including five that it classified as “critical.” For instance, one critical defect was that people who had finished creating applications — an early step in enrolling — got incorrect messages that their applications were incomplete if they tried to sign back in.

All told, of the 45 items in which CGI had expressed high confidence at the late August meeting in Baltimore, most were still not ready by the time consumers were supposed to be able to start to buy health plans online through the federal marketplace, according to a government official familiar with the project who spoke on the condition of anonymity to discuss private information.

During those crucial final weeks before the marketplace opened, the official said, CGI often delivered components on time, but they contained such faulty computer code that features did not hold up under closer scrutiny — or failed later if more than several thousand people at a time tried to use them. These included essential but arcane parts of the Web site, as well as facets that have attracted substantial public and congressional attention, such as a feature — still not working — that was supposed to let insurance-seekers browse the health plans available to them without first registering for an online account.

But not all the blame falls on CGI. Much must rest with politics:

Work to build the online marketplace was hindered, in part, by White House micromanagement and political sensitivities that delayed policy and regulatory decisions, fierce Republican opposition to the law, and the fact that no one at the CMS or elsewhere at Department of Health and Human Services, of which the agency is a part, had the job of managing the project full time.

The Post also notes that the anonymous feature that would allow “window shopping”–checking prices without having to register, was spiked because of technical problems.

The “anonymous shopping” feature — in which consumers were to be able to browse health plans without creating an account — also was rated as a high-confidence item. It never made it onto the “pre-flight” assessment because, by then, government officials had decided that it was too flawed to offer at the outset.

A senior administration official, who asked not to be identified because of the sensitivity of the matter, said he was not aware of any indication the “pre-flight” review ever reached the White House.

Though earlier reports suggest that it was a political decision  in nature, to avoid sticker shock.

As the Post wrote in an earlier piece, the political constraints were strong:

Based on interviews with more than two dozen current and former administration officials and outsiders who worked alongside them, the project was hampered by the White House’s political sensitivity to Republican hatred of the law — sensitivity so intense that the president’s aides ordered that some work be slowed down or remain secret for fear of feeding the opposition. Inside the Department of Health and Human Services’ Centers for Medicare and Medicaid, the main agency responsible for the exchanges, there was no single administrator whose full-time job was to manage the project. Republicans also made clear they would block funding, while some outside IT companies that were hired to build the Web site, HealthCare.gov, performed poorly.

These interwoven strands ultimately caused the exchange not to be ready by its Oct. 1 start date. It was not ready even though, on the balmy Sunday evening of March 21, 2010, hours after the bill had been enacted, the president had stood on the Truman Balcony for a champagne toast with his weary staff and put them on notice: They needed to get started on carrying out the law the very next morning. It was not ready even though, for months beginning last spring, the president emphasized the exchange’s central importance during regular staff meetings to monitor progress. No matter which aspects of the sprawling law had been that day’s focus, the official said, Obama invariably ended the meeting the same way: “All of that is well and good, but if the Web site doesn’t work, nothing else matters.”

…

According to two former officials, CMS staff members struggled at “multiple meetings” during the spring of 2011 to persuade White House officials for permission to publish diagrams known as “concepts of operation,” which they believed were necessary to show states what a federal exchange would look like. The two officials said the White House was reluctant because the diagrams were complex, and they feared that the Republicans might reprise a tactic from the 1990s of then-Sen. Bob Dole (R-Kan.), who mockingly brandished intricate charts created by a task force led by first lady Hillary Clinton.

In the end, one of the former officials said, the White House quashed the diagrams, telling CMS, instead, to praise early work on those state exchanges that matched the hidden federal thinking

You can read more about the process in these released emails, which the Post summarizes.

On Monday, July 8, a CMS technology adviser named Jeffrey Grant fired off an e-mail titled “IT support” to two colleagues at 3:15 p.m., warning, with respect to a vital part of the project, “our entire build is in jeopardy” because he had just learned from colleagues that CGI had fewer people working on the Web site’s construction than government officials had been led to believe. Grant was referring to the Web site’s financial management system, which handles payments of government subsidies to insurers; he said it “appears to be way off track and getting worse.”

About three hours later, Henry Chao, the agency’s deputy information officer, pivoted and wrote to a group of CGI employees: “Can you get me a response on this to refute what Jeff is saying?” By later that month, Chao’s tone had become more exasperated. At 9:51 p.m. on July 29, he wrote to a group of CGI and CMS employees: “We are in bad shape. Perhaps worse than ever before and we are not even touching the hard stuff yet.”

In another life, I worked for the Department of Defense, and supervised  IT contractors. It is not a fun process. They always run behind schedule, are over budget, are not accountable, and can’t fix things quickly. This process is exacerbated because the government procurement process is a mess, with constantly changing requirements. Injecting the political dynamics into this process makes the requirements even more unbearable. I have a lot of sympathy for the inability of CGI to get this built.

Were this any other IT project, it simply would have been delayed. But because this was the cornerstone of a presidential administration, it had to be launched to prove that it would work. This was foolhardy. And the project will suffer a serious credibility gap for the foreseeable future as they scramble to make things work.

Update: The New York Times has more.

Interviews with current and former Obama administration officials and specialists involved in the project, as well as a review of hundreds of pages of government and contractor documents, offer new details into how tensions between the government and its contractors, questionable decisions and weak leadership within the Medicare agency turned the rollout of the president’s signature program into a major humiliation.

The online exchange was crippled, people involved with building it said in recent interviews, because of a huge gap between the administration’s grand hopes and the practicalities of building a website that could function on opening day.

Vital components were never secured, including sufficient access to a data center to prevent the website from crashing. A backup system that could go live if it did crash was not created, a weakness the administration has never disclosed. And the architecture of the system that interacts with the data center where information is stored is so poorly configured that it must be redesigned, a process that experts said typically takes months. An initial assessment identified more than 600 hardware and software defects — “the longest list anybody had ever seen,” one person involved with the project said.

But in contrast with WaPo, this report suggests there was pervasive White House “oversight”:

As a result, the president’s signature initiative was effectively left under the day-to-day management of Henry Chao, a 19-year veteran of the Medicare agency with little clout and little formal background in computer science.

Mr. Chao had to consult with senior department officials and the White House, and was unable to make many decisions on his own. “Nothing was decided without a conversation there,” said one agency official involved in the project, referring to the constant White House demands for oversight. On behalf of Mr. Chao, the Medicare agency declined to comment.

Update: CNN reports that the Anonymous Shopper feature was shelved even though it passed tests.

When the troubled federal health care website came online, the key “Anonymous Shopper” function was nowhere to be found — even though it passed a key test almost two weeks before HealthCare.gov launched.

That successful test, noted in documents obtained by CNN and confirmed by a source close to the project, contradicts testimony from an Obama administration official overseeing HealthCare.gov, who told lawmakers earlier this month the function was scrapped because it “failed miserably” before the October 1 launch.

Like much of the HealthCare.gov rollout, the subject has become political fodder for Republicans, who claim the decision to nix the anonymous shopper was made by administration officials worried it would produce rate estimates so high they would deter potential enrollees.

…

So why was online window shopping shelved for HealthCare.gov? That was the question lawmakers posed to Henry Chao, the top technology officer for the Centers for Medicare & Medicaid Services, a subsidiary of HHS that helped build theHealthCare.gov website.

Chao said he made the decision in conjunction with colleagues and testified before Congress last week that it was because the feature “failed so miserably that we could not conscionably let people use it.”

Yet a CMS document made public by the same committee last week tells a different story. The agency and one of its subsidiaries, the Center for Consumer Information and Insurance Oversight, was working with government contractors on the website. It determined the Anonymous Shopper feature “tested successfully,” revealed “no high severity defects open” and that “remaining lower severity defects will not degrade consumer experience.”

CMS raised questions about the “tested successfully” denotation for the feature in a statement.

In it, a spokeswoman writes: “‪CMS believes that the ‘yes’ that is written on the document in question is likely an error, because the same document also lists a number of ongoing defects and problems with the tool. Additional defects were communicated and discussed in other settings.”

‪The source close to the project, however, said the anonymous shopper function did pass testing conducted in the weeks ahead of the HealthCare.gov launch.

“This document reflects one point in time that was part of a series of ongoing updates and monitoring,” Joanne Peters, a spokeswoman for HHS, said in a written statement.

The successful test occurred on September 17, according to a source familiar with the project. The next day, in an internal e-mail obtained by CNN, Chao wrote the shopper function “isn’t needed and thus should be removed.”

During an interview with the House Oversight and Government Reform Committee on November 1, Chao said there were 20 outstanding defects with the anonymous shopper that prevented it from making it online when the health care website launched.

But according to a list that CNN obtained, only 12 defects remained when the decision to shelve HealthCare.gov was made.

Darrell Issa is convinced this was a political decision, but there is no evidence of this:

They accuse the Obama administration of sidelining the feature to hide the sticker shock of insurance plans and force Americans to jump through hoops before they can shop.

“Although, CGI officials (the contractor for the shopping tool) were not able to identify who within the administration made the decision to disable the anonymous shopping feature, evidence is mounting that political considerations motivated the decision,” Rep. Darrell Issa, the Republican chairman of the House Oversight and Government Reform Committee, wrote in a leader to federal tech executives in October.

So far, no document or testimony has revealed White House involvement in the anonymous shopper decision, and privately, some Republicans admit the accusations the decision was made inside the West Wing is motivated by politics more than evidence.