Allowing the biggest project in the news to go by without a comment would be unforgivable. All polemics aside, there is much to learn from the implementation of ACA in terms of project management and program expectations.
Most of the reasonable and informed blog posts concern the roll-out of the federal exchange website, but from a program management perspective this is too limiting. So where do we begin? Let’s start out with the much ballyhooed numbers. The most reliable source for enrollment reporting has been Charles Gaba’s ACASignups.net. Here is what he shows from the reported data:
Sub-26 year old enrollments in parent plans: 1.6M to 3.1M
Medicaid/CHIP expansion: 5.03M to 7.05M
Qualified Health Plan (QHPs) enrollment both on and off exchanges: 6.88M to 16.47M
Total new enrollments range: 13.5M to 26.6M
The reason for the range of enrollments in lieu of a hard number lies in the fact that many reported sign-ups are preliminary, firm numbers reported by the Department of Health and Human Services (HHS) lag in time, the enrollment period has been extended for those in mid-enrollment, some states are still counting, and not all insurance companies have reported off-exchange enrollment. The numbers are checked by a number of independent organizations including the Kaiser Family Foundation and the Urban Institute. When compared to the potential enrollment pool, the achievement is, indeed significant. Kaiser projected that the total potential enrollment pool across the United States was 36,440,000 people. Total new enrollments represent between 37% and 73% of that potential uninsured pool. From a program management perspective, if the total numbers were limited to the headline 7 million enrollment then the law’s implementation would be deemed problematic, but that is not the case.
Thus, there was much more to the scope of the ACA program than just the website. In addition, it appears that the flawed website roll-out was not a fatal risk in the law’s implementation. So why was that the case? Let us first look at what the law actually did.
In 2010, HHS was first tasked with enforcing the Patient’s Bill of Rights, which went into force that year. This is a mostly a compliance issue. It consisted of the following features: in terms of coverage it included the elimination of preexisting conditions exclusions for children under the age of 19, allows sub-26 year olds to be covered under their parents’ plans, elimination of arbitrary withdrawals of insurance, and the right to appeal determinations of non-payment. For costs it ended lifetime limits on coverage, provided for public review of premium increases, and limited that amount spent by insurance companies on administrative costs–limited to 15% of money collected for employer-provided healthcare and 20% for individual market policies. In terms of care, it ensured that patients could choose their primary care physician from the plan’s network, covered preventative care services without a co-pay, and removed barriers to emergency care from non-network hospitals. The act also expanded Medicaid eligibility to more people which, thanks later to a Supreme Court ruling, was left up to the states to accept or reject. As of this time, 25 states have accepted the Medicaid expansion.
In 2011, the law rolled out other features. These included a 50% discount Part D Medicare prescription drugs when patients have reached their coverage gap (known as the “Donut Hole”), providing preventative care for seniors under Medicare, reducing subsidies paid to Medicare Advantage insurance companies, establishing a Community Care Transitions Program to help senior connect to services in their communities to avoid unnecessary hospitalization, establishing a new Center for Medicare and Medicaid Innovation program, established an Independent Payment Advisory Board that was tasked with developing and submitting proposals to Congress and the President aimed at extending the life of the Medicare Trust Fund. and established a Community First Choice Option allowed states to offer home and community based services to disabled individuals through Medicaid rather than institutional care in nursing homes.
In 2012 most of the roll-out consisted of increasing payments to primary care physicians under Medicaid to be at parity with Medicare. There were also initiatives to improve quality and reduce administrative costs. Many people are unaware that the Centers for Medicare and Medicaid (CMS)–a government agency–returns to the taxpayer in the form of healthcare 98 cents of every tax dollar it collects. Thus, the agency is the standard for efficiency that is unmatched in the private insurance market.
Thus we can see that the ACA Rollout (euphemistically called Obamacare) had many aspects to it. In 2014 and 2015 there will be additional roll-outs of the law–additional open enrollment periods, the individual mandate, and employer mandates for those who have more than 50 employees.
In 2013, of course, the main event was the open enrollment period that began 1 October and just ended; and is extended for a limited number of people. In addition, people can enroll in healthcare plans outside of the open enrollment period due to certain major life changes and through off-exchange policies. The exchanges, rather than a “government takeover” were a means for qualified health plans to advertise and compare benefits in a competitive on-line marketplace and for qualifying individuals to apply for subsidies. By bringing insurers together in one place the desired result is to use market competition to bring healthcare costs down, which have run twice the per capita cost per person and as a percent of GDP over most other developed countries (most of whom have government-run single payer systems). Regardless, the mix of private and public healthcare in the current U.S. system under the ACA is most similar to Switzerland’s system. Under the exchange concept, states could choose to form their own exchange and agency to manage the implementation of the law, they could opt to run a state-federal government cooperative exchange, or they could leave the exchange (which is healthcare.gov) to HHS. For the first open enrollment period 17 states opted to run their own exchanges, seven opted to run a cooperative exchange, and 26 decided to rely fully upon the federal exchange.
The most thoughtful, though in hindsight premature and self-limiting, assessment of the ACA implementation that I’ve come across was posted by Loren Thompson in Forbes entitled “Healthcare.gov Diagnosis: The Government Broke Every Rule of Project Management.” There were others but this post was both the most thoughtful and specific. Lets take a look at Mr. Thompson’s specific points from both the perspective of the entire scope of the program and our present point in time.
a. The federal website “is central to implementation of the most sweeping reform of federal healthcare services in half a century.” Though in the heat of the moment that did seem to be the case, particularly for those listening to the partisans and the headlines and unfamiliar with the details of the law, this statement is patently false. Thus, Mr. Thompson’s initial assertion is flawed. As we have seen above, aspects of implementing the law in both 2010 and 2011 were well within HHS’s core competency, consisting of regulatory changes to the private healthcare market and expansions and adjustments to long-standing federal programs. The healthcare.org effort, in reality, turned out to be a sub-project within a larger programmatic effort. It was a key element of the open enrollment period but its issues did not affect programmatic success. This fact will become clearer as we run through the remainder of this analysis.
b. “Unrealistic requirements. This is the first time anybody has ever tried to develop a single web-site where diverse users could (1) establish an on-line identity, (2) review hundreds of health-insurance options, (3) enroll in a specific plan, and (4) determine eligibility for federal subsidies — all in real time….” To expect such a system to perform without error from the get-go probably would be unrealistic. But there seems to have been risk handling in place that acknowledges that the agency expected some glitches, especially when an unexpectedly large number of states decided to forgo their own exchanges, changing the initial scope significantly. The risk handling in this case was the establishment of local enrollment help centers a well as telephonic support. Mr. Thompson makes it seem as if this was an after-the-fact decision, but the issue of facilitators in state and local government offices was an area of controversy for those states that attempted to undermine the law well before the start of the open enrollment period. Thus, it appears that HHS’s risk, though unfortunately realized, was successfully handled and mitigated by all of the alternative methods of enrollment, including the state run exchanges that were deployed.
c. “Technical complexity. As often occurs with poorly-planned weapon projects, unrealistic requirements for HealthCare.gov resulted in an extraordinarily complicated system that is difficult to maintain. There are just too many moving pieces. A typical user might have to navigate 75 screens to get to their goal of obtaining insurance, and the whole system contains over a thousand screens. A total of 55 contractors were hired to produce the various pieces, and in order for all the steps to work CMS had to involve five federal agencies, 36 states, and 300 private-sector insurers offering well over 4,000 plans.” Mr. Thompson’s analysis essentially is saying: “doing this is very hard.” For someone who supposedly came from the defense analysis community he seems to forget that most complex projects have a great deal of technical complexity. I do not know if a typical user has to navigate 75 screens but from speaking with others who successfully signed up, it was much less than that, so he is again overreaching. The statement regarding the number of contractors and subcontractors, numbers of screens, insurers, and plans are all really big numbers intended to prove “technical complexity” but in the end don’t pass the Pisano “so-what?” test. Defense programs typically involve more elements than this and successfully deploy and maintain their systems all of the time, even when things don’t go swimmingly out of the gate.
d. “Integration responsibility. The government has difficulty maintaining organic expertise in information technologies, because the private sector often hires away the best talent. An executive at a big tech firm engaged in government contracting once remarked to me that whenever he meets with government project leaders, he always knows he is talking to the people industry didn’t want. Despite weak organic IT capabilities, though, CMS decided it would take charge of integrating all the parts in HealthCare.gov, and testing the end product to assure functionality. The results show why the military almost always hires outside companies to serve as lead integrator.” As I have written here previously, well over 90% of all IT projects in both the private and public sector either fail to meet expectations or fail outright. But the real fallacy in this statement once again lies in defining the program as the sub-project. CMS was the integrator for the program but subcontracted out the work to the private sector–a fact he provided in the paragraph above but self-contradicts here. The comment from the big tech firm is a gratuitous insult to project management professionals that is neither factual nor productive. We don’t hear government personnel using the latest tech roll-out failure–such as the buggy roll-out of Windows 8 or the latest version 6 Apple iPhone operating system–as indicative of the industry’s inability to hire the best people. This statement also ignores the fact that most of that “private” technology is well subsidized through patent monopolies and much of it originated from publicly-funded R&D projects.
e. “Fragmented authority. There seems to have been a great deal of infighting within CMS over how the web-site would operate and what the user experience would feel like. With three different parts of the bureaucracy contending for control — the IT shop, the policy shop, and the communications shop — key decisions were often delayed, guidance to contractors was inconsistent, and nobody was truly in charge. Government employees appear to have concealed critical information from each other, and on occasion mandated that certain features be implemented or suppliers be used despite contractor warnings that problems would result.” I have no doubt that some of this was probably true for the website development since it is usually found in almost all IT development projects. Changing requirements during development is a main factor in IT project failure as I have discussed and cited previously in this blog, but CMS seems to have recovered very quickly, so this was not the decisive systemic problem that Mr. Thompson posited when he wrote his critique. What is missing from Mr. Thompson’s critique from a programmatic perspective, however, were the external factors that created risk to system requirements. The legal challenges to the law, the changes brought about by most states rejecting to run their own exchanges, and the shift in the state Medicare mandate dictated that course corrections be made very late into the program, which also affected the website effort.
f. “Loose metrics. Perhaps the most important factor in keeping complex projects on track is for managers to utilize rigorous, unambiguous performance metrics in measuring progress. The government said in a report released on Sunday that it has made “improvements in the site’s key operating metrics over the last several weeks,” which is a tacit admission that it didn’t initially have adequate ways of measuring progress….” I think that Mr. Thompson confuses performance measure terminology here. The metrics that CMS reported as improving were the technical operating metrics, not performance management metrics. Given Mr. Thompson’s assertion that the schedule was aggressive below, it is probably certain that CMS had in place adequate metrics to determine project status. I will be addressing what they did use below.
g. “Inadequate testing. The Washington airports authority announced this week that it would delay opening a new subway line to Dulles Airport so that additional testing of software could be conducted, stating that its overriding goal is “safety.” The people overseeing HealthCare.gov clearly had a different management philosophy. Despite repeated warnings from contractors that more testing of system components was needed, CMS was determined to see the site go live on its planned debut date of October 1. Because important decisions about the site were still being made only days before this date, there was almost no end-to-end testing of the site before it became operational — which is why hundreds of software bugs had to be found and fixed later.” HHS and CMS certainly have some flexibility in the law’s implementation. But the subway example and the satellite example below simply are not valid comparisons. All software is buggy and no one was going to be placed in harm’s way by a faulty roll-out. Once a satellite is in space it is very hard to recover the program and they have been known to occur. I will respond fully to this conclusion below, but there is more than adequate documentation to show that the website components were tested prior to the roll-out. Mr. Thompson’s criticism, however, regarding an end-to-end test is valid, since it appears that initial product testing coincided with system roll-out. This is a symptom of the programmatic environment, however, not a systemic issue.
h. “Aggressive schedules. You wouldn’t think that standing up a web-site after literally years of planning might entail overly aggressive schedules, but in the case of HealthCare.gov the disorganized bureaucracy took so long to make design choices that the back end of the project was way too hurried for comfort. When the Pentagon develops a missile-warning or weather satellite, it sometimes delays launches for years to make sure all software and hardware issues are resolved. One vital satellite called the Space Based Infrared System was delayed for over a year due to concerns about software glitches; when the satellite finally reached orbit, though, it worked perfectly. CMS chose to stick with its schedule even as problems multiplied, and got a site that didn’t work.” As stated above, HHS and CMS had some flexibility in implementing the law, but the open enrollment process was not one of those areas. The CMS team used Agile processes in the website development, particularly in development of the front-end GUI and back-end data services hub. Agile development is (supposedly) designed to address projects that have hard stops–such as those often found in private industry for product roll-outs and in the case of healthcare.gov. There are documented problems with Agile from both a project management and software management perspective, which I will not go into now. But Mr. Thompson’s criticism is backwards. CMS did not deliver the detailed specifications for the sub-project to CGI Federal, who was awarded the contract in December 2011, until the spring of 2013. This left about 7 months for the team to deliver the final product. But–and this is a big “but”–this delay must be put into perspective by the disruptive government shutdowns, changing scope dictated by the Supreme Court ruling on the Medicare mandate, opposition from states in setting up exchanges, sequestration, and targeted reduced funding provided to the effort by Congressional opponents.
i. “Administrative blindness. The Center for Medicare & Medicaid Services may not have had good management practices or metrics for identifying problems, but that doesn’t mean it didn’t get plenty of warnings about potential problems with HealthCare.gov. Outside consultants and contractors on the project repeatedly warned government officials about functional difficulties with some features of the site, lack of adequate testing, poor protection of sensitive information, and the like. Sometimes CMS listened, but much of the time it was in denial about how defective the site was. It never adequately informed the White House about potential problems, and never subjected HealthCare.gov to systematic review until after the site went live and nearly collapsed.” In the end Mr. Thompson repeats his own false assertions here to support an additional one.
The stated programmatic objectives of the ACA are to extend healthcare to previously uninsured citizens and to reduce healthcare costs both per capita and as a percentage of GDP. The primary method of achieving this goal is through the healthcare exchanges. Access to the information on the exchanges was and is not limited to the functioning of the healthcare.org website, which recovered very quickly and effectively despite all of the issues that, at first, seemed to overwhelm it.
Thus, the initial assessment of this program is that it is too early to know if it will achieve a reduction in healthcare costs, particularly since there are still significant trade and patent monopoly barriers to price competition in this market. But we can know if it is achieving its other main goal of expanding coverage, especially in a period of slack job opportunities, creating headwinds to the achievement of this goal compounded by resistance by half of the states to Medicaid expansion. Thus far, in addition to the numbers cited above, the preliminary results are encouraging:
The ACA will no doubt be the subject of books and Harvard Business School cases studies in the years to come. My postmortem of the program at this point in time is that though the website effort was a failure upon roll-out, that it was only one (though a highly publicized and highly visible) element of the program, which was not decisive in determining program success or failure. The risk handling strategies employed were effective in this outcome. Its visibility and the resultant emphasis placed on it was probably decisive in the website’s rapid recovery. Finally, the number of people now covered by healthcare and benefiting from implementation of the law extends well beyond the highly publicized number of 7 million enrolled through the exchanges.
Update: Additional verification that one of the goals of the program have been met by the latest Gallup survey:
Life, Project Management, and Everything 
2 thoughts on “The ACA: Sub-Project Failure, Program Success”
Comments are closed.