technical performance measurement

Technical Foul — It’s Time for TPI in EVM

For more than 40 years the discipline of earned value management (EVM) has gone through a number of changes in its descriptions, governance, and procedures.  During that same time its community has been resistant to improvements in its methodology or to changes that extend its value when taking into account other methods that either augment its usefulness, or that potentially provide more utility in the area of performance management.  This has been especially the case where it is suggested that EVM is just one of many methodologies that contribute to this assessment under a more holistic approach.

Instead, it has been asserted that EVM is the basis for integrated project management.  (I disagree–and solely on the evidence that if it was so, then project managers would more fully participate in its organizations and conferences.  This would then pose the problem that PMs might then propose changes to EVM that, well…default to the second sentence in this post).  As evidence it need only be mentioned that there has been resistance to such recent developments in using earned schedule, technical performance, and risk–most especially risk based on Bayesian analysis).

Some of this resistance is understandable.  First, it took quite a long time just to get to a consensus on the application of EVM, though its principles and methods are based on simple and well proven statistical methods.  Second, the industries in which EVM has been accepted are sensitive to risk, and so a bureaucracy of practitioners have grown to ensure both consensus and compliance to accepted methods.  Third, the community that makes up practitioners of EVM consist mostly of cost analysts, trained in simple accounting, arithmetic, and statistical methodology.  It is thus a normal human bias to assume that the path of one’s previous success is the way to future success, though our understanding of the design space (reality) that we inhabit has been enhanced through new knowledge.  Fourth, there is a lot of data that applies to project management, and the EVM community is only now learning of the ways that this other data impacts our understanding of measuring project performance and the probability of reaching project goals in rolling out a product.  Finally, there is the less defensible reason that a lot of people and firms have built their careers that depends on maintaining the status quo.

Our ability to integrate disparate datasets is accelerating on a yearly basis thanks to digital technology, and the day in achieving integration of all relevant factors in project and enterprise performance is inevitable.  To be frank, I am personally engaged in such projects and am assisting organizations in moving in this direction today.  Regardless, we can make an advance in the discipline of performance management by pulling down low hanging fruit.  The most reachable one, in my opinion, is technical performance measurement.

The literature of technical performance has come quite a long way, thanks largely to the work of the Institute for Defense Analyses (IDA) and others, particularly the National Defense Industrial Association through the publication of their predictive measures guide.  This has been a topic of interest to me since its study was part of my duties back when I was still wearing a uniform.  The early results of these studies resulted in a paper that proposed a method of integrating technical performance, earned value, and risk.  A pretty comprehensive overview of the literature and guidance for technical performance can be found at this presentation by Glen Alleman and Tom Coonce given at EVM World in 2015.  It must be mentioned that Rick Price of Lockheed Martin also contributed greatly to this literature.

Keep in mind what is meant when we decide to assess technical performance within the context of R&D.  It is an assessment against expected or specified:

a.  Measures of Effectiveness (MoE)

b.  Measures of Performance (MoP), and

c.  Key Performance Parameters (KPP)

The opposition from the project management community to widespread application of this methodology took two forms.  First, it was argued, the method used to adjust the value of earned (CPI) seemed always to have a negative impact.  Second, there are technical performance factors that transcend the WBS, and so it is hard to properly adjust the individual control accounts based on the contribution of technical performance.  Third, some performance measures defy an assessment of value in a time-phased manner.  The most common example has been tracking weight of aircraft, which has contributors from virtually all components that go into it.

Let’s take these in order.  But lest one think that this perspective is an artifact from 1997, just a short while ago, in the A&D community, the EVM policy office at DoD attempted to apply a somewhat modest proposal of ensuring that technical performance was included as an element in EVM reporting.  Note that the EIA 748 standard states this clearly and has done so for quite some time.  Regardless, the same three core objections were raised in comments from the industry.  Thus, this caused me to ask some further in-depth questions and my revised perspective follows below.

The first condition occurred, in many cases, due to optimism bias in registering earned value, which often occurs when using a single point estimate of percent complete by a limited population of experts contributing to an assessment of the element.  Fair enough, but as you can imagine, its not a message that a PM wants to hear or will necessarily accept or admit, regardless of the merits.  There are more than enough pathways to second guessing and testing selection bias at other levels of reporting.  Glen Alleman in his Herding Cats blog post of 12 August has a very good post listing the systemic reasons for program failure.

Another factor is that the initial methodology did possess a skewing toward more pessimistic results.  This was not entirely apparent at the time because the statistical methods applied did not make that clear.  But, to critique that first proposal, which was the result of contributions from IDA and other systems engineering technical experts, the 10-50-90 method in assessing probability along the bandwidth of the technical performance baseline was too inflexible.  The graphic that we proposed is as follows and one can see that, while it was “good enough”, if rolled up there could be some bias that required adjustment.

TPM Graphic

 

Note that this range around 50% can be interpreted to be equivalent to the bandwidth found in the presentation given by Alleman and Coonce (as well as the Predictive Measures Guide), though the intent here was to perform an assessment based on a simplified means of handicapping the handicappers–or more accurately, performing a probabilistic assessment on expert opinion.  The method of performing Bayesian analysis to achieve this had not yet matured for such applications, and so we proposed a method that would provide a simple method that our practitioners could understand that still met the criteria of being a valid approach.  The reason for the difference in the graphic resides in the fact that the original assessment did not view this time-phasing as a continuous process, but rather an assessment at critical points along the technical baseline.

From a practical perspective, however, the banding proposed by Alleman and Coonce take into account the noise that will be experienced during the life cycle of development, and so solves the slight skewing toward pessimism.  We’ll leave aside for the moment how we determine the bands and, thus, acceptable noise as we track along our technical baseline.

The second objection is valid only so far as any alignment of work-related indicators vary from project to project.  For example, some legs of the WBS tree go down nine levels and others go down five levels, based on the complexity of the work and the organizational breakdown structure (OBS).  Thus where we peg within each leg of the tree the control account (CA) and work package (WP) level becomes relative.  Do the schedule activities have a one-to-one relationship or many-to-one relationship with the WP level in all legs?  Or is the lowest level that the alignment can be made in certain legs at the CA level?

Given that planning begins with the contract spec and (ideally) proceed from IMP –> IMS –> WBS –> PMB in a continuity, then we will be able to determine the contributions of TPM to each WBS element at their appropriate level.

This then leads us to another objection, which is that not all organizations bother with developing an IMP.  That is a topic for another day, but whether such an artifact is created formally or not, one must achieve in practice the purpose of the IMP in order to get from contract spec to IMS under a sufficiently complex effort to warrant CPM scheduling and EVM.

The third objection is really a child of the second objection.  There very well may be TPMs, such as weight, with so many contributors that distributing the impact would both dilute the visibility of the TPM and present a level of arbitrariness in distribution that would render its tracking useless.  (Note that I am not saying that the impact cannot be distributed because, given modern software applications, this can easily be done in an automated fashion after configuration.  My concern is in regard to visibility on a TPM that could render the program a failure).  In these cases, as with other indicators that must be tracked, there will be high level programmatic or contract level TPMs.

So where do we go from here?  Alleman and Coonce suggest adjusting the formula for BCWP, where P is informed by technical risk.  The predictive measures guide takes a similar approach and emphasizes the systems engineering (SE) domain in getting to an assessment to determine the impact of reported EVM element performance.  The recommendation of the 1997 project that I headed in assignments across Navy and OSD, was to inform performance based on a risk assessment of probable achievement at each discrete performance milestone.  What all of these studies have in common, and in common with common industry practice using SE principles, is an intermediate assessment, informed by risk, of a technical performance index against a technical performance baseline.

So let’s explore this part of the equation more fully.

Given that we have MoE, MoP, and KPP are identified for the project, different methods of determining progress apply.  This can be a very simplistic set of TPMs that, through the acquisition or fabrication of compliant materials, meet contractual requirements.  These are contract level TPMs.  Depending on contract type, achievement of these KPPs may result in either financial penalties or financial reward.  Then there are the R&D-dependent MoEs, MoPs, and KPPs that require more discrete time-phasing and ties to the physical completion of work documented by through the WBS structure.  As with EVM on the measurement of the value of work, our index of physical technical achievement can be determined through various methods: current EVM methods, simulated Monte Carlo technical risk, 10-50-90 risk assessment, Bayesian analysis, etc.  All of these methods are designed to militate against selection bias and the inherent limitations of limited sample size and, hence, extreme subjectivity.  Still, expert opinion is a valid method of assessment and (in cases where it works) better than a WAG or coin flip.

Taken together these TPMs can be used to determine the technical achievement of the project or program over time, with a financial assessment of the future work needed to bring it in line.  These elements can be weighted, as suggested by Coonce, Alleman, and Price, through an assessment of relative risk to project success.  Some of these TPIs will apply to particular WBS elements at various levels (since their efforts are tied to specific activities and schedules via the IMS), and the most important project and program-level TPMs are reflected at that level.

What about double counting?  A comparison of the aggregate TPIs and the aggregate CPI and SPI will determine the fidelity of the WBS to technical achievement.  Furthermore, a proper baseline review will ensure that double counting doesn’t occur.  If the element can be accounted for within the reported EVM elements, then it need not be tracked separately by a TPI.  Only those TPMs that cannot be distributed or that represent such overarching risk to project success need be tracked separately, with an overall project assessment made against MR or any reprogramming budget available that can bring the project back into spec.

My last post on project management concerned the practices at what was called Google X.  There incentives are given to teams that identify an unacceptably high level of technical risk that will fail to pay off within the anticipated planning horizon.  If the A&D and DoD community is to become more nimble in R&D, it needs the necessary tools to apply such long established concepts such as Cost-As-An-Independent-Variable (CAIV), and Agile methods (without falling into the bottomless pit of unsupported assertions by the cult such as elimination of estimating and performance tracking).

Even with EVM, the project and program management community needs a feel for where their major programmatic efforts are in terms of delivery and deployment, in looking at the entire logistics and life cycle system.  The TPI can be the logic check of whether to push ahead, or finishing the low risk items that are remaining in R&D to move to first item delivery, or to take the lessons learned from the effort, terminate the project, and incorporate those elements into the next generation project or related components or systems.  This aligns with the concept of project alignment with framing assumptions as an early indicator of continued project investment at the corporate level.

No doubt, existing information systems, many built using 1990s technology and limited to line-and-staff functionality, do not provide the ability to do this today.  Of course, these same systems do not take into account a whole plethora of essential information regarding contract and financial management: from the tracking of CLINs/SLINs, to work authorization and change order processing, to the flow of funding from TAB to PMB/MR and from PMB to CA/UB/PP, contract incentive threshold planning, and the list can go on.  What this argues for is innovation and rewarding those technology solutions that take a more holistic approach to project management within its domain as a subset of program, contract, and corporate management–and such solutions that do so without some esoteric promise of results at some point in the future after millions of dollars of consulting, design, and coding.  The first company or organization that does this will reap the rewards of doing so.

Furthermore, visibility equals action.  Diluting essential TPMs within an overarching set of performance metrics may have the effect of hiding them and failing to properly identify, classify, and handle risk.  Including TPI as an element at the appropriate level will provide necessary visibility to get to the meat of those elements that directly impact programmatic framing assumptions.

River Deep, Mountain High — A Matrix of Project Data

Been attending conferences and meetings of late and came upon a discussion of the means of reducing data streams while leveraging Moore’s Law to provide more, better data.  During a discussion with colleagues over lunch they asked if asking for more detailed data would provide greater insight.  This led to a discussion of the qualitative differences in data depending on what information is being sought.  My response to more detailed data was to respond: “well there has to be a pony in there somewhere.”  This was greeted by laughter, but then I finished the point: more detailed data doesn’t necessarily yield greater insight (though it could and only actually looking at it will tell you that, particularly in applying the principle of KDD).  But more detailed data that is based on a hierarchical structure will, at the least, provide greater reliability and pinpoint areas of intersection to detect areas of risk manifestation that is otherwise averaged out–and therefore hidden–at the summary levels.

Not to steal the thunder of new studies that are due out in the area of data later this spring but, for example, I am aware after having actually achieved lowest level integration for extremely complex projects through my day job, that there is little (though not zero) insight gained in predictive power between say, the control account level of a WBS and the work package level.  Going further down to element of cost may, in the words of the character in the movie Still Alice, where “You may say that this falls into the great academic tradition of knowing more and more about less and less until we know everything about nothing.”  But while that may be true for project management, that isn’t necessarily so when collecting parametrics and auditing the validity of financial information.

Rolling up data from individually detailed elements of a hierarchy is the proper way to ensure credibility.  Since we are at the point where a TB of data has virtually the same marginal cost of a GB of data (which is vanishingly small to begin with), then the more the merrier in eliminating the abuse associated with human-readable summary reporting.  Furthermore, I have long proposed through this blog and elsewhere, that the emphasis should be away from people, process, and tools, to people, process, and data.  This rightly establishes the feedback loop necessary for proper development and project management.  More importantly, the same data available through project management processes satisfy the different purposes of domains both within the organization, and of multiple external stakeholders.

This then leads us to the concept of integrated project management (IPM), which has become little more than a buzz-phrase, and receives a lot of hand waves, mostly by technology companies that want to push their tools–which are quickly becoming obsolete–while appearing forward leaning.  This tool-centric approach is nothing more than marketing–focusing on what the software manufacturer would have us believe is important based on the functionality baked into their applications.  One can see where this could be a successful approach, given the emphasis on tools in the PM triad.  But, of course, it is self-limiting in a self-interested sort of way.  The emphasis needs to be on the qualitative and informative attributes of available data–not of tool functionality–that meet the requirements of different data consumers while minimizing, to the extent possible, the number of data streams.

Thus, there are at least two main aspects of data that are important in understanding the utility of project management: early warning/predictiveness and credibility/traceability/fidelity.  The chart attached below gives a rough back-of-the-envelope outline of this point, with some proposed elements, though this list is not intended to be exhaustive.

PM Data Matrix

PM Data Matrix

In order to capture data across the essential elements of project management, our data must demonstrate both a breadth and depth that allows for the discovery of intersections of the different elements.  The weakness in the two-dimensional model above is that it treats each indicator by itself.  But, when we combine, for example, IMS consecutive slips with other elements listed, the informational power of the data becomes many times greater.  This tells us that the weakness in our present systems is that we treat the data as a continuity between autonomous elements.  But we know that the project consists of discontinuities where the next level of achievement/progress is a function of risk.  Thus, when we talk about IPM, the secret is in focusing on data that informs us what our systems are doing.  This will require more sophisticated types of modeling.

Technical Ecstacy — Technical Performance and Earned Value

As many of my colleagues in project management know, I wrote a series of articles on the application of technical performance risk in project management back in 1997, one of which made me an award recipient from the institution now known as Defense Acquisition University.  Over the years various researchers and project organizations have asked me if I have any additional thoughts on the subject and the response up until now has been: no.  From a practical standpoint, other responsibilities took me away from the domain of determining the best way of recording technical achievement in complex projects.  Furthermore, I felt that the field was not ripe for further development until there were mathematics and statistical methods that could better approach the behavior of complex adaptive systems.

But now, after almost 20 years, there is an issue that has been nagging at me since publication of the results of the project studies that I led from 1995 through 1997.  It is this: the complaint by project managers in resisting the application of measuring technical achievement of any kind, and integrating it with cost performance, the best that anyone can do is 100%.  “All TPM can do is make my performance look worse”, was the complaint.  One would think this observation would not only not face opposition, especially from such an engineering dependent industry, but also because, at least in this universe, the best you can do is 100%.*  But, of course, we weren’t talking about the same thing and I have heard this refrain again at recent conferences and meetings.

To be honest, in our recommended solution in 1997, we did not take things as far as we could have.  It was always intended to be the first but not the last word regarding this issue.  And there have been some interesting things published about this issue recently, which I noted in this post.

In the discipline of project management in general, and among earned value practitioners in particular, the performance being measured oftentimes exceeds 100%.  But there is the difference.  What is being measured as exceeding 100% is progress against both a time-based and fiscally-based linear plan.  Most of the physical world doesn’t act nor can it be measured this way.  When measuring the attributes of a system or component against a set of physical or performance thresholds, linearity against a human-imposed plan oftentimes goes out the window.

But a linear progression can be imposed on the development toward the technical specification.  So then the next question is how do we measure progress during the development curve and duration.

The short answer, without repeating a summarization of the research (which is linked above) is through risk assessment, and the method that we used back in 1997 was a distribution curve that determined the probability of reaching the next step in the technical development.  This was based on well-proven systems engineering techniques that had been used in industry for many years, particularly at pre-Lockheed Martin Martin Marietta.  Technical risk assessment, even using simplistic 0-50-80-100 curves, provides a good approximation of probability and risk between each increment of development, though now there are more robust models.  For example, the use of Bayesian methodology, which introduces mathematical rigor into statistics, as outlined in this post by Eliezer Yudkowsky.  (As an aside, I strongly recommend his blogs for anyone interested in the cutting edge of rational inquiry and AI).

So technical measurement is pretty well proven.  But the issue that then presents itself (and presented itself in 1997) was how to derive value from technical performance.  Value is a horse of a different color.  The two bugaboos that were presented as being impassible roadblocks were weight and test failure.

Let’s take weight first.  On one of my recent trips I found myself seated in an Embraer E-jet.  These are fairly small aircraft, especially compared to conventional commercial aircraft, and are lightweight.  As such, they rely on a proper distribution and balance of weight, especially if one finds oneself at 5,000 feet above sea level with the long runway shut down, a 10-20 mph crosswind, and a mountain range rising above the valley floor in the direction of takeoff.  So the flight crew, when the cockpit noted a weight disparity, shifted baggage from belly stowage to the overhead compartments in the main cabin.  What was apparent is that weight is not an ad hoc measurement.  The aircraft’s weight distribution and tolerances are documented–and can be monitored as part of operations.

When engineering an aircraft, each component is assigned its weight.  Needless to say, weight is then allocated and measured as part of the development of subsystems of the aircraft.  One would not measure the overall weight of the aircraft or end item without ensuring that the components and subsystems did not conform to the weight limitations.  The overall weight limitation of an aircraft will very depending on mission and use.  If a commercial-type passenger airplane built to takeoff and land from modern runways, weight limitations are not as rigorous.  If the aircraft in question is going to takeoff and land from a carrier deck at sea then weight limitations become more critical.  (Side note:  I also learned these principles in detail while serving on active duty at NAS Norfolk and working with the Navy Air Depot there).  Aside from aircraft weight is important in a host of other items–from laptops to ships.  In the latter case, of which I am also intimately familiar, weight is important in balancing the ship and its ability to make way in the water (and perform its other missions).

So given that weight is an allocated element of performance within subsystem or component development, we achieve several useful bits of information.  First off, we can aggregate and measure weight of the entire end item to track if we are meeting the limitations of the item.  Secondly, we can perform trade-off.  If a subsystem or component can be made with a lighter material or more efficiently weight-wise, then we have more leeway (maybe) somewhere else.  Conversely, if we need weight for balance and the component or subsystem is too light, we need to figure out how to add weight or ballast.  So measuring and recording weight is not a problem. Finally, we allocate and tie performance-wise a key technical specification to the work, avoiding subjectivity.

So how to do we show value?  We do so by applying the same principles as any other method of earned value.  Each item of work is covered by a Work Breakdown Structure (WBS), which is tied (hopefully) to an Integrated Master Schedule (IMS).  A Performance Management Baseline (PMB) is applied to the WBS (or sometimes thought a resource-loaded IMS).  If we have properly constructed our Integrated Management Plan (IMP) prior to the IMS, we should clearly have tied the relationship of technical measures to the structure.  I acknowledge that not every program performs an IMP, but stating so is really an acknowledgement of a clear deficiency in our systems, especially involving complex R&D programs.  Since our work is measured in short increments against a PMB, we can claim 100% of a technical specification but be ahead of plan for the WBS elements involved.

It’s not as if the engineers in our industrial activities and aerospace companies have never designed a jet aircraft or some other item before.  Quite a bit of expertise and engineering know-how transfers from one program to the next.  There is a learning curve.  The more information we collect in that regard, the more effective that curve.  Hence my emphasis in recent posts on data.

For testing, the approach is the same.  A test can fail, that is, a rocket can explode on the pad or suffer some other mishap, but the components involved will succeed or fail based on the after-action report.  At that point we will know, through allocation of the test results, where we are in terms of technical performance.  While rocket science is involved in the item’s development, recording technical achievement is not rocket science.

Thus, while our measures of effectiveness, measures of performance, measures of progress, and technical performance will determine our actual achievement against a standard, our fiscal assessment of value against the PMB can still reflect whether we are ahead of schedule and below budget.  What it takes is an understanding of how to allocate more rigorous measures to the WBS that are directly tied to the technical specifications.  To do otherwise is to build a camel when a horse was expected or–as has been recorded in real life in previous programs–to build a satellite that cannot communicate, a Navy aircraft that cannot land on a carrier deck, a ship that cannot fight, and a vaccine that cannot be delivered and administered in the method required.  We learn from our failures, and that is the value of failure.

 

*There are colloquial expressions that allow for 100% to be exceeded, such as exceeding 100% of the tolerance of a manufactured item or system, which essentially means to exceed its limits and, therefore, breaking it.

Days of Future Passed — Legacy Data and Project Parametrics

I’ve had a lot of discussions lately on data normalization, including being asked the question of what constitutes normalization when dealing with legacy data, specifically in the field of project management.  A good primer can be found at About.com, but there are also very good older papers out on the web from various university IS departments.  The basic principals of data normalization today consist of finding a common location in the database for each value, reducing redundancy, properly establishing relationships among the data elements, and providing flexibility so that the data can be properly retrieved and further processed into intelligence in such as way as the objects produced possess significance.

The reason why answering this question is so important is because our legacy data is of such a size and of such complexity that it falls into the broad category of Big Data.  The condition of the data itself provides wide variations in terms of quality and completeness.  Without understanding the context, interrelationships, and significance of the elements of the data, the empirical approach to project management is threatened, since being able to use this data for purposes of establishing trends and parametric analysis is limited.

A good paper that deals with this issue was authored by Alleman and Coonce, though it was limited to Earned Value Management (EVM).  I would argue that EVM, especially in the types of industries in which the discipline is used, is pretty well structured already.  The challenge is in the other areas that are probably of more significance in getting a fuller understanding of what is happening in the project.  These areas of schedule, risk, and technical performance measures.

In looking at the Big Data that has been normalized to date–and I have participated with others in putting a significant dent in this area–it is apparent that processes in these other areas lack discipline, consistency, completeness, and veracity.  By normalizing data in sub-specialties that have experienced an erosion in enforcing standards of quality and consistency, technology becomes a driver for process improvement.

A greybeard in IT project management once said to me (and I am not long in joining that category): “Data is like water, the more it flows downstream the cleaner it becomes.”  What he meant is that the more that data is exposed in the organizational stream, the more it is questioned and becomes a part of our closed feedback loop: constantly being queried, verified, utilized in decision making, and validated against reality.  Over time more sophisticated and reliable statistical methods can be applied to the data, especially if we are talking about performance data of one sort or another, that takes periodic volatility into account in trending and provides us with a means for ensuring credibility in using the data.

In my last post on Four Trends in Project Management, I posited that the question wasn’t more or less data but utilization of data in a more effective manner, and identifying what is significant and therefore “better” data.  I recently heard this line repeated back to me as a means of arguing against providing data.  This conclusion was a misreading of what I was proposing.  One level of reporting data in today’s environment is no more work than reporting on any other particular level of a project hierarchy.  So cost is no longer a valid point for objecting to data submission (unless, of course, the one taking that position must admit to the deficiencies in their IT systems or the unreliability of their data).

Our projects must be measured against the framing assumptions in which they were first formed, as well as the established measures of effectiveness, measures of performance, and measures of technical achievement.  In order to view these factors one must have access to data originating from a variety of artifacts: the Integrated Master Schedule, the Schedule and Cost Risk Analysis, and the systems engineering/technical performance plan.  I would propose that project financial execution metrics are also essential in getting a complete, integrated, view of our projects.

There may be other supplemental data that is necessary as well.  For example, the NDIA Integrated Program Management Division has a proposed revision to what is known as the Integrated Baseline Review (IBR).  For the uninitiated, this is a process in which both the supplier and government customer project teams can come together, review the essential project artifacts that underlie project planning and execution, and gain a full understanding of the project baseline.  The reporting systems that identify the data that is to be reported against the baseline are identified and verified at this review.  But there are also artifacts submitted here that contain data that is relevant to the project and worthy of continuing assessment, precluding manual assessments and reviews down the line.

We don’t yet know the answer to these data issues and won’t until all of the data is normalized and analyzed.  Then the wheat from the chaff can be separated and a more precise set of data be identified for submittal, normalized and placed in an analytical framework to give us more precise information that is timely so that project stakeholders can make decisions in handling any risks that manifest themselves during the window that they can be handled (or make the determination that they cannot be handled).  As the farmer says in the Chinese proverb:  “We shall see.”

Over at AITS.org — Red Queen Race: Why Fast Tracking a Project is Not in Your Control

“Well, in our country,” said Alice, still panting a little, “you’d generally get to somewhere else—if you run very fast for a long time, as we’ve been doing.”

“A slow sort of country!” said the Queen. “Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast, as that!” —Through the Looking-Glass and What Alice Found There, Chapter 2, Lewis Carroll

There have been a number of high profile examples in the news over the last two years concerning project management.  For example, the initial rollout of the Affordable Care Act marketplace web portal was one of these, and the causes for its faults are still being discussed. As I write this, an article in the New York Times indicates that the fast track efforts to create an Ebola vaccine are faltering…

To read the remainder of this post please to go to this link.

No Bucks, No Buck Rogers — Project Work Authorizations, Change Control, and Cash Flow

As I’ve written here most recently, the most significant proposal coming out of the Integrated Program Management Conference (IPMC) this year was the comprehensive manner of integrating all essential elements of a project, presented by Glen Alleman et al.  In their presentation, Alleman, Coonce, and Price, present a process flow (which, in my estimation, should be mirrored in data and information flow) in which program artifacts were imbued with measures of effectiveness, measures of performance, and measures of progress, to achieve an organic integration of all parts of the project that allow the project team to make a valid assessment of achievement against the plan, informed by risk and opportunity.  (Emphasis my own).  The three-legged stool of cost, schedule, and technical performance are thereby integrated properly at the appropriate level of the project structure, and done in such a way as to overcome the rigidity and fallacy of the single point estimate.

But, as is always the case with elegant models, while they replicate a sufficient portion of reality to allow us to make our assessments using statistical methods, there are other elements that we have purposely left out because our present models do not incorporate them into the normal and normative process.  They are considered situational, and so lie just outside of the process flow, though they insert themselves when necessary–and much more frequently than desired.  I am referring to the availability of money and resources, and the manner in which they affect the project: through work authorizations (WADs) and baseline change requests (BCRs).

I have seen situations where fully 90% of the effort in project management is devoted to manage and adjust the plan based on baseline changes.  This is particularly the case where estimates are poorly developed due to the excuse of uncertainty.  Of course there is uncertainty–that’s the purpose of developing a plan.  The issue isn’t the presence of risk (and opportunity) but that our risks are educated ones, that is, informed by familiarity with similar efforts, engineering assessment, core competency, and other empirical factors.  This is where the most radical elements of the Agile Cult gets it wrong–in focusing on risk and assuming that the only way to realize opportunity is to forgo the empirical process.  This is not only a misreading of risk and opportunity assessment in project management, it is a sort of neo-Luddite position regarding scientific management.

The environment in which a project operates undergoes change.  The framing assumptions of the project determine the expectations of scope, cost, and what defines success.  The concept of framing assumptions was fully developed in a RAND study that I covered in a previous blog post.  Most often, but not always, the change in framing assumptions is reflected in the WAD and BCR process, most often in the latter.  Thus, we have a means of determining and taking account of changes in framing assumptions.  This is in the normal process of project management, as opposed to the more obvious examples of a complete replan or over target baseline (OTB).

So where do we track WADs and BCRs in our processes that will provide us sufficient indicators in our measures of effectiveness, performance, and progress that our resources (both size and type) many not be sufficient or that these changes are sufficient enough that our framing assumptions have changed?  I would argue that the linkage for resources must also be made through the Integrated Master Plan (IMP) and reflect in the IMS, cross-referenced to the PMB.  Technology can provide the remainder of the ability to integrate these elements and provide the process flow necessary to provide early warning.  This integration goes beyond the traditional focus on cost and schedule (and the newly reintroduced emphasis on technical achievement).  It involves integration with resource management systems (personnel, skillset assignments, etc.) as well as financial management systems to determine the availability of money (both its sufficiency and “color”*) being applied to the right place at the right time.

Integrating these elements together then allows for more sophisticated methods of determining project success through the introduction of metrics that provide correlations between the elements.  It also answers, absent politics, the optimum level of both analysis and reporting.

*The “color” of money applies mostly to public investments in which monies appropriated are designed by their purpose:  operations, maintenance, engineering, R&D, etc.

Note: This post was modified to add a point of clarification in applying WADs and BCRs to the PMB.

IMPish Grin — The Connection for Technical Measures (and everything else)

Glen Alleman at his Herding Cats blog has posted his presentation on the manner of integrating technical performance measures in a cohesive and logical manner with project schedule and cost measurement.  Many in the DoD and A&D-focused project community are aware of the work of many of us in this area (my own paper is posted on the College of Performance Management library page here) but the work of Alleman, Coonce, and Price take these concepts a step further.  I wrote an earlier post about the white paper but the presentation demonstrates clearly the flow of logic in constructing not only a model in which technical performance is incorporated into the project plan through measures of effectiveness that are derived from the statement of work, but then makes the connection to measures of progress and measures of performance, clearly outlining the proper integration of the core elements of project planning, execution, and control.

The key artifact that ties the essential elements together: cost, schedule, and technical performance; is the Integrated Master Plan (IMP).  The National Defense Industrial Association Integrated Program Management Division’s (NDIA IPMD) Planning and Scheduling Excellence Guide (PASEG) (link broken) seems to forget this essential step–the artifact that is necessary to allow for the construction of a valid Integrated Master Schedule (IMS).  I think part of the reason for the omission is the mistaken belief that this is an unnecessary artifact–that it is a “nice to have” if the program sponsor remembers to put it in the contact deliverables.  This makes its construction negotiable and vulnerable as a discretionary cost item, which it clearly is not–or, at least, should not be.  Even the Wikipedia entry is confused by the classification of the IMP, characterizing it first as primarily a DoD-specific artifact, a contractual artifact, and–oh, by the way–a necessary step in civic and urban planning (also known as construction project management).  The PASEG does mention summary schedules and perhaps in those rare cases, based on the work being performed and the contract type, some stripped down kind of IMP will do, but regardless of what it is called (and IMP still serves as a good shorthand) then the ability to trace measures of effectiveness to measures of progress is still needed in complex project management.

Thus the IMP is this: it is the fulcrum of integrated project management.  When tying the measures of effectiveness to specific tasks related to the WBS, it is the IMP that provides the roadmap to the working, day-to-day tools that will be used to measure progress–cost, schedule, and technical achievement and assessment against plan, all informed by risk.  For those of us in the technology community, continuing to sell discrete, swim-lane focused apps that do not support this construction are badly out of date.

Ace of Base(line) — A New Paper on Building a Credible PMB

Glen Alleman, a leading consultant in program management (who also has a blog that I follow), Tom Coonce of the Institute for Defense Analyses, and Rick Price of Lockheed Martin, have jointly published a new paper in the College of Performance Management’s Measureable News entitled “Building A Credible Performance Measurement Baseline.”

The elements of their proposal for constructing a credible PMB, from my initial reading, are as follows:

1.  Rather than a statement of requirements, decision-makers should first conduct a capabilities gap analysis to determine the units of effectiveness and performance.  This ensures that program management decision-makers have a good idea of what “done” looks like, and ensures that performance measurements aren’t disconnected from these essential elements of programmatic success.

2.  Following from item 1 above, the technical plan and the programmatic plan should always be in sync.

3.  Earned value management is but one of many methods for assessing programmatic performance in its present state.  At least that is how I interpret what the are saying, because later in their paper they propose a way to ensure that EVM does not stray from the elements that define technical achievement.  But EVM in itself is not the end-all or be-all of performance management–and fails in many ways to anticipate where the technical and programmatic plans diverge.

4.  All work in achieving the elements of effectiveness and performance are first constructed and given structure in the WBS.  Thus, the WBS ties together all elements of the project plan.  In addition, technical and programmatic risk must be assessed at this stage, rather than further down the line after the IMS has been constructed.

5.  The Integrated Master Plan (IMP) is constructed to incorporate the high level work plans that are manifested through major programmatic events and milestones.  It is through the IMP that EVM is then connected to technical performance measures that affect the assessment of work package completion that will be reflected in the detailed Integrated Master Schedule (IMS).  This establishes not only the importance of the IMP in ensuring the linkage of technical and programmatic plans, but also makes the IMP an essential artifact that has all too often be seen as optional, which probably explains why so many project managers are “surprised” when they construct aircraft that can’t land on the deck of a carrier or satellites that can’t communicate in orbit, though they are well within the tolerance bands of cost and schedule variances.

6.  Construct the IMS taking into account the technical, qualitative, and quantitative risks associated with the events and milestones identified in the IMP.  Construct risk mitigation/handling where possible and set aside both cost and schedule margins for irreducible uncertainties, and management reserve (MR) for reducible risks, keeping in mind that margin is within the PMB but MR is above the PMB but within the CBB.  Furthermore, schedule margin should be transitioned from a deterministic one to a probabilistic one–constructing sufficient margin to protect essential activities.  Cost margin in work packages should also be constructed in the same manner-based on probabilistic models that determine the chances of making a risk reducible until reaching the point of irreducibility.  Once again, all of these elements tie back to the WBS.

7.  Cost and schedule margin are not the same as slack or float.  Margin is reserve.  Slack or float is equivalent to overruns and underruns.  The issue here in practice is going to be to get the oversight agencies to leave margin alone.  All too often this is viewed as “free” money to be harvested.

8.  Cost, schedule, and technical performance measurement, tied together at the elemental level of work–informing each other as a cohesive set of indicators that are interrelated–and tied back to the WBS, is the only valid method of ensuring accurate project performance measurement and the basis for programmatic success.

Most interestingly, in conclusion the authors present as a simplified case an historical example how their method proves itself out as both a common sense and completely reasonable approach, by using the Wright brothers’ proof of concept for the U.S. Army in 1908.  The historical documents in that case show that the Army had constructed elements of effectiveness and performance in determining whether they would purchase an airplane from brothers.  All measures of project success and failure would be assessed against those elements–which combined cost, schedule, and technical achievement.  I was particularly intrigued that the issue of weight of the aircraft was part of the assessment–a common point of argument from critics of the use of technical performance–where it is demonstrated in the paper how the Wright brothers actually assessed and mitigated the risk associated with that measure of performance over time.

My initial impression of the paper is that it is a significant step forward in bringing together all of the practical lessons learned from both the successes and failures of project performance.  Their recommendations are a welcome panacea to many of the deficiencies implicit in our project management systems and procedures.

I also believe that as an integral part of the process in construction of the project artifacts, that it is a superior approach than the one that I initially proposed in 1997, which assumed that TPM would always be applied as an additional process that would inform cost and schedule at the end of each assessment period.  I look forward to hearing the presentation at the next Integrated Program Management Conference, at which I will attempt some live blogging.

Frame by Frame: Framing Assumptions and Project Success or Failure

When we wake up in the morning we enter the day with a set of assumptions about ourselves, our environment, and the world around us.  So too when we undertake projects.  I’ve just returned from the latest NDIA IPMD meeting in Washington, D.C. and the most intriguing presentation at the meeting was given by Irv Blickstein regarding a RAND root cause analysis of major program breaches.  In short, a major breach in the cost of a program is defined by the Nunn-McCurdy amendment that was first passed in 1982, in which a major defense program breaches its projected baseline cost by more than 15%.

The issue of what constitutes programmatic success and failure has generated a fair amount of discussion among the readers of this blog.  The report, which is linked above, is full of useful information regarding Major Defense Acquisition Program (also known as MDAP) breaches under Nunn-McCurdy, but for purposes of this post readers should turn to page 83.  In setting up a project (or program), project/program managers must make a set of assumptions regarding the “uncertain elements of program execution” centered around cost, technical performance, and schedule.  These assumptions are what are referred to as “framing assumptions.”

A framing assumption is one in which there are signposts along the way to determine if an assumption regarding the project/program has changed over time.  Thus, according to the authors, the precise definition of a framing assumption is “any explicit or implicit assumption that is central in shaping cost, schedule, or performance expectations.”  An interesting aspect of their perspective and study is that the three-legged stool of program performance relegates risk to serving as a method that informs the three key elements of program execution, not as one of the three elements.  I have engaged in several conversations over the last two weeks regarding this issue.  Oftentimes the question goes: can’t we incorporate technical performance as an element of risk?  Short answer:  No, you can’t (or shouldn’t).  Long answer: risk is a set of methods for overcoming the implicit invalidity of single point estimates found in too many systems being used (like estimates-at-complete, estimates-to-complete, and the various indices found in earned value management, as well as a means of incorporating qualitative environmental factors not otherwise categorizable), not an element essential to defining the end item application being developed and produced.  Looked at another way, if you are writing a performance specification, then performance is a key determinate of program success.

Additional criteria for a framing assumption are also provided in the RAND study.  These are that the assumptions must be determinative, that is, the consequences of the assumption being wrong significantly affects the program in an essential way.  They must also be unmitigable, that is, the consequences of the assumption being wrong are unavoidable.  They must be uncertain, that is, the outcome or certainty of it being right or wrong cannot be determined in advance.  They must be independent and not dependent on another event or series of events.  Finally, they must be distinctive, in setting the program apart from other efforts.

RAND then applied the framing assumption methodology to a number of programs.  The latest NDIA meeting was an opportunity to provide an update of conclusions based on the work first done in 2013.  What the researchers found was that framing assumptions which are kept at a high level, be developed early in a program’s life cycle, and should be reviewed on a regular basis to determine validity.  They also found that a program breached the threshold when a framing assumption became invalid.  Project and program managers, and requirements personnel have at least intuitively known this for quite some time.  Over the years, this is the reason given for requirements changes and contract modifications over the course of development that result in cost, performance, and schedule impacts.

What is different about the RAND study is that they have outlined a practical process for making these determinations early enough for a project/program to be adjusted with changing circumstances.  For example, the numbers of framing assumptions of all MDAPs in the study could be boiled down to four or five, which are easily tested against reality during the milestone and other reviews held over the course of a program.  This is particularly important given the lengthened time-frames of major acquisitions from development to production.

Looking at these results, my own observation is that this is a useful tool for identifying course corrections that are needed before they manifest into cost and schedule impacts, particularly given that leadership at PARCA has been stressing agile acquisition strategies.  The goal here, it seems, is to allow for course corrections before the inertia of the effort leads to failure or–more likely–the development and deployment of an end item that does not entirely meet the needs of the Defense Department.  (That such “disappointments” often far outstrip the capabilities of our adversaries is a topic for a different post).

I think the court is still out on whether course corrections, given the inertia of work and effort already expended at the point that a framing assumption would be tested as invalid, can ever truly be offsetting to the point of avoiding a breach, unless we then rebrand the existing effort as a new program once it has modified its structure to account for new framing assumptions.  Study after study has shown that project performance is pretty well baked in at the 20% mark.  For MDAPs, much of the front-loaded efforts in technology selection and application have been made.  After all, systems require inputs and to change a system requires more inputs, not less, to overcome the inertia of all of the previous effort, not to mention work in progress.   This is basic physics whether we are dealing with physical systems or complex adaptive (economic) systems.

Certainly, more efficient technology that affects the units of measurement within program performance can result in cost savings or avoidance, but that is usually not the case.  There is a bit of magical thinking here: that commercial technologies will provide a breakthrough to allow for such a positive effect.  This is an ideological idea not borne out by reality.  The fact is that most of the significant technological breakthroughs we have seen over the last 70 years–from the microchip to the internet and now to drones–have resulted from public investments, sometimes in public-private ventures, sometimes in seeded technologies that are then released into the public domain.  The purpose of most developmental programs is to invest in R&D to organically develop technologies (utilizing the talents of the quasi-private A&D industry) or provide economic incentives to incorporate technologies that do not currently exist.

Regardless, the RAND study has identified an important concept in determining the root causes of overruns.  It seems to me that a formalized process of identifying framing assumptions should be applied and done at the inception of the program.  The majority of the assessments to test the framing assumptions should then need to be made prior to the 20% mark as measured by program schedule and effort.  It is easier and more realistic to overcome the bow-wave of effort at that point than further down the line.

Note: I have modified the post to clarify my analysis of the “three-legged stool” of program performance in regard to where risk resides.

Let’s Get (Technical) — The Crux of Predictive Measures

For many years since the publication of my various papers on technical performance measurement, I have been asked to update my perspectives.  Over the years I largely declined, mostly this was due to the fact that I had nothing of importance to add to the conversation.  I had staked out what I believed to be a reasonable method of integration between the measurement of technical achievement in human effort and the manner in which the value of that achievement could be documented, along with a reasonable model of technical risk to inform us of our ability to achieve success in the next increment of our technical baseline.  A little background may be helpful.

The development of the model was a collaborative one.  I was the project manager of a team at the Naval Air Systems Command (NAVAIR) that had spent several years attempting to derive “value” from technical achievement, including technical failure.  (More on this later).  The team had gone down many blind allies and had attempted various methods of integrating technical performance measurement into earned value metrics.  Along the way, however, I grew dissatisfied with both the methods and the lack of progress in the project.  Thus, the project team changed course, with the expected turnover of personnel, some of whom did not agree with the change.  As the project manager I directed my team to end the practice of NIH–“Not Invented Here”–and to scour the literature to determine what the efforts already directed at the problem could teach us.  I was also given the resources of a team of mathematicians, physicists, and statisticians familiar with the issues regarding systems engineering who were organized into a reserve unit at NAVAIR, and was assisted by Matt Goldberg of Luri-Goldberg risk fame, who was then working at the Institute for Defense Analysis (IDA).

Among the published literature was a book that had been recommended to me by senior engineering personnel at the (at that time) newly formed Lockheed Martin Corporation.  The book was Technical Risk Management by Jack V. Michaels, who also collaborated on a book called Design to Cost.  Both of these works, but in particular the former one, were influential in my selected approach to technical performance management.  I was also strongly influenced by the work of Daniel Dennett at Tufts in his philosophical and technical observations of strong AI.  It seemed (and seems) to me that the issue of measuring both the intrinsic and practical value of technical achievement requires an understanding of several concepts: of cognition that includes the manner in which we define both progress and learning; the concept of systems, their behavior and the manner in which they respond to stimulus–thus, the evolutionary nature of human systems and the manner that they adapt; the fallacy of reification in measurement and the manner that we overcome it; an understanding of technical risk in the systems engineering process; an understanding of the limitations in measurement of those technical planning systems both in terms of fidelity and horizon; and the way the universe works on the level of the system being developed, given the limitations of the inputs, outputs, and physics involved in its development.

While our understanding needed to be interdisciplinary and comprehensive, conversely, the solution needed to be fairly simple and understandable.  It needed to pass what I called the “So-What? Test.”  That is, would the addition of complexity that improved accuracy pass a test in which there was no good answer to the question “so what?” when looking at the differences in the results.  I have applied this test to software development and other efforts.  For example, is it really worth the added complexity to add that additional button for some marginal advantage if its absence makes no difference in the performance and acceptance of the product?

In the end I selected an approach that I felt was coherent and directed the team to apply the approach to several retrospective analyses, as well as one live demonstration project.  In every case the model demonstrated that it would have been a better predictor than cost and schedule indicators alone in project performance.  In addition, integration of technical achievement which, after all, was one of the cornerstones of the WBS approach in its adoption in the Department of Defense in the early ’90s, improved the early warning capabilities in predicting the manifestation of technical risk, which was then reflected in both cost and schedule performance.

I then published several papers on our findings in collaboration with my team, but then decided to publish my own perspectives separately at the end of my role in the project, which is linked in the first paragraph to this post.  Greatly assisting me with criticism and suggestions during the writing of my paper was Jim Henderson, a colleague whom I respect greatly who was a senior cost analyst at the time at NAVAIR.  He resisted my efforts to credit his assistance at the time for reasons of his own, but I suspect that he would not object now and it is only fitting that I give him the credit that he is due in influencing my own thinking; though I take full responsibility for the opinions and conclusions expressed in it.

Underlying this approach were several, to some new, artifacts deemed essential to the approach.  First among these was the establishment of a technical performance baseline.  The purpose of this baseline is to drive an assessment of current capabilities and then to break down the effort into increments that involve assessments of progress and testing.  These increments should be tied to the WBS and the resources associated with the system being developed.  Second, was the realization that technical achievement was best assessed in increments of short duration through a determination of the risk involved in successfully achieving the next milestone in the technical performance baseline.  This approach was well documented and in use in systems engineering technical risk assessments (TRAs) and, as such, would not cause major changes to a process that was reliable and understood.  Finally, and most (as it turns out) controversially, was the manner in which we “informed” cost performance in deriving value from the TRA.  The last portion of the model was, admittedly, its most contingent portion, though well within the accepted norms of assessment.

It is at the point of applying the value of technical achievement that we still find the most resistance, and so the reason why I have decided to reenter the conversation.  What is the value of failure?  For example, did Space X derive no value from its Falcon 9 rocket exploding?  Here is the video:

 

From the perspective of an outside non-technical observer, the program seems to have suffered a setback, but not so fast.  Space X later released an announcement that indicated that it was reviewing the data from the failed rocket, which had detected an anomaly and self-destructed.  Thus, what we know is that the failed test has caused a two week delay (at least).  Additional resources will need to be expended to study the data from the failure.  The data will be used to review the established failure modes and routines that the engineers developed.  There is both value and risk associated with these activities.  Some of them increase risk in the achievement of the next planned milestone, some handle and mitigate future risks, additional time and resources–perhaps anticipated or perhaps not–will actually be expended as a result of the test.

So how would the model I put forth–we’ll call it the PEO(A) Model for the organization that sponsored it–handle this scenario.  First we would assess the risk associated with achieving the next milestone based on the expert opinion of the engineers for the system.  The model is a simple one–10, 50, 90, 100%.  We trace the WBS elements and schedule activities associated with the system and adjust them appropriately, showing the schedule delay and assigning additional resources.  We inform earned value by the combination of risk and resources.  That is, if our chance of achieving our next milestone is 50%, that is at about the level of chance, then the resources dedicated to the effort may need to be increased an appropriate amount and is reflected in the figure for earned value.

There are two main objections to this approach.

The first objection encounted challenges the engineers’ assessment as being “subjective.”  This objection is invalid on its face.  Expert opinion is both a valid and accepted approach in systems engineering and it seems there is a different bias that underlies the objection.  For example, I have seen this objection raised where the majority of a project is being managed using percent complete as an earned value method, which is probably the most subjective and inaccurate method applied, oftentimes by Control Account Managers (CAMs) that are removed one or two levels (or, at least, degrees of separation) from the actual work.  This goes back to the occasional refrain that I often heard during program reviews by the PM: “How do I make that indicator green?”  Well, there are only two direct ways with variations: game the system, or actually perform the work to the level of planned achievement and then accurately measure progress.  I suspect the fear here is that the engineers will be too accurate in their assessments, and so the fine game of controlling the indicators to avoid being micromanaged and making enough progress to support the indicators is undermined.  Changing this occasionally encountered mindset is a discussion for a different blog post.

There is certainly room for improvement in using this method.  Since we are dealing with a time-phased technical performance plan with short duration milestones, providing results based on an assessment of achieving the next increment, we can always apply simulated Monte Carlo at our present position to get the most likely range of probable outcomes.  Handicapping the handicapper is an acceptable way of assessing future performance with a great deal of fidelity.  Thus, the model proposed could be adjusted to incorporate the refinement.  I am not certain, however, if it would pass the “so-what?” test.  It would be interesting to find out.

The second objection encountered challenges the maximum achievement at 100%.  I find this objection both amusing at one level and understandable given the perspective of the individuals raising the objection.  First the amusing (I hope) response: in this universe the best you can ever do is 100%.  You can’t “give 110%,” you can’t “achieve 150%,” etc.  This sounds catchy in trendy books on management, in get rich quick schemes, and is heard too often by narcissists in business.  But in reality the best you can do is 100%–and if you’ve lived long enough with your feet on the ground you know that 100% is a rare level of achievement that exists in limited, well-defined settings.  In risk, 100% probability is even rarer, and so when we say that our chance of achieving the next level in a technical performance plan is 100%, we are saying that all risks have been eliminated and/or mitigated to zero.

The serious part of this objection, though, surrounds the financial measurement of technical performance, to wit: what if we are ahead of plan?  In that case, the argument goes, we should be able to show more than 100% of achievement.  While understandable, this objection demonstrates a misunderstanding of the model.  The assessment from one milestone to the next is one based on risk.  The 100% chance of achievement allows the project to claim full credit for all effort up to that point.  The actual expenditures and the resulting savings in resources and schedule, documented through normal cost and schedule performance measurement, will reflect the full effect of being on plan or ahead of plan.  So while it is true that technical performance measurement can only, in the words of one critic, “take away performance,” this does not pass the “so-what?” test–at least not in its practical application.

What the objection does do is expose the underlying weakness and uncertainty held with any model that concerns itself with deriving “value” from success and failure in developmental efforts.  It is a debate that has raged in systems engineering circles for quite some time, not to mention the expansive definition of value during discussions of investment and its return.  But that is not what the PEO(A) model of technical performance measurement (TPM) was about.  The intent of proposing a working model of technical achievement was geared specifically to defining our terms and aims with precision, and then to come up with a model to meet those aims.  It measures “value” only insomuch that the value is determined by the resources committed to achieving the increment in time.  The base “value” of a test failure, as in the Space X example, is derived from the assessment of technical risk in the wake of the postmortems.

It should not be controversial to advocate for finding a way of integrating technical performance into project management measures, particularly in their role in determining predictive outcomes.  We can spend a great deal of time and money building a jet that, in the end, cannot land on an aircraft carrier, a satellite that cannot communicate after it achieves orbit, a power plant that cannot achieve full capacity, or a ship that cannot operate in the conditions intended, if we do not track the technical specifications that define the purpose of the system being developed and deployed.  I would argue that technical achievement is the strategic center of measurements upon which all others are constructed.  The PEO(A) model is one model that has been proposed and has been in use since its introduction in 1997.  I would not argue that it cannot be improved or that alternatives may achieve an equal or better result.

In the days prior to satellite and GPS navigation was the norm, when leaving and entering port–to ensure that we avoided shoal water and stayed within the channel–ships would take a fix on a set of geographical points known as bearing measurements.  Lines were drawn from the points of measurement and where they intersected was the position of the ship.  This could be achieved by sight, radar, or radio.  What the navigation team was doing was determining the position of the ship in a three dimensional world and representing it on a two dimensional chart.  The lines formed in the sighting are called lines of position (LOP).  Two lines of position are sufficient to establish a “good” fix, but we know that errors are implicit due to variations in observation, inconvenient angles, the drift of the ship, and the ellipse error in all charts and maps (the earth not being flat).  Thus, at least three points of reference are usually the minimum acceptable from a practical perspective in establishing an “excellent” fix.  The more the merrier.  As an aside, ships and boats still use this method to verify their positions–even redundantly–since depending too heavily on technology that could fail could be the difference between a routine day and tragedy.

This same principle applies to any measurement of progress in human endeavor.  In project management we measure many things.  Oftentimes they are measured independently.  But if they do nothing to note the position of the project in space and time–to allow it to establish a “fix” on where it is–then they are of little use.  The points must converge in order to achieve value in the measurement.  Cost, schedule, qualitative risk, technical performance, and financial execution are our bearing measurements.  The representation of these measurements on a common chart–the points of integration–will determine our position.  Without completing this last essential task is like being adrift in a channel.