KPIs

I’ve Got Your Number — Types of Project Measurement and Services Contracts

Glen Alleman reminds us at his blog that we measure things for a reason and that they include three general types: measures of effectiveness, measures of performance, and key performance parameters.

Understanding the difference between these types of measurement is key, I think, to defining what we mean by such terms as integrated project management and in understanding the significance of differing project and contract management approaches based on industry and contract type.

For example, project management focused on commodities, with their price volatility, emphasizes schedule and resource management. Cost performance (earned value) where it exists, is measured by time in lieu of volume- or value-based performance. I have often been engaged in testy conversations where those involved in commodity-based PM insist that they have been using Earned Value Management (EVM) for as long as the U.S.-based aerospace and defense industry (though the methodology was borne in the latter). But when one scratches the surface the approaches in the details on how value and performance is determined is markedly different–and so it should be given the different business environments in which enterprises in each of these industries operate.

So what is the difference in these measures? In borrowing from Glen’s categories, I would like to posit a simple definitional model as follows:

Measures of Excellence – are qualitative measures of achievement against the goals in the project;

Measures of Performance – are quantitative measures against a plan or baseline in execution of the project plan.

Key Performance Parameters – are the minimally acceptable thresholds of achievement in the project or effort.

As you may guess there is sometimes overlap and confusion regarding which category a particular measurement falls. This confusion has been exacerbated by efforts to define key performance indicators (KPIs) based on industry, giving the impression that measures are exclusive to a particular activity. While this is sometimes the case it is not always the case.

So when we talk of integrated project management we are not accepting that any particular method of measurement has primacy over the others, nor subsumes them. Earned Value Management (EVM) and schedule performance are clearly performance measures. Qualitative measures oftentimes measure achievement of technical aspects of the end item application being produced. This is not the same as technical performance measurement (TPM), which measures technical achievement against a plan–a performance measure. Technical achievement may inform our performance measurement systems–and it is best if it does. It may also inform our Key Performance Parameters since exceeding a minimally acceptable threshold obviously helps us to determine success or failure in the end. The difference is the method of measurement. In a truly integrated system the measurement of one element informs the others. For the moment these systems presently tend to be stove-piped.

It becomes clear, then, that the variation in approaches differs by industry, as in the example on EVM above, and–in an example that I have seen most recently–by contract type. This insight is particularly important because all too often EVM is viewed as being synonymous with performance measurement, which it is not. Services contracts require structure in measurement as much as R&D-focused production contracts, particularly because they increasingly take up a large part of an enterprise’s resources. But EVM may not be appropriate.

So for our notional example, let us say that we are responsible for managing an entity’s IT support organization. There are types of equipment (PCs, tablet computers, smartphones, etc.) that must be kept operational based on the importance of the end user. These items of hardware use firmware and software that must be updated and managed. Our contract establishes minimal operational parameters that allow us to determine if we are at least meeting the basic requirements and will not be terminated for cause. The contract also provides incentives to encourage us to exceed the minimums.

The sites we support are geographically dispersed. We have to maintain a help desk but also must have people who can come onsite and provide direct labor to setup new systems or fix existing ones–and that the sites and personnel must be supported within a particular time-frame: one hour, two hours, and within twenty-four hours, etc.

In setting up our measurement systems the standard practice is to start with the key performance parameters. Typically we will also measure response times by site and personnel level, record our help desk calls, and track qualitative aspects of the work: How helpful is the help desk? Do calls get answered at the first contract? Are our personnel friendly and courteous? What kinds of hardware and software problems do we encounter? We collect our data from a variety of one-off and specialized sources and then we generate reports from these systems. Many times we will focus on those that will allow us to determine if the incentive will be paid.

Among all of this data we may be able to discern certain things: if the contract is costing more or less than we anticipated, if we are fulfilling our contractual obligations, if our personnel pools are growing or shrinking, if we are good at what we do on a day-to-day basis, and if it looks as if our margin will be met. But what these systems do not do is allow us to operate the organization as a project, nor do they allow us to make adjustments in a timely manner.

Only through integration and aggregation can we know, for example, how the demand for certain services is affecting our resource demands by geographical location and level of service, on a real=time basis where we need to make adjustments in personnel and training, whether we are losing or achieving our margin by location, labor type, equipment type, hardware vs. software; our balance sheets (by location, by equipment type, by software type, etc.), if there is a learning curve, and whether we can make intermediate adjustments to achieve the incentive thresholds before the result is written in stone. Having this information also allows us to manage expectations, factually inform perceptions, and improve customer relations.

What is clear by this example is that “not doing EVM” does not make measurement easy, nor does it imply simplification, nor the absence of measurement. Instead, understanding the nature of the work allows us to identify those measures within their proper category that need to be applied by contract type and/or industry. So while EVM may not apply to services contracts, we know that certain new aggregations do apply.

For many years we have intuitively known that construction and maintenance efforts are more schedule-focused, that oil and gas exploration more resource- and risk-focused, and that aircraft, satellites, and ships more performance-focused. I would posit that now is the time for us to quantify and formalize the commonalities and differences. This also makes an integrated approach not simply a “nice to have” capability, but an essential capability in managing our enterprises and the projects within them.

Note: This post was updated to correct grammatical errors.

I Can See Clearly Now (The Risk Is Gone) — Managing and Denying Risk in PM

Just returned from attending the National Defense Industrial Association’s Integrated Program Management Division (NDIA IPMD) quarterly meeting.  This is an opportunity for both industry and government to share common concerns and issues regarding program management, as well as share expertise and lessons learned.

This is one among a number of such forums that distinguishes the culture in aerospace and defense in comparison to other industry verticals.  For example, in the oil and gas industry the rule of thumb is to not share such expertise across the industry, except in very general terms through venues such as the Project Management Institute, since the information is considered proprietary and competition sensitive.  I think, as a result, that the PM discipline suffers for this lack of cross pollination of ideas, resulting in an environment where, in IT infrastructure, the approach is toward customization and stovepipes, resulting in a situation where solutions tend to be expensive, marked by technological dead ends and single point failures, a high rate of IT project failure, and increased expense.

Among a very distinguished venue of project management specialists, one of the presentations that really impressed me by its refreshingly candid approach was given by Dave Burgess of the U. S. Navy Naval Air Systems Command (NAVAIR) entitled “Integrated Project Management: ‘A View from the Front Line’.”  The charts from his presentation will be posted on the site (link in the text on the first line).  Among the main points that I took from his presentation are:

a.  The time from development to production of an aircraft has increased significantly from the 1990s.  The reason for this condition is implicit in the way that PM is executed.  More on this below in items d and e.

b.  FY 2015 promises an extremely tight budget outlook for DoD.  From my view given his chart it is almost as if 2015 is the budgetary year that Congress forgot.  Supplemental budgets somewhat make up for the shortfalls prior to and after FY 2015, but the next FY is the year that the austerity deficit-hawk pigeons come home to roost.  From a PM perspective this represents a challenge to program continuity and sustainability.  It forces choices within program that may leave the program manager with a choice of the lesser of two evils.

c.  Beyond the standard metrics provided by earned value management that it is necessary for program and project managers to identify risks, which requires leading indicators to inform future progress.

This is especially important given the external factors of items a and b above.  Among his specific examples, Mr. Burgess demonstrated the need for integration of schedule and cost in the development of leading indicators.  Note that I put schedule ahead of cost in interpreting his data, and in looking at his specific examples there was an undeniable emphasis on the way in which schedule drives performance, given that it is a measure of the work that needs to be accomplished with (hopefully) an assessment of the resources necessary to accomplish the tasks in that work.  For example, Mr. Burgess demonstrated the use of bow waves to illustrate that the cumulative scope of the effort as the program ramps over time up will overcome the plan if execution is poor.  This is a much a law of physics as any mathematical proof.  No sky-hooks exist in real life.

From my perspective in PM, cost is a function of schedule.  All too often I have seen cases where the project management baseline (PMB) is developed apart from and poorly informed by the integrated master schedule (IMS).  This is not only foolhardy it is wrong.  The illogic of doing so should be self-evident but the practice persists.  It mostly exists because of the technological constraints imposed by the PM IT systems being stovepiped, which then drive both practice and the perception of what industry views is possible.

Thus, this is not an argument in favor of the status quo, it is, instead, an argument to dump the IT tool vendor refusing to update their products whose only interest is to protect market share and keep their proprietary solution sticky.  The concepts of sunk costs vs. prospective costs are useful in this discussion.  Given the reality of the tight fiscal environment in place and greater constraints to come, the program and project manager is facing the choice of paying recurring expenses for outdated technologies to support their management systems, or selecting and deploying new ones that will reduce their overheads and provide better and quicker information.  This allows them to keep people who, despite the economic legend that robots are taking our jobs, still need to make decisions in effectively managing the program and project.  It takes a long time and a lot of money to develop an individual with the skills necessary to manage a complex project of the size discussed by Mr. Burgess, while software technology generations average two years.  I’d go with keeping the people and seeking new, innovative technologies on a regular basis, since the former will always be hard and expensive (if done right) and the latter for the foreseeable future will continue on a downward cost slope.  I’ll expand on this in a later post.

d.  There is a self-reinforcing dysfunctional systemic problem that contributes to the condition described in item “a,” which is the disconnect between most likely estimates of the cost of a system and the penchant for the acquisition system to award based on a technically acceptable approach that is the lowest bid.  This encourages unrealistic expectations in forming the plan once the contract is awarded, which eventually is modified, through various change rationales, that tend to bring the total scope back to the original internal most-likely estimate.  Thus, Procuring Contracting Officers (PCOs) are allowing contractors to buy-in, a condition contrary to contracting guidance, and it is adversely affecting both budget and program planning.

e.  That all too often program managers spend time denying risk in lieu of managing risk.  By denying risk, program and project managers focus on a few elements of performance that they believe give them an indication of how their efforts are performing.  This perception is reinforced by the limited scope of the information looked at by senior personnel in the organization in their reports.  It is then no surprise that there are “surprises” when reality catches up with the manager.

It is useful to note the difference between program and project management in the context of the A&D vertical.  Quite simply, in this context, a program manager is responsible for all of the elements of the system being deployed.  For the U.S. Navy this includes the entire life-cycle of the system, including the logistics and sustainment after deployment.  Project management in this case includes one element of the system; for example, development and production of a radar, though there are other elements of the program in addition to the radar.  My earlier posts on the ACA program–as opposed to the healthcare.gov site–is another apt example of these concepts in practice.

Thus, program managers, in particular, need information on all of the risks before them.  This would include not only cost and schedule risk, which I would view as project management level indicators, but also financial and technical risk at the program level.  Given the discussions this past week, it is apparent that our more familiar indicators, while useful, require a more holistic set of views that both expand and extend our horizon, while keeping that information “actionable.”  This means that our IT systems used to manage our business systems require more flexibility and interoperability in supporting the needs of the community.

 

 

What’s Your Number (More on Metrics)

Comments from my last post mainly centered around: but are you saying that we shouldn’t do assessments or analysis?  No.  But it is important to define our terms a bit better and realize that what we monitor is not equal and not measuring the same kind of thing.

As I have written here, our metrics fall into categories but each has a different role or nature and are generally rooted in two concepts.  These concepts are quality assurance (QA) and quality control (QC)–and they are not one and the same.  As our specialties have fallen away over time, the distinction between QA and QC has been lost.  The evidence of this confusion can be found not only in Wikipedia here and here, but also in project management discussion groups such as here and here.

QA measures the quality of the processes involved in the development and production of an end item.  It tends to be a proactive process and, therefore, looks for early warning indicators.

QC measures the quality in the products.  It is a reactive process and is focused on defect correction.

A large part of the confusion as it relates to project management is that QA and QC has its roots in iterative, production-focused activities.  So knowing which subsystems within the overall project management system we are measuring is important in understanding whether it serves a QA or QC purpose, that is, that is has a QA or QC effect.

Generally, in management, we categorize our metrics into groupings based on their purpose.  There are Key Performance Indicators (KPIs), which are categorized as diagnostic indicators, lagging indicators, and leading indicators.  There are Key Risk Indicators (KRIs), which measure future adverse impacts.  KRIs are qualitative and quantitative measures that must be handled or mitigated in our plans.

KPIs and KRIs can serve QA and QC purposes and it is important to know the difference so that we can understand what the metric is telling us.  The dichotomy between these effects is not closed.  QC is meant to drive improvements in our processes so that we can shift (ideally) to QA measures in ensuring that our processes will produce a high quality product.

When it comes to the measurement of project management artifacts, our metrics regarding artifact quality, such those applied to the Integrated Master Schedule (IMS) is actually a measure rooted in QC.  The defect has occurred in a product (the IMS) and now we must go back and fix it.  It is not that QC is not an essential function.

It just seems to me that we are sophisticated enough now to establish systems in the construction of the IMS and other artifacts, that is, to be proactive (avoiding errors), in lieu of being reactive (fixing errors).  And–yes–at the next meetings and conferences I will present some ideas on how to do that.

 

Close to the (Leading) Edge

Much has been made in project management circles that most of the indicators that we have to work with are lagging indicators. These are indicators that record what happened but don’t inform the future.  But is that true in most cases?  If not, which indicators are leading and what new leading indicators do we need to be looking at in constructing a project management toolkit–those at the leading edge?

In order to determine this we also need to weed out those measures that are neither lagging nor leading.  These are diagnostic measures that simply indicate the health of a system.  For example, inventory accuracy in the latest warehouse samples, the number of orders successfully fulfilled on the initial contact, etc.  These oftentimes are most effectively used in tracking iterative, industrial-like activities.

Even leading indicators differ.  A leading indicator for a project is not like a leading indicator for larger systems.  Even complex projects possess limited data points for assessing performance.  For some of these projects, where the span of control does not exceed the capabilities of a leader this usually does not matter.  In those cases, very simple indicators are pretty good at determining trends as long as the appropriate measures are used.  It is also at this level that large changes can occur very rapidly since minor adjustments to the project, underlying technology, or personnel dynamics will determine the difference between success and failure.  Thus, our models of social systems at this level often assume rational human behavior or, to use the colloquialism, “all things being equal” in order to predict trends.  This is also the level at which our projects are valid for shorter spans, since overwhelming external or internal events or systems can present risks that the simple system will not be able to fully overcome, when “all things are not equal.”

More complex systems, particularly in project management, are able to employ risk handling methodologies, either overtly or unconsciously.  Needless to say the former usually garners more reliable and positive results.  In this environment the complaint for EVM is that it is “merely looking in the rear view mirror” with the implication that it is not of great value to project or program managers.  But is this entirely true?  I suspect that some of this is excuse peddling because of what I refer to as the Cult of Optimism.

There is no doubt that there is a great deal of emphasis on what has occurred in the past.  Calculations of work performed, resources expended, and time passed; all of these record what has happened.  Many of these are lagging indicators, but they also contain essential information in informing the future.  The key in understanding what is a leading indicator is understanding the dynamics of causation.  Some lagging indicators have overlap.  Some are better at measuring a particular dimension of the project than others.  It is important to know the difference so that the basis for leading indicators can be selected.

For example, regression is a popular method in determining EAC/ETC predictions.  The problem is that these results are often based on very weak correlations and causality.  It is a linear method that is used to measure outcomes in a non-linear system.  This then causes us to introduce other methods to strengthen the credibility of the estimate through engineering and expert opinion.  When possible, parametrics are introduced.  But does this get us there?  I would posit that it doesn’t because it is simply a way of throwing everything at hand at the issue in hopes that something hits the target.  Instead, our methods must be more precise and our selection methods proven. I would posit that starting with something like the Granger causality test in comparing two different time series and determining if one is predictive of another would be useful in weeding out the wheat from the chaff.

For example, years ago I headed a project to identify and integrate technical performance into project management performance measures.  Part of the study we conducted included a retrospective analysis to determine correlations and causality between key technical performance measures over the life of a project and its performance over time.  What we found is that, carefully chosen, TPMs are a strong indicator of future performance when informed by probabilities over the time series.

So is it true that we are “merely” looking in the rear view mirror?  No.  We only have the past to inform the future, thus the comment is irrelevant.  Besides “the past is never dead.  It’s not even past…”  The aggregation of our efforts to achieve the goals of the project will determine the likely outcomes in the future.  So the challenge isn’t looking in the rear view mirror–after all in real life we do this all the time to see what’s gaining on us–but in picking those elements from past performance that, given correlation and causality, will inform the probabilities of future outcomes.

Note:  Grammatical changes made to the post.