For What It’s Worth — More on the Materiality and Prescriptiveness Debate and How it Affects Technological Solutions

The underlying basis on the materiality vs. prescriptiveness debate that I previously wrote about lies in two areas:  contractual compliance, especially in the enforcement of public contracts, and the desired outcomes under the establishment of a regulatory regime within an industry.  Sometimes these purposes are in agreement and sometimes they are in conflict and work at cross-purposes to one another.

Within a simple commercial contractual relationship, there are terms and conditions established that are based on the expectation of the delivery of supplies and services.  In the natural course of business these transactions are usually cut-and-dried: there is a promise for a promise, a meeting of the minds, consideration, and performance.  Even in cases that are heavily reliant on services, where the terms are bit more “fuzzy,” the standard is that the work being performed be done in a “workmanlike” or “professional” manner, usually defined by the norms of the trade or profession involved.  There is some judgment here depending on the circumstances, and so disputes tend to be both contentious and justice oftentimes elusive where ambiguity reigns.

In research and development contracts the ambiguities and contractual risks are legion.  Thus, the type of work and the ability to definitize that work will, to the diligent contract negotiator, determine the contract type that is selected.  In most cases in the R&D world, especially in government, contract types reflect a sharing and handling of risk that is reflected in the use of cost-plus type contracts.

Under this contract type, the effort is reimbursed to the contractor, who must provide documentation on expenses, labor hours, and work accomplished.  Overhead, G&A, and profit is negotiated based on a determination of what is fair and reasonable against benchmarks in the industry, which will be ultimately determined through negotiation of the parties.  A series of phases and milestones are established to mark the type of work that is expected to be accomplished over time.  The ultimate goal is the produce a prototype end item application that meets the needs of the agency, whether that agency is the Department of Defense or some other civilian agency in the government.

The period of performance of the contracts in these cases, depending on the amount of risk in research and development in pushing the requisite technology, usually involving several years.  Thus, the areas of concern given the usually high dollar value, inherent risk, and extended periods, involve:

  1. The reliability, accuracy, quality, consistency, and traceability of the underlying systems that report expenditures, effort, and progress;
  2. Measures that are indicative of whether all of the aspects of the eventual end item will meet elements that constitute expectations and standards of effectiveness, performance, and technical achievement.  These measures are conducted within the overall cost and schedule constraints of the contracted effort;
  3. Assessment over the lifecycle of the contract regarding external, as well as internal technical, qualitative, and quantitative risks of the effort;
  4. The ability of items 1 through 3 above to provide an effective indication or early warning that the contractual vehicle will significantly vary from either the contractual obligations or the established elements outlining the physical, performance, and technical characteristics of the end item.
  5. The more mundane, but no less important, verification of contractual performance against the terms and conditions to avoid a condition of breach.

Were these the only considerations in public contracting related to project management our work in evaluating these relationships, while challenging, would be fairly cut-and-dried given that they would be looked at from a contracting perspective.  But there is also a systemic purpose for a regulatory regime.  These are often in conflict with one another.  Such requirements as compliance, surveillance, process improvement, and risk mitigation are looking at the same systems, but from different perspectives with, ultimately, differing reactions, levels of effectiveness, and results.  What none of these purposes includes is a punitive purpose or result–a line oftentimes overstepped, in particular, by private parties.  This does not mean that some regulations that require compliance with a law do not come with civil penalties, but we’ll get to that in a moment.

The underlying basis of any regulatory regime is established in law.  The sovereign–in our case the People of the United States through the antecedent documents of governance, including the U.S. Constitution and Constitutions of the various states, as well as common law–possesses an inherent right to regulate the health, safety, and welfare of the people.  The Preamble of the U.S. Constitution actually specifies this purpose in writing, but in broader terms.  Thus, the purposes of a regulatory regime when it comes to this specific issue are what are at issue.

The various reasons usually are as follows:

  1. To prevent an irreversible harm from occurring.
  2. To enforce a particular level of professional conduct.
  3. To ensure compliance with a set of regulations or laws, especially where ambiguities in civil and common law have yielded judicial uncertainty.
  4. To determine the level of surveillance of a regulated system that is needed based on a set of criteria.
  5. To encourage particular behaviors.
  6. To provide the basis for system process improvement.

Thus, in applying a regulation there are elements that go beyond the overarching prescriptiveness vs. materiality debate.  Materiality only speaks to relevance or significance, while prescriptiveness relates to “block checking”–the mindless work of the robotic auditor.

For example, let’s take the example of two high profile examples of regulation in the news today.

The first concerns the case of where Volkswagen falsified its emissions test results for a good many of its vehicles.  The role of the regulator in this case was to achieve a desired social end where the state has a compelling interest–the reduction of air pollution from automobiles.  The regulator–the Environmental Protection Agency (EPA)–found the discrepancy and issued a notice of violation of the Clean Air Act.  The EPA, however, did not come across this information on its own.  Since we are dealing with multinational products, the initial investigation occurred in Europe under a regulator there and the results passed to the EPA.  The corrective action is to recall the vehicles and “make the parties whole.”  But in this case the regulator’s remedy may only be the first line of product liability.  It will be hard to recall the pollutants released into the atmosphere and breach of implicit contract with the buyers of the automobiles.  Whether a direct harm can be proven is now up to the courts, but given that executives in an internal review (article already cited) admitted that executives knew about deception, the remedies now extend to fraud.  Regardless of the other legal issues,

The other high profile example is the highly toxic levels of lead in the drinking water of Flint, Michigan.  In this case the same regulator, the EPA, has issued a violation of federal law in relation to safe drinking water.  But as with the European case, the high levels of lead were first discovered by local medical personnel and citizens.  Once the discrepancy was found a number of actions were required to be taken to secure proper drinking water.  But the damage has been done.  Children in particular tend to absorb lead in their neurological systems with long term adverse results.  It is hard to see how the real damage that has been inflicted will make the damaged parties whole.

Thus, we can see two things.  First, the regulator is firmly within the tradition of regulating the health, safety, and welfare, particularly the first category and second categories.  Secondly, the regulatory regime is reactive.

While obviously the specific illnesses caused by the additional pollution form Volkswagen vehicles is probably not directly traceable to harm, the harm in the case of elevated lead levels in Flint’s water supply is both traceable and largely irreversible.

Thus, in comparing these two examples, we can see that there are other considerations than the black and white construct of materiality and prescriptiveness.  For example, there are considerations of irreversible harm, prevention, proportionality, judgment, and intentional results.

The first reason for regulation listed above speaks to irreversible harm.  In these cases proportionality and prevention are the main concerns.  Ensuring that those elements are in place that will prevent some catastrophic or irreversible harm through some event or series of events is the primary goal in these cases.  When I say harm I do not mean run of the mill, litigious, constructive “harm” in the legal or contractual sense, but great harm–life and death, resulting disability, loss of livelihood, catastrophic market failure, denial of civil rights, and property destruction kind of harm.  In enforcing such goals, these fall most in line with prescriptiveness–the establishment of particular controls which, if breached, would make it almost impossible to fully recover without a great deal of additional cost or effort.  Furthermore, when these failures occur a determination of culpability or non-culpability is determined.  The civil penalties in these cases, where not specified by statute, are ideally determined by proportionality of the damage.  Oftentimes civil remedies are not appropriate since these often involve violations of law.  This occurs, in real life, from the two main traditional approaches to audit and regulation being rooted in prescriptive and judgmental approaches.

The remainder of the reasons for regulation provide degrees of oversight and remedy that are not only proportional to the resulting findings and effects, but also to the goal of the regulation and its intended behavioral response.  Once again, apart from the rare and restricted violations given in the first category above, these regulations are not intended to be enforced in a punitive manner, though there can be penalties for non-compliance.  Thus, proportionality, purpose, and reasonableness are additional considerations to take into account.  These oftentimes fall within the general category of materiality.

Furthermore, going beyond prescriptiveness and materiality, a paper entitled Applying Systems-Thinking to Reduce Check-the-Box Decisions in the Audit of Complex Estimates, by Anthony Bucaro at the University of Illinois at Urbana-Champaign, proposes an alternative auditing approach that also is applicable to other types of regulation, including contract management.  The issue that he is addressing is the fact that today, in using data, a new approach is needed to shift the emphasis to judgment and other considerations in whether a discrepancy warrants a finding of some sort.

This leads us, then, to the reason why I went down this line of inquiry.  Within project management, either a contractual or management prerogative already exists to apply a set of audits and procedures to ensure compliance with established business processes.  Particular markets are also governed by statutes regulating private conduct of a public nature.  In the government sphere, there is an added layer of statutes that prescribe a set of legal and administrative guidance.  The purposes of these various rules varies.  Obviously breaking a statute will garner the most severe and consequential penalties.  But the set of regulatory and administrative standards often act at cross purposes, and in their effect, do not rise to the level of breaking a law, unless they are necessary elements in complying with that law.

Thus, a whole host of financial and performance data assessing what, at the core, is a very difficult “thing” to execute (R&D leading to a new end item), offers some additional hazards under these rules.  The underlying question, outside of statute, concerns what the primary purpose should be in ensuring their compliance.  Does it pass the so-what? test if a particular administrative procedure is not followed to the letter?

Taking a broader approach, including a data-driven and analytical one, removes much of the arbitrariness when judgment and not box-checking is the appropriate approach.  Absent a consistent and wide pattern that demonstrates a lack of fidelity and traceability of data within the systems that have been established, auditors and public policymakers must look at the way that behavior is affected.  Are there incentives to hide or avoid issues, and are there sufficient incentives to find and correct deficiencies?  Are the costs associated with dishonest conclusions adequately addressed, and are there ways of instituting a regime that encourages honesty?

At the core is technology–both for the regulated and the regulator.  If the data that provides the indicators of compliance come, unhindered, from systems of record, then dysfunctional behaviors are minimized.  If that data is used in the proper manner by the regulator in driving a greater understanding of the systemic conditions underlying the project, as well as minimizing subjectivity, then the basis for trust is established in determining the most appropriate means of correcting a deficiency.  The devil is in the details, of course.  If the applied technology simply reproduces the check-block mentality, then nothing has been accomplished.  Business intelligence and systems intelligence must be applied in order to achieve the purposes that I outlined earlier.


I Still Can’t Find What I’m Looking For — The Materiality vs. Prescriptiveness Controversy in Project Management Oversight

I’ve had a number of conversations over the last four months on this issue, mostly from the technical solution side rather than the issue side.  But intent does eventually translate into action and so it is an important issue to tackle.  Usually this issue comes up within the context of the purpose of audit.  But then, we have to understand the purpose of audit itself within the context of project management.

For example, it is assumed that in project management that we are dealing with an oversight and regulatory regime.  There are dichotomies to this.

The hardcore auditor states that their job is to ensure that the project organization is compliant to a specific set of criteria.  These criteria will have attribute associated with them that clarify what an auditor should find as proof of compliance.  Sounds cut and dried enough–and from the standpoint of the auditor it is.  But what of the rationale behind the criteria and attributes?

No doubt, especially in highly technical fields, these have been published built on the long experience of practitioners of the discipline.  Where there are competing interests–which pretty much defines all economic relationships–the criteria and attributes will be informed by a type of in-group politics.  The end result is to prescribe (hence prescriptiveness) how something is to be done–approaching standardization–in order to establish some common basis of what constitutes minimally acceptable business practices.  Some of the guidance associated with these disciplines can vary. For example, we’re all, at least vaguely, familiar with GAAP in accounting (though here there are many types of GAAP depending on the type and size of business), there are food sanitation standards, and in project management, particularly in government contracting, there are standards for business systems, including criteria for the establishment of an earned value management system.

There are some areas that require more prescriptiveness than others.  For example, in visiting ones doctor a look around the examination room gives us some clue of this effect.  Examination rooms must have a minimum number of items in them, and among these items are the ubiquitously red hazardous waste bins.  These bins must be a certain color (red), a certain size, contain particular markings identifying their purpose and what they may contain, the hazardous nature of their possible contents, and must have a special bag within them so that the hazardous fluids and waste do not permanently contaminate the bin making its reuse prohibited.  Each of these attributes that I’ve mentioned must be met.  It is assumed that these attributes have been established for good reason associated with the health and safety of the immediate personnel engaged in the medical profession, but also serve the health and safety of the community in general.

Thus, areas that are deemed hazardous, potentially hazardous, or containing much risk that can do harm far beyond the entity immediately involved tend to be the most prescriptive.  Naval aviation is another good example, because it is so closely tied to research and development within the aviation project management community.  Project managers in Navy aviation, for example, are considered to be the life-cycle managers of their systems.  Thus, though a project may be established for a specific purpose, the long term view is not automatically eschewed in favor of immediate concerns.  It promotes a more balanced view.

The guidance Navy aviation follows is in the Naval Air Training and Operating Procedures Standardization (NATOPS) manual.  NATOPS consists of very specific guidance and checklists on the do’s and don’ts of flying, personnel readiness, and maintaining an aircraft based on the experience of previous operations.  It has been said and repeated often that NATOPS was written with the blood of every aviator that came before.  Because operating a high performance aircraft in military operations can be so hazardous, the guidance in this case is very specific.  There is very little, if any, flexibility in the application of the guidance–and with good reason.

But not every action in life has the same effect as every other.  An administrative error, for example, on a form may or may not have an effect far beyond requiring a simple correction.  An error in decision making, especially under stress or duress, requires an understanding of mitigating factors.  Human action, by definition, is filled with error.  This is a given and one who does not accept this very basic fact is neither mature enough to effectively engage with the world, nor qualified to assume any kind of leadership position or role.  Some lessons can be simply “learned” given that action to acknowledge and correct the error is immediately taken.  Others that have greater impact may require more calibrated levels of response.  This is where materiality comes into play.

Materiality is the application not only of experience, but of judgment reasonably applied, and a number of elements go into that judgment.  Once again, the military services, probably more than any civilian organization, undergoes regular and intensive rounds of testing, audit, and oversight on all of its financial, operational, and administrative systems.  Given that public monies and the public interest is involved, sometimes the results of these audits find themselves into the news.  When such events occur, oftentimes a great deal of time, money, and resources are diverted from other activities–including those dedicated to correcting the identified error–to addressing inquiries.  The news media, of course, are simply serving the purpose of the press within the idealized framework of a democratic society.  (Journalistic standards are a different topic).  It would be nice, from the military’s perspective, that these errors (and in some cases misbehavior by individuals) didn’t happen, but they are a part of life in an extremely large and complex organization that also deals with a large number of resources.

Thus, the definition of materiality in these cases also involve considerations of perception, as well as minimizing both the frequency and impact of simple or compound error, incompetence, or misbehavior.

More discussion to follow.