Standards & Metrics

« Go back

To measure performance it is necessary to have standards and metrics. Processes typically support a standard defined by the business or a regulatory agency. In case of regulatory requirements the metric and means of data collection may be prescribed.

Metrics represent data collected on some output. Outputs are created by a process. Process maturity can vary widely from ad hoc (where the process is effectively no process), to highly defined. The degree of rigor is defined by business needs.

In keeping with the Hisenberg uncertainty principle, the act of measurement affect other aspects of the system. People optimize what is measured so it is critically important that the overall system of measurements is designed to prevent gaming the metric regime to the detriment of other important process or aspects of the business.

Metrics should support well defined business objectives to provide insight in performance and process improvement. Analysis of metrics requires understanding of statistical process control techniques to correctly interpret trends, set proper control vs. specification limits, determine proper cause and effect relationships, and discern special causes from common cause. The works of Dr. W. Edwards Deming are the authoritative source on these subjects. Here are some references to help you begin your investigation of Dr. Deming's teachings.

General overview of Dr. Deming and links to further sources:
Sources
Description and script for the Read Bead Experiment:
Read Bead Experiment
Description and tools to conduct and understand the Funnel Experiment:
Funnel Experiment

It is important to recognize that collection, analysis and reporting require effort and impact processes. In general, the fewer metrics that achieve the purpose the better. Ideally, the collection processes are non-intrusive and the data flow naturally from the processes being measured.

The technical arena comes replete with many industry standards and best practices. These range from modeling languages to device and service interfaces to support things like platform independence and service oriented architectures. Everything not supported by industry standards is by default a proprietary standard.

Early on in the analysis phase the standards to be utilized must be defined. Frequently the standards are dictated by existing IT architecture and platform standards. As technology changes, architectures must also evolve or be reworked to accommodate newly adopted technology.

Many IT operational metrics are driven by ITIL related processes and service level agreements. Implementation metrics may be put in place to gauge compliance with standards and best practices. Many serve as surrogate measures of complexity and risk.

As with business oriented metrics, through analysis should support specific performance objectives and care must be taken in choosing what to measure. The act of measurement can lead to optimization of performance related metrics to the detriment of other aspects important to long term success.

Other Links