The Supplemental Measurements Library is a product of the IGQ Work Group of the QuEST Forum. The library contains measurement(s) that are not “official” in the sense of being required as part of the TL 9000 Measurement Handbook. However, the measurements can be of value to individual organizations in their own particular circumstances.
The Supplemental Measurements Library began as a repository of measurements that helped support the TL 9000 requirement for process and design measurements (7.3.1.HS.2). Such measurements serve to effectively monitor, manage and improve projects, thus, helping managers to make better decisions.
Recently the Supplemental Library has expanded in the areas of security, performance, pre-deployment and software development. While these measurements are not required as part of the Measurements Handbook, they can be of value to individual organizations that have needs for measurements in these areas.
The Supplemental Measurements Library includes the following groups of measurements:
- General Measurement
- Process Measurements
- Security Measurements
1. General Measurements
The General Measurements section is a group of miscellaneous measurements that organizations have found useful. This group current includes measurement pages for:
Ethernet Frame-Level Performance
This set of performance measurements (Frame Loss Ratio, Frame Delay, and Frame Delay Variation) provides a basis for measuring Ethernet network quality.
Failure Transparency for Technical Support
This measurement gives an indication of the reliability aspect of the software of the system while handling exceptions, using a further classification of problem reports as Problem reports resulting in Service Interruption or Problem reports with Service Continuation.
Invoice Accuracy – Line Items
This measurement provides a basis for evaluating invoice accuracy in order to reduce the cost of invoice processing
IP Packet Performance
This set of performance measurements (IP Packet Transfer Delay, IP Delay Variation and IP Packet Loss Ratio) provide a basis for determining whether a Service Provider has implemented an efficient network design with sufficient levels of network resources for its IP network.
Product Pre-Deployment (PPD)
This set of measurements (Pre-Deployment Problem Reports, Pre-Deployment Problem Report Fixed Responsiveness, Pre-Deployment On-Time Delivery, and Pre-Deployment Supplier Readiness) can be used to measure the organization’s performance as it pertains to activities associated with testing a product at a customer’s site (e.g. a customer’s lab environment), prior to deploying the product in the production environment.
Problem Report Backlog Management
This measurement can be used to manage the open, unresolved problem reports in a software project.
2. Process Measurements
Process measurements provide visibility within the organization about the implementation status and the quality of the product being delivered. They are intended to help identify the need for corrective action and to drive continual improvement within the organization’s design and development processes.
Measurements in this group can help support the TL 9000 requirement for Design and Development Process Quality Measurement Planning and Implementation (Section 7.3.1.HS.2) in the areas of project schedule, test execution, and test phase defect monitoring. They have been defined so as to be applicable to the various phases of design and development
The group of Process Measurements current includes measurement pages for:
Defect Removal Effectiveness
This measurement can be used to measure the effectiveness of a project in finding and removing defects. A good defect removal process promotes the release of products with lower latent defects, generating high customer confidence.
Phase Transition Monitoring
This measurement can be used to provide insight into an organization’s transition criteria, keeping the organization aware of the status of deliverables that may eventually affect product deliverables.
This measurement can help the organizations in assessing the robustness of their development processes by monitoring key schedule milestones. Frequent occurrences of milestone slippage may indicate planning, resource allocation, testing, requirements churn, and other quality related problems.
Peer Review Defect Tracking
This measurement can help provide organizations with data by which they can assess the quality of work and focus efforts on removing defects prior to delivery. This also enables focus on improving the processes and activities that are used to develop the products.
This measurement can help provide management and members of the project team with periodic assessment of the extent to which requirements are traceable to design, development, and test phases that implement and verify product performance compliant with customers’ requirements.
Requirements and Design Stability
This measurement can help provide organizations with data by which they can assess the amount of change in requirements and design of the product during the development process and thus, assess the product’s readiness. The measurement can also help organizations and customers understand the impact of customer requested changes during the development process.
This measurement can help provide organizations with data by which they can assess the progress of their test planning and test execution activities. The information obtained from this measurement should be used for in-process tracking to determine the need for corrective action during a project’s appropriate test life cycle phases.
In addition process measurements, a guidance document is provided to help organizations set up and operate an effective design process measurement system. The document provides useful information on collecting, reporting and analyzing measurements, as well as references to additional information that is available out in the industry.
3. Security Measurements
The Security Measurements are a recent addition to the Supplemental Measurements Library. The Security Measurements guidance document is the result of an Executive Board chartered Next Generation Measurements Security Sub Team to identify existing industry security measurements that would be most relevant to the TL 9000 user community. Based on a study of various Standards Development Organizations (SDOs), the sub team found the published measurements from Center for Internet Security (CIS) and the National Institute for Standards and Technology (NIST) to be most relevant, focusing on operational measurements applicable to an operations environment (not a development environment or supply chain) for the purpose of monitoring the effectiveness of security management within that operation.
The Security Measurements guidance document identifies over thirty measurements to help in the tracking and benchmarking of product security. Measurements include:
Mean Time to Incident Discovery
The measurement can be used to determine the effectiveness in detecting security incidents.
Mean Time to Incident Recovery
This measure can be used to determine the effectiveness in recovering from security incidents.
Security Testing Coverage
This measurement can be used to help track the percentage of applications in the organization that have been subjected to security testing.
Security Related Outages
These measurements are a subset of those in SONE, focusing on outages caused by security issues.
The Supplemental Measurements Library is supported by the IGQ In-Process Measurements sub-team. The measurements typically originate from sub-teams of Work Groups. However, any organization may petition measurements that it has found useful to be in the library. Please contact Kathy Musumeci (firstname.lastname@example.org) for more information on the Supplemental Measurements Library.