Measurement, Analysis, Improvement
Measurements should induce the parts to do what is good for the whole, and measurements should direct managers to those parts that need their attention. – Eliyahu Goldratt
Management decisions that are based upon information obtained by dashboards that are built from basic key process indicators (KPIs) alone will not satisfy the needs of the business, or your stakeholders. Dashboards are use basic KPIs with the expectation that they will provide direction… they will not. Dashboards only present basic KPI information results in people staring at the dashboard like deer staring at headlights… bewildered and ‘frozen’, without answers, and without the information they need to make the informed decisions that are required of responsible management. So what is the correct use of KPIs?
Basic KPI information, however, is extremely valuable if it is appropriately utilized within algorithms designed to define results that are associated with operational objectives. Moreover, when these results are properly designed to be aggregated as components within the system as a whole, they then provide the capability to analyze systemic constraint in addition to emerging quality issues within the scope of the objective. This is very powerful for management control over the system. How then are KPIs appropriately utilized within algorithms?
Simple metrics provide basic information in the form of quantitative data. KPIs are created when data from simple metrics are combined to provide quantitative information. KPIs are then used to build analytical algorithms that are defined to provide understanding about just what it is that we want to know about current circumstances, and therefore are strategically constructed to be predictive in nature. Analytical algorithms enable us to better understand what could happen if the underlying drivers of the data are not modified. Well-designed algorithms enable responsible management to provide the intelligent direction that is referred to within my favorite quote:
“Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives.” - origin unknown
I use this quote often because I believe it to be accurate and succinct. The “high intention, sincere effort, intelligent direction” must come from top management… those who are responsible to ensure that the “skillful execution” of the organizations’ system, by the employees who work within the system, will yield high-quality results.
Responsible management must ensure the sub-system that is needed to measure-analyze-improve the QMS is well designed and well defined. Such a system would then be available to top management of being capably executed, measured, monitored, analyzed, reported, and reviewed to enable the organization to responsibly address the constraints and issues of an the QMS. It is up to responsible management to ensure that this type of active management of the QMS exists as an independent and managed component within the QMS itself.
The medical device industry utilizes the EN ISO 13485:2016[1] standard to define the foundational requirements of a Quality Management System (QMS). The element of Measurement, Analysis, Improvement is found within 13485:2016, § 8 and establishes the requirements for managing the ‘quality’ of the Quality Management System (QMS).
The remainder of this paper is intended to provide both structure and guidance for the creation of the Measurement, Analysis, Improvement section within a Quality Manual, that can be utilized to actively manage the ‘quality’ of the Quality Management System. This will enable a more technical and less superficial approach to this important management responsibility. Each section will contain both policy and discussion elements that reflect the paradigm and opinion of this writer, nothing more. However, these opinions were formed from experience, and may hold meaning for those Quality Management Professionals who are working to improve the ‘quality’ of medical devices offered by their organization to the healthcare market.
Note: - The Mgmt-Ctrl Quality System(s) Management (QSM) solution was designed to strategically measure, manage and improve the quality of products and processes, inclusive of the cost of poor quality. The QSM solution will completely satisfy the requirements for 13485:2016, § 8, Measurement, Analysis and Improvement. It is our mission to support patient care by improving product and process quality within the medical device industry, through the provision of the QSM solution. The Quality System(s) Management (QSM) solution is now available to the medical device industry.
-----------------------------------------------------------------------------------------------------------------------------
8 – Measurement, Analysis and Improvement
The organization’s product and process quality is measured, monitored, analyzed, reported and reviewed utilizing a system of measurement-analysis-improvement that is designed to strategically manage the ‘quality’ of the Quality Management System (QMS). This quality system management sub-system identifies and defines the appropriate methodology and technology, inclusive of statistical techniques, to strategically manage systemic information that is utilized by responsible management to identify and address emerging quality issues and systemic constraints to provide ongoing and continual improvement over the product ‘quality’ that is resultant from the execution of production processes and QMS processes alike.
8.1 – General
Policy
The Quality Management System (QMS), defined within this Quality Manual, is measured, monitored, analyzed, reported and reviewed, as defined within SOP XXXXX, Quality System Management, to:
demonstrate the conformity of product;
ensure the conformity of the QMS; and
maintain the effectiveness of the QMS.
SOP XXXXX identifies the appropriate methods, including statistical techniques, and the extent of their use to ensure the QMS processes are:
measured to identify emerging quality issues and process constraints;
managed to ensure the appropriate escalation of issues and constraints; and
improved to address identified issues and constraints.
Discussion
Quality management (the management of ‘quality’) is an active endeavor that requires a systemic approach to actively measure, manage, and improve product and process quality. It is an unreasonable expectation that ongoing continual improvement will appropriately occur simply because that requirement is stated within a job description and/or a quality policy.
Employees within the organization are expected to execute their defined work within the QMS to fulfill their obligation to the company, enabling the company to fulfill its obligation to the patients and users who count on its products for safe and effective patient care. The provision of an independent and systemic approach to identify and target QMS issues and constraints for improvement ensures the ongoing capability for the compliance, effectiveness, and efficiency of the work executed by the employees who work within the system.
8.2 – Monitoring and Measurement
Specific requirements of monitoring and measurement are further defined per 13485:2016, § 8.2.1 through § 8.2.6, below.
8.2.1 – Feedback
Policy
The methods for obtaining and using product and process quality information as feedback to ensure customer requirements are being met are defined within SOP XXXXX, Quality System Management. This system of product and process ‘quality’ measurement:
includes provisions to gather data from production as well as post-production activities;
serves as input (feedback) to risk management activity for monitoring and maintaining product requirements;
serves as input (feedback) to risk management activity for monitoring and maintaining process requirements;
serves as input (feedback) to product realization activity;
serves as input (feedback) to product and process improvement activity; and
serves as input (feedback) to applicable regulatory activity requiring the review of post-production activities as feedback.
Discussion
Feedback requires a process defining the measurement, monitoring, analysis, reporting and review of product and process parameters, in the form of output data, that affect and determine ‘quality’. The requirements to ensure valid results associated with each of these discrete activities (measure | monitor | analyze | report | review) must be independently defined. It is therefore recommended that SOP XXXXX be created as a ‘top level’ SOP that identifies subordinate procedures specifically and independently defining measurement, monitoring, analysis, reporting and review activity and requirements that ensure quality objectives (13485:2016, § 5.4.1) are achieved.
8.2.2 – Complaint Handling
Policy
The timely handling of complaints is defined within SOP XXXXX, Complaint Handling, ensuring regulatory requirements are met for this crucial product quality feedback and data/information management process. This process includes requirements and responsibilities for
receiving and recording information
evaluating information to determine if the feedback constitutes a ‘complaint’;
investigating complaints;
ensuring any complaints that are not investigated are appropriately justified, and that such justification is appropriately documented;
determining the need to report the information to the appropriate regulatory authorities;
handling of complaint-related product;
determining the need ‘escalate’ to initiate potential corrections and/or corrective actions;
ensuring the exchange of relevant information related to activities conducted by external parties who may have contributed to the issue (complaint), and documenting that information exchange;
providing input (feedback) to risk management activity for monitoring and maintaining process requirements;
providing input (feedback) to product realization activity;
providing input (feedback) to product and process improvement activity; and
maintaining complaint handling records in accordance with 13485:2016, § 4.2.5, Control of Records.
Discussion
While post-market surveillance (PMS) activity can be used to measure residual risk using external data, the complaint handling system has the ability to provide internal data to augment the data derived from PMS activity.
Complaints received about an organizations’ products serve as an indication that the product failed to meet the requirements of the market. This information should be consistently utilized in review of the Product Realization process in accordance with 13485:2016, § 7.
8.2.3 – Reporting to Regulatory Authorities
Policy
The timely notification and reporting to regulatory authorities of complaints that meet specified reporting criteria of adverse events, or criteria for the issuance of advisory notices, is defined within SOP XXXXX, Reporting to Regulatory Authorities, inclusive of the requirements for maintenance of notification and reporting records in accordance with 13485:2016, § 4.2.5, Control of Records.
Discussion
When adverse events are identified, and advisory notices must be issued, worldwide requirements for doing so must be completed as prescribed by each regulatory authority, in a specific manner and timeframe for doing so. These regulatory requirements must be incorporated into the QMS.
8.2.4 – Internal Audit
Policy
The requirements and responsibilities for conducting internal audits at planned intervals, inclusive of recording and reporting the results, is defined within SOP XXXXX, Internal Auditing. The purpose of internal auditing is to determine if the Quality Management System is:
compliant to regulatory requirements and applicable standards;
effectively implemented and maintained;
effective in obtaining results that satisfy quality objectives in accordance with 13485:2016, § 5.4.1;
The management responsible for the area being audited ensure escalation of identified issues for necessary corrections and corrective actions without undue delay.
Records of the audits and their results, including identification of the processes and areas audited and the conclusions is maintained in accordance with 13485:2016, § 4.2.5, Control of Records.
Discussion
Your internal auditing group can make a significant difference towards understanding the compliance and effectiveness of the processes within organizations’ QMS… if allowed to do so. All too often that group is solely focused upon compliance to regulatory requirements. There are, however, three aspects to measuring compliance, which is a binary measurement, Y | N :
compliant, as written, to regulation/standard, requirements;
compliant in adherent execution to procedural requirements, as written;
compliant in achieving effective results, because without effective results the process isn’t even useful to the company in the manner as intended by the regulation/standards.
The Internal Audit group should be provided the authority and independence to assess the organization in such a manner so as to provide accurate and important feedback to responsible management.
8.2.5 – Monitoring and Measurement of Processes
Policy
Suitable methods for measurement and monitoring processes, to determine if the processes achieved results as components within the Quality Management System (QMS), are defined within SOP XXXXX, Quality System Management.
Discussion
QMS processes should be measured to be:
compliant, as written, to regulation/standard, requirements;
compliant in adherent execution to procedural requirements, as written;
effective in achieving planned results; and
efficient in operational execution.
The individual ‘quality’ of each of processes within the system of processes (QMS), contribute to the ‘quality’ of the system as a whole.
8.2.6 – Monitoring and Measurement of Product
Policy
Suitable methods for measurement and monitoring CtQ (critical to quality) product characteristics (CQA – critical quality attributes) and production process characteristics (CPP – critical process parameters) in addition to post-market residual risk, to determine if the ‘quality’ requirements of the product are being achieved, are defined within SOP XXXXX, Quality System Management.
Evidence of conformity to the acceptance CQA & CPP criteria is maintained in accordance with 13485:2016, § 4.2.5, Control of Records. Product release and service delivery occur only after planned and documented arrangements to determine conformity have been satisfactorily completed.
Discussion
Product ‘quality’ can be measured throughout the production phase of the product realization process and in conjunction with post-market surveillance (PMS) activity. The ‘quality’ of the product to be produced (as designed and validated) using the production process (as validated), is determined through a thorough understanding of the capability of the product characteristics (CQA) and process parameters (CPP), in addition to the understanding of the variation (common cause and special cause variation) associated with these characteristics.
8.3 – Control of Nonconforming Product
Specific requirements for the control of nonconforming product are defined per 13485:2016, § 8.3.1 through § 8.3.4, below.
8.3.1 – General
Policy
Product that does not conform to product requirements is identified and controlled to prevent its unintended use or delivery, as defined within SOP XXXXX, Nonconformance.
This procedure defines the associated controls, responsibilities and authorities to ensure the identification, documentation, segregation, evaluation, and disposition of nonconforming product.
All nonconforming product is subjected to the need for investigation and the notification of any external party that may be responsible for the nonconformity.
Records of nonconformance, and associated actions, are maintained in accordance with 13485:2016, § 4.2.5, Control of Records.
Discussion
A nonconformance is simply the failure to fulfill a documented requirement. It is interesting to this writer that there is no discussion of nonconformance in association with the failure to fulfill the requirements associated with a QMS process. A nonconformance procedure should be designed to be capable of handling both product and process nonconformity. Additionally, the rules of escalation to CAPA and/or Field Corrective Action (for product) should be contained within the Nonconformance procedure. In doing so, the requirements of following sections can be addressed within consideration of the defined escalation requirements:
8.3.2 – Actions in response to nonconforming product detected before delivery
8.3.3 – Actions in response to nonconforming product detected after delivery
8.3.4 – Rework
Policy
Rework of product is performed in accordance with SOP XXXXX, Rework, and includes consideration of the potential adverse effect(s) of the rework on the product quality. Regardless, reworked product is submitted to the same review and approval as product that did not undergo rework activity. All reworked product is fully verified to ensure it meets applicable acceptance criteria and regulatory requirements.
Records of rework are maintained in accordance with 13485:2016, § 4.2.5, Control of Records.
Discussion
Rework of product should be considered as an exception to a defined production process and not as a routine event within a production process. Believe it or not, there are still medical device companies that allow their production workers to continually ‘rework’ the product until it meets specification. Henry Ford invented the ‘production line’ in 1913, where defined methodology replaced antiquated ideas of building cars haphazardly. The ‘quality’ of a medical device should be determined by design, and controlled by validated and verified CQA’s and CPP’s, not by the random opinion of production employees who act as ‘artists’.
8.4 – Analysis of Data
Policy
The methods for analysis of product and process quality are defined within SOP XXXXX, Quality System Management.
Discussion
The organization’s product and process quality is measured, monitored, analyzed, reported, reviewed, and improved utilizing a system of measurement-analysis-improvement that is designed to strategically manage the ‘quality’ of the Quality Management System (QMS). This quality system management sub-system identifies and defines the appropriate methodology and technology, inclusive of statistical techniques, to strategically manage systemic information that is utilized by responsible management to identify and address emerging quality issues and systemic constraints to provide ongoing and continual improvement over the product ‘quality’ that is resultant from the execution of production processes and QMS processes alike.
8.5 – Improvement
Policy
The methods for improvement of product and process quality are defined within SOP XXXXX, Corrective and Preventive Action.
Discussion
The organization’s product and process quality is measured, monitored, analyzed, reported and reviewed and improved utilizing a system of measurement-analysis-improvement that is designed to strategically manage the ‘quality’ of the Quality Management System (QMS). This quality system management sub-system identifies emerging quality issues and systemic constraints that are to addressed through the Corrective and Preventive Action process.
---------------------------------------------------------------------------------------------------------------------------
Summary | Conclusion
The ability of an organization to manage product and process quality is only achieved when products and processes are measured, monitored, analyzed, reported, reviewed, and improved utilizing a system of measurement-analysis-improvement that is designed to strategically manage the ‘quality’ of the Quality Management System (QMS). The quality of your products are simply a reflection of the quality of design, production and QMS processes. Design and production processes are controlled from within the QMS. Measure, monitor and improve the ‘quality’ of the QMS… and you control the ‘quality’ of your product.
The Mgmt-Ctrl Quality System(s) Management (QSM) solution was designed to strategically measure, manage and improve the quality of products and processes, inclusive of the cost of poor quality. The QSM solution will completely satisfy the requirements for 13485:2016, § 8, Measurement, Analysis and Improvement. It is our mission to support patient care by improving product and process quality within the medical device industry, through the provision of the QSM solution. The Quality System(s) Management (QSM) solution is now available to the medical device industry.
[1] EN ISO 13485:2016, Medical devices – Quality management systems – Requirements for regulatory purposes