June 2012 -

Cost of Quality (COQ)

Manufacturing, inspection, exclusions

Q: Hello, I am having trouble locating basic summary information about the “cost of quality” for various industries. In brief: what is the cost of NOT having good quality in specific industries, such as automotive. Put another way, what is it worth to a “typical” company or industry to enact good or better quality practices? I realize this is an abstract question with many underlying variables, and looked at some old (1999) work on this, but am seeking any summary information you might know of. Thanks so much.

A: Thank you for contacting ASQ and the Quality Information Center. I received your request for some information regarding cost of quality.

Donald L. Siebels, in his book The Quality Improvement Glossary, defines quality costs as “a measure of the cost specifically associated with the achievement or nonachievement of product or service quality, including all product or service requirements established by the company and its contracts with customers and society. More specifically, quality costs are the total of the cost incurred by (1) investing in the prevention of nonconformances to requirements; (2) appraising a product or service for conformance to requirements; and (3) failure to meet requirements. These can then be categorized as prevention, appraisal, and failure”. Additionally, Siebels defines the cost of poor quality (COPQ) as “costs associated with providing poor quality products or services”.

For more information on this topic, please visit ASQ’s website.

ISO 17025 Clause 5.4.2 – Selection of Methods

Q: We are working with the Mexican Accreditation Entity (EMA) for certification to ISO/IEC 17025:2005 General requirements for the competence of testing and calibration laboratories. Clause 5.4.2 states: The laboratory shall confirm that it can properly operate standard methods before introducing the tests or calibrations.

We are a testing laboratory and work with Method 21 – Determination of Volatile Organic Compound, EPA 40 CFR Ch.1 ( 01/07/04 Edition ) Test: Monitoring of Fugitive Emissions.

The question is: What would be the best way or a way to confirm the method? Or, to put it another way, how can we satisfy the requirements in clause 5.4.2 ?

A: The questioner is referring to clause 5.4.2 from ISO/IEC 17025:2005. An excerpt of this clause is below. Please refer to ISO/IEC 17025:2005 for the full clause.

5.4.2 Selection of methods

“…Methods published in international, regional or national standards shall preferably be used….. Laboratory-developed methods or methods adopted by the laboratory may also be used if they are appropriate for the intended use and if they are validated…. The customer shall be informed as to the method chosen. The laboratory shall confirm that it can properly operate standard methods before introducing the tests or calibrations.…”

Since the questioner is using the published methods, there is no need for validation of the method unless the method is modified.

However, the proficiency of being able to apply the published method needs to be demonstrated. This can be demonstrated by a documented Gage R & R study, Analysis of Variance (ANOVA) or Design of Experiments (DOE) study as appropriate to show proficiency in being able to utilize the test method properly.

The results from these studies may also be used to estimate the uncertainty of measurement for the tests. Reporting uncertainty of measurement with both test and calibration results is a requirement in ISO/IEC 17025:2005. The ILAC P14 document is a good guidance document on reporting uncertainty.

Dilip A Shah
ASQ CQE, CQA, CCT
President, E = mc3 Solutions
Chair, ASQ Measurement Quality Division (2012-2013)
Secretary and Member of the A2LA Board of Directors (2006-2014)
Medina, Ohio
http://www.emc3solutions.com

For more on this topic, please visit ASQ’s website.

Voice of the Customer (VOC)

About ASQ's Ask the Standards Expert program and blog

Q: I’m looking for general information on “Voice of the Customer”. Specifically, I’m putting a slide show presentation together and was interested in any guidance that you could offer, thanks.

A: Thank you for contacting ASQ. I received your request for information regarding voice of the customer. Voice of the customer can be defined as “expressed requirements and expectations of customers relative to products or services, as documented and disseminated to the members of the providing organization” (from The Quality Improvement Glossary by Donald L. Siebels).

For more on this topic, please visit ASQ’s website.

Visual Fill Requirements

Pharmaceutical sampling

Q: I work for a consumer products company where more than 60% of our products have a visual fill requirement. This means, aside from meeting label claim, we must ensure the fill level meets a visual level.

What is the industry standard for visual fills?

We just launched Statistical Process Control (SPC), and we notice that our products requiring visual fills show significant variability.

A: This is an interesting question. The NIST SP 1020-2 Consumer Package Labeling Guide and the Fair Packaging and Labeling Act, along with any other industry standards, regulate how you must label a product “accurately.” However, it appears you have been burdened with a separate, and somewhat conflicting requirement — a visual fill requirement.

In most cases, you probably cannot satisfy both requirements without variability. The laws and standards will direct labeling requirements with regard to accuracy, and your company is liable for that. If you choose to use visual fill standards for “in-process” quality assurance, then you would need a fairly broad range between the upper and lower acceptance limits.

Personally, I would use weights and measures as needed to meet customer and legal requirements. These are the data I would use for SPC records.

If your company has a need (or a desire) to use visual fill levels for a gage, then generating a work instruction telling employees where a caution level is would be a way to start. In other words, “If the visual level is above point A or below point B, immediately notify management.” If you are to remain compliant with what you put on a label, visuals will change from run to run. Using them as a guide for production personnel can be a helpful tool, but not as a viable SPC input.

Bud Salsbury
ASQ Senior Member, CQT, CQI

For more on this topic, please visit ASQ’s website.

Design of Experiments (DOE)

ISO 13485, medical devices, medical device manufacturing, design of experiments

Q: I am looking for research articles or review papers on Design of experiments (DOE) specially focused on Response surface methods, Split Plot designs, MANOVA, and Repeated measures designs and analysis. Any help to locate these articles will be greatly appreciated.

A: Thank you for contacting ASQ. I received your request for information on the topic of Design of Experiments (DOE). Design of Experiments is defined as “a method for carrying out carefully planned experiments on a process. By using a prescribed plan for the set of experiments and analyzing the data according to certain procedures, a great deal of information can be obtained from a minimum number of experiments” (from The Quality Toolbox, 2nd Ed. by Nancy R. Tague, Quality Press, 2005).

Use the link below to review information on your topics of interest.

Quality in Radical Innovation

Employees, Training, Working, Learning, Duties, Tasks, DFSS, Innovation

Question

Design for Six Sigma (DFSS) involves the discovery, development, and understanding of critical to quality areas and fosters innovation. However, studies have shown that using focus groups, interviews, and etc., based on current users only bring forth ideas relative to incremental innovation, as the only knowledge that most customers have is of current products. But we know that the greatest potential for return is in radical innovation.

My question is: what useful tools are there for determining critical to quality areas of radical innovation products, or products that are new to market where customers have little to no knowledge of?

Answer

These are great questions that are not easy to answer as posed.

One of the dilemmas I’ve seen with companies building radical innovation without enough knowledge to identify the important quality aspects is that the company is often under intense pressure to get to market. In some cases, the innovation presents clear aspects that have to be controlled to create an acceptable product. In some cases, the issues are unknown.

I do not agree the work within a group only reflects the knowledge already present. One of the best tools in these situations is carefully crafted questions posed to those most familiar with the new technology. Given my personal bias, I would ask: “What will fail? Why?” and then ask about material, process, and feature performance variation. Focusing on the failure mechanisms and variation will often lead the team to uncover those aspects of the product that require well crafted specifications and monitoring.

Not a fancy tool, just a question or two. Yet, the focus is on what will cause the innovation to not meet the customer’s expectations. What could go wrong? Make it visible, talked about, and examined. Creating a safe atmosphere (no blame or personal attacks) to explore failure permits those most vested in making the product work examine the boundaries and paths that lead to failure.

Once the process of safely examining failures starts, a range of tools assist with the refinement and prioritization. Failure Modes and Effects Analysis (FMEA) and Highly Accelerated Life Testing (HALT), provide means to further discover areas to explore the paths to failure. I mention creating a safe environment first, because using FMEA and HALT when someone’s reputation or status is threatened generally leads to these tools being very ineffective.

One more thought on a safe environment for the exploration of failures. Focus on the process, materials and interaction with customers and their environment. “How can we make this better, more resilient, more robust, etc.?” Not, “Why did you design it this way?” or, “This appears to be a design mistake.” All involved have the same goal to create a quality product or service, yet there may be a lot unknown related to those conditions that lead to product failure. An open and honest exploration to discover the margins and product weaknesses is most effective in a safe environment for those concerned. And, by the way, this includes vendors, contractors, suppliers, and all those involved with the supply chain, development and manufacturing processes.

Fred Schenkelberg
Voting member of U.S. TAG to ISO/TC 56
Voting member of U.S. TAG to ISO/TC 69
Reliability Engineering and Management Consultant
FMS Reliability
fmsreliability.com

For more on this topic, please visit ASQ’s website.

Defining Qualification, Verification, and Validation

Q: I understand the hierarchy, but I would be hard pressed, if asked, to give a clear definition of the terms: qualification, verification, and validation. Can one of the experts help explain these terms? Thank you.

A: This is a great question and I hope I’ll be able to help you.

To begin, I refer you to ISO 9000:2005 Quality management systems – Fundamentals and vocabulary. As you may already know, this document is used to define/describe many terms used in the ISO 9000 series, including the three words you question.

In 9000:2005, under clause 3.8 Terms relating to examination, we find:

3.8.4 verification
Confirmation, through the provision of objective evidence, that specified requirements have been fulfilled
NOTE 1 The term “verified” is used to designate the corresponding status.
NOTE 2 Confirmation can comprise activities such as
–          performing alternative calculations,
–          comparing a new design specification with a similar proven design specification,
–          undertaking tests and demonstrations, and
–          reviewing documents prior to issue.

3.8.5 validation
Confirmation, through the provision of objective evidence, that the specified requirements for a specific intended use or application have been fulfilled
NOTE 1 The term “validated” is used to designate the corresponding status.
NOTE 2 The use conditions for validation can be real or simulated.

Validation definition, as provided by ASQ's Quality Glossary.

3.8.6 qualification process
Process to demonstrate the ability fulfill specified requirements
NOTE 1 The term “qualified” is used to designate the corresponding status.
NOTE 2 Qualification can concern person, products, processes or systems.
EXAMPLE Auditor qualification process, material qualification process.

I’ll try to expand on these definitions in hopes of making things a bit more clear. Keep in mind that qualification, verification, and validation are individual processes, but the explanations below (from Boston Scientific) should help you recognize their individuality as well as their interdependence.

Validation is an act, process, or instance to support or collaborate something on a sound authoritative basis.

Verification is the act or process of establishing the truth or reality of something.

Qualification is an act or process to assure something complies with some condition, standard, or specific requirements.

For example:

A design verification verifies that a frozen (static) design meets top level product specifications.

A process validation validates that the on-going (dynamic) manufacturing process produces product that meets product/print specifications and consist of installation qualifications, operational qualifications, process performance qualifications, a product performance qualification and perhaps process verifications.

An installation qualification qualifies that equipment was installed correctly and are a subset of a process validation (or possibly a test method validation).

Validation Examples:
• Design validation, sterilization validation, test method validation, software validation, and process validation.

Verification Examples:
• Design verification and process verification.

Qualification Examples:
• Installation qualification, operational qualification, process performance qualification, product performance qualification, and supplied material qualification.

After reading all of this, I am confident you would be able to explain qualification. An old and trusty phrase to help summarize the other two is: Validation – Are we producing the right product?; Verification – Are we producing the product right?

Bud Salsbury
ASQ Senior Member, CQT, CQI

ISO 9001 & Time to Retrieve Records

Q: I am looking for an interpretation for ISO 9001:2008 Quality management systems–Requirements, clause 4.2.4 Control of records: “Records shall remain legible, readily identifiable and retrievable.”

What is considered readily retrievable (i.e., 24 hrs, 48 hrs, 8 hrs, 1 hr)? I have a customer who thinks traceability records should be available within an hour of a request. I interpret readily as 24 hrs. The current ISO and TS specifications do not indicate a time, so a reasonable time to me is 24 hrs to pull the information together.

In addition, the customer’s supplier requirements also do not have any specified time for document retrieval. I did contact our third party registrar auditor and he indicated that 24 hrs would be considered readily retrievable as long as there were no customer specific requirements.

A: There appears to be some confusion between records being “readily retrievable” vs. a customer’s request for the delivery of copies of records. These are two separate issues.

The first issue: What is meant by “readily retrievable?” ISO 9001 does not prescribe any specific timeline or define the term “readily retrievable.” However, the intent of this requirement is to ensure that objective evidence is available to provide proof of conformance or evidence that requirements have been met. If the organization is unable to provide records upon request during an audit, the auditor will very likely document this as a nonconforming condition. Records must be available upon demand.

The second issue is response time to customer requests for records. Although records or evidence of conformance may be “readily retrievable” within the organization, the response time needed for an organization to provide copies of records to a customer may vary based upon the organization’s work load and availability of resources. So, it may take an organization an hour, a day or a week to deliver copies of records to a customer. In the event that the timely delivery of records is critical, requirements for the delivery of records should be stated in a contract or in a PO to provide a timeline or a delivery schedule. The delivery of copies of records or documents to customers is not addressed in ISO 9001, clause 4.2.4.

Bill Aston
ASQ Senior Member
Managing Director of Aston Technical Consulting Services
Kingwood, TX
www.astontechconsult.com

For more on this topic, please visit ASQ’s website.

AS9100C: Scoring the Aerospace QMS

Airplane, aerospace, AS9100

Q: I’m reviewing the scoring method used for auditing AS9100C – Requirements for Aviation, Space and Defense Organizations, and I don’t see any verbiage to show what would be considered an acceptable overall score. I’m curious to know if the score is more subjective to the discretion of the auditor or if the threshold for “acceptable” or “not acceptable” exists somewhere as a guideline. Thank you to anyone able to offer insight.

A: The AS9101D auditing standard (currently not sold by ASQ) has scoring to provide an indicator of how robust your quality management system is operating (QMS), which is based upon the findings identified during your audit. There is not a required score to “pass” the audit and receive certification. The AS9101D score is recorded in the OASIS database, which your current and potential customers may review.

AS9100C requires the use of the AS9101D auditing standard, which has eliminated scoring.

Buddy Cressionnie
International Aerospace Quality Group Americas AS9100 Lead
Voting member of the U.S. TAG to ISO/TC 176
Southlake, TX

For more on this topic, please visit ASQ’s website.

Calibration of AutoCAD Software

About ASQ's Ask the Standards Expert program and blog

Q: To what extent must an engineering firm, specializing in railway infrastructure and transportation, have its AutoCAD software “calibrated” or verified?

Also, what about software designed to calculate earthwork quantities for railway alignments laid out on topographic mapping for all levels of studies – pre-feasibility through preliminary engineering (not for final design, operation simulation and design dynamic system models)? This type of software is utilized by competent draft persons and engineers, but it is not verified prior to use or periodically calibrated.

We don’t confirm “the ability of computer software to satisfy the intended application…”

Your assistance or reference is appreciated

A: AutoCAD is considered “Commercial -Off-The-Shelf” (COTS) software. It is purchased without modification and cannot be modified by the end-user. A similar example would be Excel spreadsheet software. The COTS software by itself should be considered validated and used as is provided it is configured per the software manufacturer’s instructions.

The functionality of the software (distance, volume, formulae and other functions) is fit to be used as intended. If an application is created using COTS software (Excel Templates, AutoCAD applications), then it must be validated and records of validation must be kept.

It should also be noted that definitions of verification and validation are not clearly understood. So, I am repeating them here:

ISO/IEC Guide 99:2007—International vocabulary of metrology—Basic and general concepts and associated terms, defines these terms as:

Verification: provision of objective evidence that a given item fulfills specified requirements

Validation: verification, where the specified requirements are adequate for an intended use

Further explanation:

Validation is a quality assurance process of establishing evidence that provides a high degree of assurance that a product, service, or system accomplishes its intended requirements. This often involves acceptance of fitness for purpose with end users and other product stakeholders.

It is sometimes said that validation can be expressed by the query “Are you building the right product?” and verification by “Are you building it right?”

“Building the right thing” refers back to the user’s needs, while “Are we building the product right?” checks that the specifications are correctly implemented by the system. In some contexts, it is required to have written requirements for both as well as formal procedures or protocols for determining compliance.

For more on this topic, please visit ASQ’s website.