The Audit Advantage

June 8, 2001

The Audit Advantage

Quality system auditing can improve safety, operational integrity

By Louis A. Sorrentino III, Vice President, SH&E

June 2001

The practice of quality system auditing has emerged as one of the most effective diagnostic tools available to all aspects of aviation operations, from air carriers to airports, from ground handlers to airframe manufacturers. Here’s a look at the implementation and merits of such a system.

Say the word "audit" around April 15th and watch what happens: faces pale, panic begins to set in, and horror stories race through people’s minds. Now, say "quality system audit" and most people unfamiliar with organizations and programs such as the European Joint Aviation Authority (JAA) and the JAR OPS 1 standard, similar to our FAA’s FARs (Federal Aviation Regulations), will assume you are referring to some form of customer service quality issue or maybe even quality control. Not so.
Within the aviation community, the words audit and, more recently, quality system audit have taken on an entirely new meaning among management and operations personnel.
In fact, with the advent of the JAA JAR-OPS 1, the benefits of an effective quality system have yet to be fully realized by many of those foreign air carriers required to comply with the standard.

So what is quality system and quality systems auditing? The JAA in JAR –OPS 1.035 states that a quality system shall be established by an operator "to monitor compliance with, and the adequacy of, procedures required to ensure safe operational practices and airworthy aeroplanes."
Further, it states that "The Quality System must include a Quality Assurance Programme that contains procedures designed to verify that all operations are being conducted in accordance with all applicable requirements, standards, and procedures."
So for those operators that have a quality system in place, such as the one outlined in JAR OPS 1.035, one could assume that the operator is conducting quality audits or oversight of the technical functions that support flight operations. In most cases, that assumption would be correct, but like most industries, our aviation industry is not insulated from operational pressures such as over-extending line and staff responsibility, inefficient internal tasking, cost-cutting efforts, expense management, staffing constraints, restricted travel budgets, time and the lack thereof.
To ensure operational integrity, the Quality System Audit has emerged as the most effective tool to determine an Operator’s level of compliance to both the regulations it operates to and the technical programs it has developed to perform specific functions. And to complicate matters, there are many ways in which an audit can be performed.

Auditing History
Most of us familiar with auditing recall a rigid, bureaucratic compliance checklist that either you passed or failed — yes or no; black or white. The style of that audit, the methodology, set the tone, allowing very little in the way of understanding corporate culture, operational constraints, management styles, and so forth.
For the most part, it was merely based on a set of regulations or compliance indices. Usually little tangible feedback was offered from the auditors, because again they were auditing, checking to see if a program or element was evident in writing or that the correct string of words came out of the respondent’s mouth.
In most cases, the appropriateness and effectiveness of the program or element being audited was never determined because that was not the focus. There was no requirement for an evaluation of the way in which different programs or departments (e.g., flight operations and maintenance) worked together.

Process Auditing
Process auditing emerged as the methodology from which a thorough understanding of the administrative and technical elements of an organization’s operation could be attained following examination and evaluation. The intent is to determine the appropriateness and effectiveness of those technical and administrative elements that support the operation.
Process auditing is looking at the entire organization, the way it is organized, the training, the culture, the way in which different departments communicate internally and the meaningfulness of the corporate safety statement or mission statement, to name just a few.
Over the past few years, I have been the beneficiary of working with some very special individuals and companies dedicated to operational integrity and improving the safety performance of the industry. It is within this environment that the concept of systematic process evaluation has grown.
At SH&E, we have developed and refined our evaluation/audit methodology called safety architecture, based on industry best practices observed over the years and elements of ATOS, the FAA’s program for operational integrity known as the Air Transportation Oversight System. ATOS is a systematic approach to FAA certification and surveillance oversight, using safety principals and risk management to ensure that air carriers have safety built into their operating system.
The safety architecture methodology also incorporates ISO 9001 quality system auditing concepts and uses detailed checklists, skilled technical auditors, and selected evaluation techniques which enable auditors to perform forensic assessments of the technical areas they are auditing.
The heart of the safety architecture methodology is a detailed examination of the safety structure (goals, scope, culture, organization) that supports the technical programs (systems, controls, and work process) and what if any risk management system (safety designed into operation, data collection, risk analysis, and warning thresholds) is in place to analyze operational data and decisionmaking processes. This process provides a comprehensive look at the entire company: departments that have any role in the decisionmaking processes of technical operations.

Comprehensive Approach
Why look at the entire company, rather than focusing on just the flight department or the contracted services provider? It is because people outside of what you and I would consider to be the traditional department have direct and indirect decisionmaking authority which effects how that department performs its function. Many times, well-meaning executives are making decisions that can affect the operational integrity of a technical department without a full understanding of the ramifications and downstream effects that their decisions may have.
This is one reason why FAA mandated that air carriers should have a comprehensive and effective safety department as well as a director of safety reporting to the chief executive (see Dec. 1995 Final Rule, "The Commuter Operations and General Certification and Operations Requirements"). In essence, it is so decisions can be made with complete knowledge and understanding of the potential safety repercussions in a given scenario.
Just consider a typical domestic commercial flight: The aircraft is touched by crewmembers; maintenance providers; fueling companies; catering; baggage and cargo handlers; the U.S. Mail; freight forwarders; security screeners; cleaners, lavatory and potable water; deice/anti-icing; bus services; and many more. Our past practice of focusing on just the main three — flight ops, maintenance, and ATC — in accident scenarios leaves an entire population out there that can easily be attributed as the proximate cause of a failure mode (accident).
Recent history has shown that focusing on the pointy end, the folks in the cockpit, is no longer the only area where decisionmaking skills and process integrity need to be evaluated. We have more than enough examples of this in the aircraft accident case studies and NTSB reports.

Ground Services
Another example of the benefits of a program audit can be found in the ground handling business where many of the services provided by ground handlers are not regulated by FAA. Companies offering these services may believe that they have a comprehensive program in place, only to find out, by participating in a program quality audit, that they lack certain performance and integrity elements.
For instance, a company may perform the physical ramp handling, but lack a defined program outlining the standard that it will train its employees to, and possibly miss important elements that would comprise a successful program such as what constitutes acceptable training, qualifications of instructors, types of training offered and timelines for each module, content of training module, instructors’ guides by training module, test development, test grading processes, certification processes, recordkeeping, and data tracking.
In this particular example, while a standard or regulation defining the exact training course material for, say, ramp handlers may not exist, there are industry best practices that have emerged as the best, from a safety and efficiency point of view, based on current knowledge.
Airlines are looking for program quality and integrity practices behind the scenes, as an integral part of the service a company is offering. If it isn’t written down, as in a defined comprehensive program, than how can it be managed or evaluated?

About the Author

Lou Sorrentino is vice president and partner at SH&E, an international air transport consultancy based in Cambridge, MA. He is responsible for its safety practice, and also serves as managing director of the FlightSafety Foundation’s Internal Evaluation Program. He is a commercial pilot and serves on the FlightSafety Foundation’s Int’l Advisory Committee, which promotes flight and ground safety initiatives within the global aviation community and the National Cargo Security Council. He can be contacted at [email protected].