A recent government-industry workshop identified the big challenges associated with collecting, analyzing, and using information from voluntary reporting systems. This article summarizes the proceedings showing that one size does not fit all but that effective use of data is good for organizations of all sizes.
The annual workshop
For four consecutive years, the FAA’s Office of Aviation Safety (AVS) Chief Scientific and Technical Advisory (CSTA) program, and the Human Factors Division of the Civil Aerospace Medical Institute (CAMI) conducted an annual workshop dedicated to maintenance human factors. The 2013 workshop addressed not only challenges but also solutions associated with the collection, analysis, use, and effectiveness evaluation of voluntarily reported event data.
Twenty-three invited attendees came from government, research and development, manufacturing, airlines, and maintenance, repair, and overhaul organizations. They worked together to show success stories on use of data. There are many organizational and process-oriented factors that affect collection, analyses, and implementation. The organization’s management, the collective labor force, the individual worker, and the FAA share the responsibility for success. The group identified the top ten challenges and solutions (details in 60 page final report to be published in November)(Avers and Johnson, 2013 (www.hfskyway.faa.gov). Here are three selected examples:
- Corporate and individual resistance
- Data, data, data - Consistency in data analysis
- Scaling data systems to any size organization
Let’s look at these three categories.
Corporate and individual resistance
Generally speaking, most segments of the aviation maintenance industry are a bit reluctant to collect data. That situation has a number of contributing factors ranging from:
- corporate lawyers who are not comfortable about documenting error,
- mid-level managers who are pressed more by delivery schedules than by data collection,
- individual mechanics who are fearful of reprise from the employer or FAA and from ridicule from co-workers.
The transition from hiding negative events to reporting and learning from errors requires time and corporate commitment. Some organizations are making the necessary cultural change faster than others. Most of the workshop delegates were from organizations that are making the necessary safety culture transition and were seeing the high value of this change. They are the early adopters of best application of voluntarily reported data. All reported success stories as well as growing pains. For example:
- AAR Corp. mentioned that it used maintenance and the FAA Return on Investment procedure to show how fatigue training reduced damage and employee injury.
- Delta Airlines said that the data helps the company learn what it does not know because of employee reports.
- Southwest Airlines used its data to see how errors increased as hours on duty increased.
The primary message is that the organization needs demonstrated senior commitment and support for optimal collection and use of event data. The commitment must permeate everyone in the organization regardless of the organization size.
Of course, one might say that evolving requirements for safety management systems (SMS) will ensure data collection. However, there is usually a difference between meeting a regulatory requirement and fully capitalizing on the “spirit” of the regulation. This article is about the “spirit” and the process more that about regulatory compliance.
The workshop delegates made suggestions on how to overcome corporate and individual reluctance to collecting and using data. First, all parties must see “What’s in it for me?” For the individual they may see that organizational changes, like new procedures or equipment, are a result of voluntary reporting systems. Management may see that voluntary reporting, a key element of a safe culture, reduces injuries, delays, and rework. Everyone wins!
The delegates said that successful organizations are using voluntary reports to communicate an awareness of hazards and hazard reporting before they contribute to unacceptable risk to the organization. Some commented that such communications can also include information about how small safety investments have large financial return and impact on flight and worker safety. See the final report for more detail.
Data, data, data
Challenges associated with data are inseparable from the cultural issues described above. Without universal commitment to data collection, analysis and application are unlikely to succeed. At the same time, poor implementation of a safety data system can affect the commitment of both management and labor.
Many challenges were discussed like: format, inefficiency, experience with analysis/interpretation, logical implementation, and automated report generation tools. None of these issues are trivial but there are solutions. Once the senior corporate commitment is demonstrated (i.e. sufficient funding allocated) the work can commence/continue. Delegates warned that selecting the data team is critical. A group of bright mechanics and managers will fail. Likewise a group of experienced data analysts will fail. Someone must understand the technical content and another person must understand the analytic process. It take a multi disciplines to ensure success.
There must be some method to the madness of data collection. For example, different departments must coordinate their efforts. An avionics shop has different issues than an engine assembly shop. The flight line is certainly different than the hangar. However, they are likely more similarities than differences. Shared data reporting formats will ensure that analytic procedures can also be shared. Safety management, based on data, is a corporate goal rather than a host of organizational stovepipes doing their own thing.
Start small and you are more likely to succeed. More than 20 years ago I was teaching a human factors course for an MRO in Hong Kong. I distinctly recall asking the safety manager if he knew where the “problems” (called hazards today) were. He leaned forward in his chair, with a big smile, and said “yes we do.” He pulled a file from the cabinet behind him and showed Excel charts with problem categories and counts. He asked me to address these issues in the five-day HF class.
In those days it was not a requirement for HF training much less a requirement for SMS. Instead of regulatory compliance, they had a corporate goal to reduce human error which would decrease cost and decrease heavy maintenance time. By the way, he was using voluntary reports based on Boeing’s Maintenance Error Decision Aid (MEDA) process. MEDA is adaptable to all size organizations. In addition, MEDA was identified as the basis for the maintenance data systems for most of the workshop delegates.
Learn from others. Safety data collection is hardly a trade secret. Pick up any magazine, like Aircraft Maintenance Technology, and there are articles about data. In October the Flight Safety Foundation AeroSafety World dedicated two articles to the topic. Dr. Bob Baron covers a number of issues, including “Garbage in, garbage out.” FAA’s Advisory Circular on the Aviation Safety Action Program (AC 120-66B) is another source on establishing a data collection system.
Find ways to demonstrate that your data collection efforts are not in vain. Measure your success not by the amount of data collected but rather by the amount for positive safety impact.
Scaling data systems to any size organization
One size does not fit all but proper effective data use is good for everyone. The delegates acknowledged that General Aviation and small carriers did not have sufficient representation at the meeting. In fact, the industry representation was from the larger carriers including Delta, American, Southwest, and United. AAR Corp. represented the MRO viewpoint. All these companies were examples of best case scenarios for collection and use of data. But, the big carriers and MROs do not have a monopoly on common sense, on safety culture, or on the ability to collect and use data. Scaling safety data up to the large organizations is no easier than scaling down to smaller organizations. The end game is the same for everyone. Understanding and managing your hazards has the promise on the inseparable trio of aircraft/flight safety, worker safety, and profit.
(Final Report Title is: The Transition from Event Reports to measurable Organizational Impact: Workshop Proceedings Report will be available at www.hfskyway.FAA.gov in late November.
Dr. Bill Johnson is the FAA Chief Scientific and Technical Advisor for Human Factors in Aircraft Maintenance Systems. Johnson is a member of the Human Factors Advisory Group to the European Aviation Safety Agency (EASA).
