Risk Management or Mismanagement? A Closer Look at AETC's New Worksheet
AETC Mishap Worksheet Part 1 with Dr. Angela Karnes Padron
In the first episode of the "20 Years Done" podcast, I discussed the recently implemented risk management worksheet with Dr. Angela Karnes Padron, an Industrial-Organizational Psychologist. Our conversation focused on several critical concerns regarding the worksheet:
Broad Implementation: We questioned the effectiveness of rolling out such a significant change across an entire major command without incremental testing at individual bases, emphasizing the challenges of large-scale change management.
Feedback and Finalization: The article mentioned ongoing feedback, indicating the worksheet wasn't fully tuned or finalized, yet it was already in use. This raised concerns about its readiness and potential for premature implementation.
Workload and Survey Fatigue: Completing the worksheet could take up to a man-hour for a team of 6, adding significant workload. We highlighted the risk of survey fatigue, where repeated tasks might lead to rushed or dishonest responses, undermining the worksheet's purpose.
Misrepresentation Incentives: The worksheet included probing questions about health and alcohol consumption, which could lead to misrepresentation due to fear of disclosure to potentially hostile supervisors. This creates an environment where honesty is discouraged.
Inappropriate Adjudication Level: The worksheet's requirement for approval from production superintendents, who are incentivized for productivity rather than safety, was another major concern. This setup could lead to conflicts of interest, where safety is compromised for operational efficiency.
Lack of Science-Based Approach: We discussed how the worksheet didn't appear to be grounded in IO psychology principles, questioning its scientific basis and overall effectiveness.
Our conversation highlighted multiple issues with the worksheet's implementation, from broad rollout and survey fatigue to incentives for dishonesty and inappropriate adjudication levels. These factors combined suggest that the worksheet, as it stands, may not effectively address the rising maintenance mishaps within the Air Force.
AETC Mishap Worksheet Part 2 with Kurtis Ott
In the second episode of the "20 Years Done" podcast, I discussed the AETC Form 802 Aircraft Maintenance ORM Worksheet with retired Chief Master Sergeant Kurtis Ott. We conducted a mock scoring of the worksheet based on a realistic scenario and identified several issues:
Environmental Factors:
Difficult to reach high-risk scores requiring Production Superintendent approval.
Task Factors:
Vague definitions for task factors like "timeline for completion" and "hazardous materials/chemicals."
The lack of clear definitions for terms like "medium priority" and "high priority" was concerning.
Human Factors:
Likely to result in moderate to high-risk scores due to maintenance culture.
High potential for dishonest reporting to avoid higher approval levels or work, or conversely, easy to manipulate the scoring to avoid work.
Production Superintendent's Role:
Inappropriate level for adjudicating high-risk tasks due to conflict of interest (focus on productivity over safety).
Implementation Guidance:
We noted the absence of implementation guidance, which we hoped would define key terms and provide clarity.
Retention of Forms:
Forms not required to be kept after a shift, making it difficult to track factors contributing to mishaps occurring later.
We concluded the worksheet was deemed poorly constructed, either too sensitive or not sensitive enough to various factors, with significant issues in implementation and adjudication.
Implementation Guidance Concerns
The implementation guidance for the AETC Form 802 Aircraft Maintenance ORM Worksheet falls short in several critical areas. The most glaring issue is the absence of actionable definitions. Terms like "moderate," "severe," "serious," and "medium urgency" are used without any clear criteria, leaving personnel to interpret them subjectively. This lack of clarity can lead to inconsistent application and misunderstanding of risk levels.
Additionally, the list of tasks requiring the worksheet is vague. For instance, "flight control operational checks" and "engine operational checks" are listed without specifying which checks fall under these categories. Does a simple built-in system test require the same scrutiny as a full engine run with special certification? Similarly, "landing gear operational checks" are listed without distinguishing between different types of checks, such as a basic position and warning check versus a more complex task requiring lifting and hoisting.
This ambiguity extends to practical effects on the flight line. Without clear definitions and criteria, maintainers may either overestimate or underestimate the risks associated with their tasks. This can result in unnecessary administrative burdens or, conversely, overlooked hazards, both of which undermine the worksheet's effectiveness. Furthermore, the guidance does not address critical aspects like what constitutes a "lockout/tag-out" scenario, adding to the confusion.
Another significant issue with the list of operational checks is the apparent lack of a foundation in actual mishaps over the last few years. It is unclear whether these tasks are identified based on recent root cause analyses of maintenance mishaps. For instance, a recent fatality in Alaska did not involve any of the listed tasks. This raises questions about whether these tasks are truly high-risk or if the worksheet's requirements are based on assumptions rather than statistical evidence of mishap trends.
Kurtis Ott’s Additional Thoughts After Reviewing the Implementation Guidance
After reviewing the Maintenance ORM worksheet and reflecting on my initial perspective that it replaces lacking leadership, I still believe this is largely the issue. However, I withheld full judgment until reading the instructions for its use. After reviewing them, I find the instructions substantially lacking. Although bases can add local requirements, universal intent and guidance are necessary, and the instructions I read are inadequate by my standards. The creators have limited the worksheet’s use to tasks that already require specialized training or substantial leadership engagement. Frankly, this whole premise seems rushed, almost as if to fill a square, with a later statement saying, “we tried.”
Recently, AMC technicians at McConnell AFB “dropped” a KC-135 on its nose, in my experience such an incident is almost always due to maintenance malpractice and not following technical data. When I was in an A-10 unit, we had technicians fire a flare across the flightline during a routine check. Several pages of cautions, notes, and warnings were written to prevent this, starting with step one: ensuring chaff flare modules were removed. The technician(s) failed, assuming the modules were removed, but due to the lack of mod trailer availability from Ammo, the buckets had not been removed. Better leadership engagement might have prevented this action. The bottom line was a lack of adherence to technical data and proper training. The individual responsible was fully qualified and believed he was fully qualified, so this checklist would most likely not have made a difference in this particular incident.
To conclude, I have little confidence that this ORM worksheet will influence a shift in accidents. In my view, leadership engagement is lacking, and to meet the desires of technicians, leaders appear to compromise standards for happiness and retention. Instead of visible leadership, they are “driving” production from behind a desk.
What the USAF has done to leadership through technology and keeping them behind a desk is a discussion for another day.
Final Thoughts
Our deep dive into the AETC Form 802 Aircraft Maintenance ORM Worksheet and its implementation guidance reveals significant flaws and concerns. From our initial discussion with Dr. Angela Karnes Padron, we highlighted issues with the worksheet’s broad and premature implementation, excessive workload, and the risk of dishonest reporting. Kurtis Ott's insights in our second episode underscored the vague definitions and the inappropriate adjudication level required for the worksheet, emphasizing the lack of practical and actionable guidance.
In military aviation, the "knock it off" protocol allows any crew member to halt operations if an unexpected risk arises, prompting a collective reassessment of the situation. However, when this protocol was introduced to aircraft maintenance, it was met with apathy and sarcasm rather than respect. During my time in aircraft maintenance, I witnessed firsthand how "knock it off" was rarely, if ever, used effectively. It became a joke among maintainers, highlighting dangerous norms rather than addressing them. This cultural disdain for the "knock it off" protocol undermines any new risk management tools, including the ORM worksheet.
The failure to acknowledge the ineffective implementation of "knock it off" within the maintenance culture suggests a fundamental misunderstanding of the environment in which the worksheet is being introduced. If leadership believes that "knock it off" is a viable safety measure, they overlook the deep-seated cultural issues that hinder its effectiveness. This misalignment indicates that the ORM worksheet, like "knock it off," may fail to gain traction and deliver the intended safety improvements.
Additionally, the implementation guidance is unsigned, not on official letterhead, and not part of any formal regulation. It appears as a simple Word document, lacking the legal weight and authority to ensure compliance. This raises further questions about the seriousness and enforceability of the new procedures.
As we move forward, it is crucial for the Air Force to refine these tools, grounding them in both scientific principles and practical experience, to ensure they serve their intended purpose without adding unnecessary burdens or creating new risks.
AETC Public Affairs Official Statement on the Worksheet
The worksheet was developed by aircraft maintenance senior noncommissioned officers (active duty and retired) and coordinated with experienced maintainers on the Air Education and Training Command Safety Directorate staff before it was released in mid-February. It has since been updated based on feedback from maintainers across the command and will be updated more in the future as more feedback comes in.
The goal is to ensure that our aircraft maintainers, who put their lives on the line every day while performing mission-critical tasks, are equipped with a similar safety tool that pilots have been required to use for many years. This worksheet creates a standardized and repeatable method to help Airmen better capture and mitigate the total risk posed by human, task, and environmental factors that they encounter as they perform these tasks.
Culture change is always challenging when implementing any new policy, which is why AETC Safety developed this worksheet with current and former maintainers, and since implementation have personally travelled to most of the affected AETC wings to brief the intent and garner feedback directly. Furthermore, AETC is examining how to best support a healthy risk-management mindset; not only to instill within AETC’s current aircraft maintainers, but also permeate across every maintenance Airman who will eventually train, lead and operate outside of the First Command. More initiatives in the future may be necessary, but those have not been determined yet.
