Strange Nall Stats

[The-Flying-Lawyer]

Original post (on Twitter) here, but I would guess a lot of you may not see it there and am curious what you think.

The final version of the 2018 Nall Report is out (here). One particular set of stats really caught my eye:

These are isolated from the pilot-related causes, so there shouldn't be any obvious ADM accidents (VFR into IMC, failure to account for density altitude on takeoff, etc.) in these data. What surprised me was the apparent correlation between more advanced training & experience (with certs as a proxy for those) and lethality. Why is that?

The "they're flying more complicated aircraft on more complicated missions" explanation doesn't seem to hold water. First, none of the fatalities involved IMC. Second, the vast majority of these accidents (117 out of 186) and roughly half of the fatalities (5 of 11) were engine-outs. You'd think an engine-out would be at least as bad for more frequently low & slow Private/Sport/Student pilots as for more frequently high & fast ATP/Commercial pilots. Third, there were zero mechanical-failure fatalities in the commercial fixed wing data, which you would think would be disproportionately high, fast, etc. and is exclusively ATP/Commercial pilots. That lines up with the fact that there were zero fatalities related to the 39 gear malfunctions on the noncommercial side.

Other observations: the CFIs on board do not appear to have been giving instruction to Private/Sport/Student pilots at the time of the accident in at least 2 out of their 5 fatalities. The Embry-Riddle Arrow breakup doesn't explain this overall gap: that was an ATP and a Private pilot. There was only one other fatality from an airframe failure (3 total).

So...what gives? Are more Private/Sport/Student pilots flatlanders and more ATP/Commercial pilots mountaineers? Are CAPS deployments disproportionately represented among the Private and Sport pilots? What am I missing?

 

[midwestpa24]

I think statistically speaking, the sample pool for these accidents is so small that the percentages are meaningless. Even a change of one accident would widely skew the percentages. I wouldn't read too much into it.

 

[wanttaja]

I think I'm voting for the "More complicated aircraft on more complicated missions" explanation. More complicated aircraft are going to be flying faster, with the opportunity for more energy to be liberated on impact. A J-3 usually hits a lot slower than a Bonanza.

In addition, my homebuilt data seems to show that high wing aircraft provide better occupant protection than low-winged planes. Those lower-time pilots are likely to be flying Cessna 172s and 182s.

I did a VERY quick comparison using two subsets of my homebuilt accident database... using two subsets I'd already split one. One contained pilots with 20,000 or more hours (140 total), the other had pilots having between 40 and 100 hours (147 total). The high-time group were all Commercial and higher, while the low-time group was a mix of Private, Sport, and Student pilots.

The high-time group had a lower fatality rate, 18.6% vs. 25%. Fifteen of the fatal accidents involving the low-time group were due to pilot miscontrol, vs. only five of the high-time group.

But when I looked at ONLY accidents involving mechanical failure, the fatality rate was almost identical, about 22%. It's a small sample size (~6-8 fatal accidents out of ~30 total)

I think the Nall results make sense, if you consider the difference being mission and aircraft. Almost all of the accidents in my two groups occurred during recreational flying, and scanning, through the aircraft types, I don't see a preponderance of high-performance aircraft on either side. So my results come out about equal for the two sides.

Ron Wanttaja

 

[The-Flying-Lawyer]

I think statistically speaking, the sample pool for these accidents is so small that the percentages are meaningless. Even a change of one accident would widely skew the percentages. I wouldn't read too much into it.

Maybe. It'd be interesting to cobble the stats together over time and find out. I flipped through the last ten years or so and as between ATP, Commercial, and Private they seem to have flipped. From 2009 to 2013, the Privates were usually highest on lethality. After 2014 (including preliminary 2019 data), it's always been Commercial or ATP.

I think I'm voting for the "More complicated aircraft on more complicated missions" explanation. More complicated aircraft are going to be flying faster, with the opportunity for more energy to be liberated on impact. A J-3 usually hits a lot slower than a Bonanza.

I think this, in conjunction with BRS (which does the same thing), seems plausible.

 

[airdale]

The first thing you need to know is the total number of hours flown during the period by rating. Then, maybe, complexity of equipment and missions. Then, at this point, you need a statistically significant number of samples. To calculate percentages to a tenth when working with single- and two-digit numbers of samples is pure innumeracy.

There was a totally bogus book a few years ago called "The Killing Zone" where the author totally ignored number of hours flown in a period and went on to get excited about an apparent correlation between total logbook hours and accident rates. As one reviewer said: "This book is trying to be scientific without understanding basic statistics. The basic premise of the book, that low-hour pilots are in "the killing zone", a time of high danger, is just plain wrong. There are simply more low-hour pilots than high-hour pilots, which leads to more low-hour accidents than high-hour accidents."

 

[The-Flying-Lawyer]

The first thing you need to know is the total number of hours flown during the period by rating. Then, maybe, complexity of equipment and missions. Then, at this point, you need a statistically significant number of samples. To calculate percentages to a tenth when working with single- and two-digit numbers of samples is pure innumeracy.

There was a totally bogus book a few years ago called "The Killing Zone" where the author totally ignored number of hours flown in a period and went on to get excited about an apparent correlation between total logbook hours and accident rates. As one reviewer said: "This book is trying to be scientific without understanding basic statistics. The basic premise of the book, that low-hour pilots are in "the killing zone", a time of high danger, is just plain wrong. There are simply more low-hour pilots than high-hour pilots, which leads to more low-hour accidents than high-hour accidents."

Totally understood on the statistical significance, but I'm not talking about comparing the quantity of accidents or the rate at which they happen (per hours flown). I'm saying the fact that the lethality % was quite a bit higher for pilots with more training rather than less when those pilots experienced roughly the same accident (engine-out in VMC, for the most part) strikes me as odd. In simple terms, ATPS experienced less than half as many of these accidents as private pilots, yet experienced close twice as many fatalities in those incidents. I'm not arguing that's statistically significant: I just find it "strange."

 

[Shepherd]

Lies, Damn Lies, and Statistics.

 

[dmspilot]

Totally understood on the statistical significance, but I'm not talking about comparing the quantity of accidents or the rate at which they happen (per hours flown). I'm saying the fact that the lethality % was quite a bit higher for pilots with more training rather than less when those pilots experienced roughly the same accident (engine-out in VMC, for the most part) strikes me as odd. In simple terms, ATPS experienced less than half as many of these accidents as private pilots, yet experienced close twice as many fatalities in those incidents. I'm not arguing that's statistically significant: I just find it "strange."

If it's not statistically significant then it's not strange. It's meaningless.

 

[Domenick]

Where's the typical PDF Nall report like I have from 2005-2017 (17th-28th)?

 

[Dana]

I think I'm voting for the "More complicated aircraft on more complicated missions" explanation. More complicated aircraft are going to be flying faster, with the opportunity for more energy to be liberated on impact. A J-3 usually hits a lot slower than a Bonanza.

One often repeated "fact" is that the most dangerous time is a pilot's second hundred hours... the idea that before that, the training is still fresh and the pilot knows he's inexperienced, and at about 100 hours it starts to get routine and he gets cocky, by 200 hours he's seen some of the things that can happen so he's more aware and thus safer. But I don't know if the statistics bear that out... any idea, Ron?

 

[wanttaja]

One often repeated "fact" is that the most dangerous time is a pilot's second hundred hours... the idea that before that, the training is still fresh and the pilot knows he's inexperienced, and at about 100 hours it starts to get routine and he gets cocky, by 200 hours he's seen some of the things that can happen so he's more aware and thus safer. But I don't know if the statistics bear that out... any idea, Ron?

Well, A) I don't have any statistics as to the number of (non-crashing) pilots by the number of total hours, and B) My main data is homebuilt accidents, and thus are probably different from run-of-the mill pilot.

But hey, lack of data hasn't stopped me before.

I took the pilot total time column on my homebuilt accident database (1998-2018), divided it into 25-hour "bins", and counted how many accident pilots fell in a particular bin. Here's the result:

The "spikiness" is probably due to NTSB analysts rounding the number of hours (490 hours? We'll list it as 500). If we use 100-hour bins instead of 25-hour, and run it all the way to 4,000:


It tends to corroborate the story, but we don't know how many pilots, total, factor into that 200-300 hour range. Might just be the most common total time, and the spike would then just be reflecting that.

Ron Wanttaja

 

[PeterNSteinmetz]

It tends to corroborate the story, but we don't know how many pilots, total, factor into that 200-300 hour range. Might just be the most common total time, and the spike would then just be reflecting that.

Yes, might well be that. Seems like a lot of people fly about that much total. I wonder if data on that aren’t available somewhere?