Reported impressive reduction in accidents due to ADS-B In

[wanttaja]

Lies, Dam* Lies and Statistics. Mark Twain.

Correlation does not verify Causation. Me (Engineer, not a Mathematician)

“Statistics are like bikinis. What they reveal is interesting. What they conceal is vital!” (Professor Aaron Levenstein)

“The purpose of analysis is insight, not bull****” (T.C. Weston, The Boeing Company, 1974. My boss, in fact....it was on a sampler on his office wall).

In my homebuilt accident analysis, I see a regular subset of "non-traditional" midairs...airplanes swapping paint while flying formation, airplanes colliding on the runway, etc....but it sounds like ADS-B aircraft weren't involved in any of these, either. There were also cases where ADS-B may not have been much advantage; sudden turns into collisions, etc. It's possible they were not included in the methodology, but, again, that leads to questions about how they chose the cases to include.

My ADSB-OUT install was only $1400 with the rebate (skybeacon) and ADSB-IN I had the year before for $130 with a Stratux. That is a lot less than $3k-$7k. Even before the ADSB-OUT mandate, the Stratux gave great weather, traffic, and GPS navigation.

For my aircraft, the cheapest approach I've found will cost about a quarter of the value of the airplane. For something that's very difficult for me to take advantage of (no spare space for a display, open cockpit in direct sunlight, no intercom to tie audible alerts to).

One thing that may be overlooked is that ADSB IN devices are providing the impetus for many VFR planes and pilots to get GPS navigation. While not necessary, Having a “fool proof” navigation source certainly results in fewer people getting lost, overflying their airport and getting low on fuel, or otherwise task saturated.

I was told that ADS-B is necessary because of GPS. In the old days, people would navigate from Town A to Town B, at some altitude within the range of the hemispheric rule with (maybe) an effort to keep it within the 100 ft Private flight exam limits. With autopilot-coupled GPS, pilots fly between precise geographic coordinates at precise integer altitudes.

There are obviously cases where this doesn't apply, such as collisions in the pattern.

Finally, it's difficult to quantitize the accidents where ADS-B is an actual contributor. Friend of mine (a CFI) went flying with another friend who had a brand new ADS-B in system installed in his Cessna. My CFI buddy had to keep pushing the yoke forward because the pilot would ignore the airplane while fiddling with the electronics.

Ron Wanttaja

 

[PeterNSteinmetz]

Yet that's exactly what you're doing — ascribing meaning to a non-statistically significant correlation (which is an oxymoron, there is no such thing) between ADS-B and mid-airs.

I beg to differ. What I am saying is that the best estimates of the rates are 0.3 and 0.0 and that may be interesting. I believe I have always been careful to note that this difference is not statistically significant. The observation that the rates are different does have some meaning. Indeed, estimation theory tells us the means of the observations are our best estimates of the actual values of the rates (by actual, I mean what is really true in reality and presumably reflected in a very large sample). I revised my prior post while you were responding -- it may be informative to review what I say about power of a test there. That is really the way to get to the bottom of this.

Now, where you might be confused is that you think I'm trying to say that ADS-B has no effect on mid-air collisions in general, or that it won't several years from now. But I'm not saying that. I'm talking about the data in THIS study, the one you started the thread about. The data show no effect.

I agree there is definitely some confusion. It may arise from exchanges like the following -

PeterNSteinmetz said: ↑
The mechanism for ADS-B reducing mid-air accidents seems more plausible than some of the other categories.

DmsPilot said:
There is no statistically significant difference in the rate of mid-airs between ADS-B equipped and non-equipped aircraft. You can't infer or speculate from the data that ADS-B caused a reduction in mid-airs when there wasn't a reduction.

You see, there was an observed reduction and it was shown in the tables 2 and 3 (apparently previously ignored). Yet this post says "when there wasn't a reduction". Confusing. And instead of assuming the other person is "wrong" it might be best to try and clarify the confusion.

It is likely useful in reducing confusion here to review what I had revised into my prior post regarding statistical power. If one is really interested in ascertaining how to best interpret the failure to obtain a statistically significant result in this test (there are two possible explanations), versus say just having an argument, probably best to go and determine what the number of operations they were using as denominators and compute the power of the test. That would be my suggestion.

 

[Capt. Geoffrey Thorpe]

You see, there was an observed reduction and it was shown in the tables 2 and 3 (apparently previously ignored). Yet this post says "when there wasn't a reduction". Confusing. And instead of assuming the other person is "wrong" it might be best to try and clarify the confusion.

Number in column C may be less than the number in column B. At first glance it would appear that C is less than B. But because of the normal variation one needs to apply statistics to determine if we have some confidence that the results for column C are, in fact, real and not just the result of random variation. When it is determined that the difference between B and C is not significant, it means that it's possible that the "real" underlying average for "C" could actually be equal or higher than the "real" average value for B.

Now, it could be that ADS-B would result in fewer mid air events. And it might make sense that it is a likely outcome. But, given the available data there is no way to demonstrate that.

You flip a coin 6 times. You get 4 heads and 2 tails. Does that mean that coins are more likely to come up heads?

 

[dmspilot]

I beg to differ. What I am saying is that the best estimates of the rates are 0.3 and 0.0 and that may be interesting.
...
You see, there was an observed reduction and it was shown in the tables 2 and 3 (apparently previously ignored). Yet this post says "when there wasn't a reduction". Confusing. And instead of assuming the other person is "wrong" it might be best to try and clarify the confusion.

I did not ignore the table, I explained with an example a 10-year old could understand why it is statistically meaningless. You want to ascribe meaning to a meaningless statistic, or insist that it is "interesting", I can't help that. That's why we have statistical analysis in the first place. So we don't accidentally ascribe meaning to data that aren't meaningful.

 

[Palmpilot]

...the 100 ft Private flight exam limits....

I thought it was 200 feet for private, with 100 feet being for the instrument rating. Or have they changed that since I took my checkrides?

 

[Palmpilot]

...Indeed, estimation theory tells us the means of the observations are our best estimates of the actual values of the rates (by actual, I mean what is really true in reality and presumably reflected in a very large sample)...

What would be a valid use for a non-statistically-significant best estimate?

 

[PeterNSteinmetz]

I did not ignore the table, I explained with an example a 10-year old could understand why it is statistically meaningless.

Glad to hear it was not ignored, but that was confusing. Perhaps that is not surprising because it turns out the proper interpretation of the failure to obtain a statistically significant result was the subject of a long debate between professional statisticians, mathematicians, and scientists. Sort of resolved into several schools of thought.

I think it would be fair to say that the recognition that one is dealing with two types of probability in any statistical test -- the significance level, that is the likelihood that one is obtaining the results by chance because there is no real difference -- and the power of test, that is the likelihood of finding a significant result if the difference is true -- are both important aspects when considering the meaning of the data and the results of the tests.

So my point is that one should pay attention to both the significance level and the power when considering this result.

It is not always easy to know the answers in a particular case because the application of statistics can be subtle. For example, there is likely an error in the interpretation of two successive statistical tests in the field of single neuron recording in neuroscience which has affected a large part of the publications in the field by professional scientists. See for example Steinmetz PN, Thorp CK (2013) Testing for effects of different stimuli on neuronal firing relative to background activity.Journal of Neural Engineering, 10: 056019.

 

[PeterNSteinmetz]

What would be a valid use for a non-statistically-significant best estimate?

Good question. For one, it can suggest areas of potentially fruitful additional data gathering. It also can be the basis of a hunch about a scientific hypothesis to be further investigated.

Or in this case, it can suggest an area where it would be useful to compute the power of the test in order to better interpret the results and decide whether it is more likely that a test failed to be significant because the test lacked power or because there is no real effect.

Finally, if you are in a betting game, bet on the mean expectation of outcomes, even if you don't have enough data to know for sure (like the rest of the cards in the pack).

 

[PeterNSteinmetz]

Number in column C may be less than the number in column B. At first glance it would appear that C is less than B. But because of the normal variation one needs to apply statistics to determine if we have some confidence that the results for column C are, in fact, real and not just the result of random variation. When it is determined that the difference between B and C is not significant, it means that it's possible that the "real" underlying average for "C" could actually be equal or higher than the "real" average value for B.

Now, it could be that ADS-B would result in fewer mid air events. And it might make sense that it is a likely outcome. But, given the available data there is no way to demonstrate that.

You flip a coin 6 times. You get 4 heads and 2 tails. Does that mean that coins are more likely to come up heads?

Nice explanation. As I explained in my other post above, for ADS-B it would be useful to determine the other important statistical aspect of the test, its power to detect a difference. That would let us better judge whether there is more likely to be no real effect or if we just don't have enough observations.

And yes, if betting on the coin, the best best would be that it is unfair, but not by much. The best estimate of the probability of getting a heads on that coin is 4/6, but it is not statistically significantly different from 1/1, so I wouldn't put too much money on it!

 

[PeterNSteinmetz]

You want to ascribe meaning to a meaningless statistic, or insist that it is "interesting", I can't help that. That's why we have statistical analysis in the first place. So we don't accidentally ascribe meaning to data that aren't meaningful.

Why this strange insistence on disagreeing with what I have written?

The differences in the estimates of the rates of mid-airs are real differences in the observations. They have meaning if you were forced to bet on what the rates are, but that difference does not reach the level of statistical significance. The differences in the observed rates are not a "meaningless statistic". That is an over-simplification.

If one really wants to understand how to best interpret this test, rather than paying attention only to the significance level, pay attention to both sides of its performance. What is the power of the test to detect a difference? That can be determined by doing some research and making a computation and that will allow a better level of discussion.

Here is a reference about the power of a test for those who may be interested (versus just arguing endlessly). It explains the factors that affect the power of a test and how that is one aspect of a test's performance. https://stattrek.com/hypothesis-test/power-of-test.aspx

If someone wants to look up what the number of operations likely being used in the denominators of the ratios are (in the reference the report gave), I am happy to help compute the power of a binomial test to detect a difference in the rates.

 

[wanttaja]

I thought it was 200 feet for private, with 100 feet being for the instrument rating. Or have they changed that since I took my checkrides?

It's been nearly 50 years, but I thought it was 100 feet, plus or minus.

Ron Wanttaja

 

[dmspilot]

Why this strange insistence on disagreeing with what I have written?

I've said what I had to say, no point in going around in circles.

If one really wants to understand how to best interpret this test, rather than paying attention only to the significance level, pay attention to both sides of its performance. What is the power of the test to detect a difference? That can be determined by doing some research and making a computation and that will allow a better level of discussion.

I think the fact that there were zero ADS-B midairs, and that there's still not a statistically significant difference from non-ADS-B midairs, shows that the test is not powerful enough.

 

[PaulS]

It's been nearly 50 years, but I thought it was 100 feet, plus or minus.

Ron Wanttaja

ACS standard is that it depends, see dmspilot post below.


Edited.

 

[EdFred]

ADSB as accident prevention is like dropping someone off in the middle of the Serengeti at night and giving them bug spray. Yeah, you might die from malaria, and DEET will help prevent getting bit by mosquitos, but chances are youre going to get eaten by a large cat. You're better off protecting yourself from the cats, but DEET is whats being mandated and touted as saving lives.

 

[dmspilot]

ACS standard +/- 200 feet for ppl test

Not really.

It's 100 feet except for the cross-country and hood work tasks which are 200 feet.

 

[PeterNSteinmetz]

I think the fact that there were zero ADS-B midairs, and that there's still not a statistically significant difference from non-ADS-B midairs, shows that the test is not powerful enough.

My own intuition agrees that this may well be a case of low power due to a very low rate of mid-airs. But I don't feel that confident about it. There are likely a very large number of operations and so the denominator and number of observations would be quite large. It is possible that there is sufficient power in the test to detect any meaningful difference between the rates which would then argue that there is no real difference. Close call and I would have to actually see the numbers to have a better opinion. I'll run the power calculation if you will find the denominators

 

[PaulS]

I've said what I had to say, no point in going around in circles.



I think the fact that there were zero ADS-B midairs, and that there's still not a statistically significant difference from non-ADS-B midairs, shows that the test is not powerful enough.

You know you guys are going round and round about whether or not ads - b lessened the chance of a mid air, which is inconclusive and since the number of mid airs is relatively small it may never be significant.

Most of the ATC system, which includes airspace, controllers, procedures, training and what not, center around preventing mid airs. We have pilots who think a mid air is a remote possibility due to big sky theory, in other words, there is so much space up there the chance of colliding is remote. I maintain that the big sky theory is wrong and the good record we have is due to how the system is designed plus the vigilance of most pilots.

Many people are concerned that all this extra information in the cockpit will cause pilots to be looking at their screens rather than outside for traffic, predicting more accidents or at least intimating this technology is bad for safety. I think this study debunks those thoughts. There has been no statistically significant change in mid air data, which means this technology has not had the adverse impact some worried about.

 

[PaulS]

Not really.

It's 100 feet except for the cross-country and hood work tasks which are 200 feet.

I stand corrected, that's what I get for doing a quick glance. I'll change my post so people don't get bad info.

 

[ElPaso Pilot]

ADSB as accident prevention is like dropping someone off in the middle of the Serengeti at night and giving them bug spray. Yeah, you might die from malaria, and DEET will help prevent getting bit by mosquitos, but chances are youre going to get eaten by a large cat. You're better off protecting yourself from the cats, but DEET is whats being mandated and touted as saving lives.

Naw, DEET’s far cheaper than ADSB.

How about printer ink. That’s $4731 per gallon.

Would a gallon or two kill mosquitoes too??

 

[Salty]

I stand corrected, that's what I get for doing a quick glance. I'll change my post so people don't get bad info.

I think what’s confusing is it’s 200 feet total, but only 100 feet above or 100 feet below.

 

[PaulS]

I think what’s confusing is it’s 200 feet total, but only 100 feet above or 100 feet below.

I flipped to the basic instrument part and it's +/- 200 ft.

 

[Matthew Rogers]

True, but the stratus/stratux is

What did the Avionics shop charge to install and setup the skybeacon?
.

I installed the three screws and two wire splices and it cost $100 for my IA to fill out the 337. That was the entirety of the installation.

 

[kyleb]

True, but the stratus/stratux is

I installed the three screws and two wire splices and it cost $100 for my IA to fill out the 337. That was the entirety of the installation.

Did the transmon work right out of the box with no sensitivity adjustment?

 

[Matthew Rogers]

Did the transmon work right out of the box with no sensitivity adjustment?

Yes, and it passed the certification flight without problems. That flight report had zero errors, not just a pass with acceptable results. So as far as the FAA is concerned it is working perfectly.

 

[WannFly]

Wow I had no idea so many pilots do not like additional situational awareness.

 

[hindsight2020]

Wow I had no idea so many pilots do not like additional situational awareness.

Not by involuntary mandate. Let's not conflate topics here.....

 

[WannFly]

Not by involuntary mandate. Let's not conflate topics here.....

I have no idea what the topic is anymore ...lol

 

[PeterNSteinmetz]

I have no idea what the topic is anymore ...lol

I must say, I have been amused to observe the drift on threads I have started. Fun to see where things lead.

 

[RyanShort1]

What I want to know is will ADS-B save us from AOA or MCAS???

 

[EdFred]

Wow I had no idea so many pilots do not like additional situational awareness.

I like adsb weather. But don't try to sell me that it's a safety thing. It's not.

 

[Capt. Geoffrey Thorpe]

I like adsb weather. But don't try to sell me that it's a safety thing. It's not.

But without ADS-B you would never know about the hundreds of aircraft that you would not have collided with!

 

[FastEddieB]

I was told that ADS-B is necessary because of GPS.

And I was under the impression that GPS greatly reduced midair chances.

In the good old days of VOR navigation, a huge amount of VFR traffic utilized airways, or navigated direct from VOR to VOR. This created a huge “funnel effect” at VOR’s. I remember how bad it was over the Miami VOR in the 1970’s and 80’s, for instance.

Now, with so many folks going direct from one airport to another, the traffic flow is far more randomized. Of course, you still have to be alert at all times, but I think GPS has markedly reduced localized congestion of VFR traffic, making midairs less likely.

Fascinating thread, BTW.

 

[EdFred]

But without ADS-B you would never know about the hundreds of aircraft that you would not have collided with!

And still won't know about the ones that will never have a xponder.

 

[Let'sgoflying!]

This thread boils down to the same old argument which I have seen going on since I started flying in 1976.

Will attempts to make flying absolutely safe cost so much that it will price everyone out of flying?

Or put another way;

Zero accidents can only achieved by zero flying - and we are getting there by making rules and regulations which make flying so expensive no one flies.

I can see both sides of the safety vs freedom argument.

However, people are never going to agree where we should be on this continuum.

No rules, no safety boxes, no requirements; More accidents/injuries/deaths.
Maximal regulation of flying; Less people are able to enjoy it.

Every time a new multi-thousand dollar requirement appears, remember; that trims off yet another X% of the bottom of pilots/owners who are able to afford and enjoy this avocation.

 

[steviedeviant]

I started flying later in life and knew nothing about the technology of airplanes prior to my intro flight a few years ago. I was shocked at how “old” the dials and technology were. I grew up on computers and screen and it was just strange. As I learned, I got a better understanding of it all. Then I moved up to glass, including ADS-B, and I will never go back. I understand it is expensive and I would give up glass, but not the ability to see others with ADS-B. Unlike a car, one hit and you are dead. It isn’t that expensive and it has already helped out in several occasions when ATC even didn’t notify until late. I am surprised by the number who seem to not appreciate it...guessing the same thing was happening when the government issued seat belt laws. Everyone wants to be a rebel.

 

[kyleb]

I am surprised by the number who seem to not appreciate it...guessing the same thing was happening when the government issued seat belt laws. Everyone wants to be a rebel.

Tens of thousands of people are killed in auto collisions a year. Many more are injured. Seatbelts cost maybe a couple of hundred dollars to install in a vehicle. The cost/benefit ratio of first order safety enhancements such as seatbelts is pretty good. We have them in airplanes too. ;-)

Midair collisions are amazingly rare. Yet, there is a multi-thousand dollar mandate to install ADSB.

The cost/benefit is out of whack on ADSB.

ADS-B wasn't sold as anti-collision for us minions. It was sold as airspace control to allow stacking more airliners into the same airspace. It isn't going to protect the minions from midairs very well because a large portion of the fleet won't equip. It will protect the airliners because the unequipped minions will be largely precluded from entering "airliner" airspace.

 

[Matthew Rogers]

I am happy to be protected by having ADSB OUT because my home airports are very close to class D Stewart (SWF) and airliners fly into there from time to time. The practice areas (not defined specifically on any chart) are right in the flight paths of those and possibly any of the New York traffic as well depending on altitude and routing. With the IN side, I can see their arrivals from far away and with the OUT, they can see me if I happen to stray into an approach path in use (yes they have TCAS and ATC has radar, but I sure as hell know that it must be hard to spot a little C150 from an airliner).

I want to be able to fly the Hudson River corridor and it sure would be nice to get an idea of how many others are flying at that time and get some sort of conflict reduction using ADSB in that tight bit of crowded airspace.

Again, it only cost $1400 with the rebate for my cheap C150. And that got split 3 ways. So for everyone complaining about cost, I am one of those who could be “priced out of aviation” because I can’t afford it. With only $40k to go around and a family, I am not in a position to even part own a plane, but it can be done cheaply (and safely with a good AP).

If you own a plane, I am sure that you can handle to cost of equipping, at least with skybeacon or gdl-82. If not, sell your plane and go back to renting, or put up with flying in the more remote airspace. Or share it with another owner or two. Three owners now means that the plane flies every week year round and we no longer have to worry about cold starts with no oil film, old oil before it gets changed, repairs and expenses get split 3 ways, so they are taken care of quickly. Better to wear the plane out with use rather than die from corrosion.

 

[Palmpilot]

I suspect that the availability of rental planes in ADS-B required areas will drop somewhat when the deadline arrives.

 

[Warmi]

And I was under the impression that GPS greatly reduced midair chances.

In the good old days of VOR navigation, a huge amount of VFR traffic utilized airways, or navigated direct from VOR to VOR. This created a huge “funnel effect” at VOR’s. I remember how bad it was over the Miami VOR in the 1970’s and 80’s, for instance.

Now, with so many folks going direct from one airport to another, the traffic flow is far more randomized. Of course, you still have to be alert at all times, but I think GPS has markedly reduced localized congestion of VFR traffic, making midairs less likely.

Fascinating thread, BTW.

It is more randomized than VOR but if you have two airplanes going direct from airport A to B and vice versa using their respective magenta lines , the GPS will still take them fairly close to each other ...

 

[X3 Skier]

After I discovered Ultralights, I bought one to fool around with in the local area. Totally impractical but essentially no real rules at all. If you just want to be airborne on nice days and thumb your nose at the FAA, get one and forget about avionics and related mandates.

Rentals for any cross country and my Aerolite 103 for cheap fun.

Cheers