With the full set of statistics about May now released, people have been doing some analysis on whether the tarmac delay rule that took effect on April 29 had an impact on cancellations. The DOT is happy to promote that the number of 3 hours+ tarmac delays in May was 5, a major reduction from the 34 on 2009. So clearly it worked, right?
No so fast, points out Brett Snyder in the Cranky Flier column. Cancellations are also way up, so that means many people people were inconvenienced to reduce the tarmac delays. So clearly it didn't work, right? [more]It's not easy comparing delay statistics month over month or year over year. Delays often start with weather, so how can you normalize for that? It's not fair to say that an increase in cancellations has anything to do with the tarmac rule. Just because airlines have threatened in the media to cancel flights more often doesn't mean they're actually doing it. It also doesn't mean that it's the wrong thing to do!I took the analysis one step further to try and normalize the data. It turns out that May 2010 was a pretty bad month for on-time flying anyways, and May 2009 was a great month for flying. So any comparison between the two is inherently flawed (read: completely meaningless). Delays, cancellations, tarmac events etc. are all a function of the conditions at the time -- drawing conclusions from a direct comparison is like saying the 1998 Yankees were better than the 2010 Yankees based on their record, or even pitching and batting stats. It's speculative at best because their stats are entirely based on how they performed against other teams and players. So it's fun to discuss, but it's a meaningless analytical comparison (and sportswriters often rely on their detailed qualitative knowledge of players to make such comparisons, not pure statistics).Now, what if you did a really detailed analysis that compared the two teams taking into account the batting and pitching stats of every other player they faced those years? While not conclusive per se, it at least would provide a sound analytical comparison. For example, was Jeter better in 1998 or 2010? A direct batting average comparison would be helpful and interesting, but not super meaningful except given the assumption that over the course of the seasons the quality of pitching he faced was equivalent in the comparison years. But what if you normalized his batting average to the stats of each individual pitcher he faced in both seasons? Now we're talking.I had a few ideas of how to do that for the tarmac delay analysis. I normalized the number of cancellations to other delay-related stats -- Namely the number of diversions, number of delayed flights, avg. number of delay minutes per delay, and avg. number of delay minutes overall. That is to say that diversions, delays, delay minutes, and cancellations are all products of weather et al. We can assume that together they represent a view of how "good or bad" any given month is. Now in order to tell if a rule had an affect on cancellations, we can see if the ratio of cancellations to any of those figures changed. In other words -- if more flights were cancelled in 2010 given similar conditions, you'd expect the number of cancellations per diversion or per delay to go up. That would show the trade-off between canceling a flight and allowing it to be delayed. (Obviously there are many other pieces of analysis that can be done, but I figured given ease of data manipulation, this was the best place to start). Out of those ratios, I chose "number of cancellations per arrival delay" as the key one. It seems to be the most relevant, as diversions are not numerous enough to show good statistics and using delay minutes introduces other areas of noise. It isn't perfect, but it's a good place to start.Now let's look at the results.In summary: Nationwide, there seems to be small impact on cancellations. The high number of cancellations for May 2010 can be explained mostly from just being a bad month. There is little to suggest that it is a result of the tarmac rule except the comments we hear from airlines (note I've heard from some airlines internally that they are indeed being super-conservative as a result of the rule, so it is not just a media party line). But if we drill down, what sort of "negligible" impact does it take to affect travelers?The figures: Overall, cancellations per arrival delay was 0.064 in May 2010, as compared to a 7-year average of 0.059 for May. 2010 represents the 2nd highest in that period, the highest was 2004 with a whopping 0.082.(As an aside, cancellations per diversion was 4.17 in May 2010, below the 7-year average of 5.20, and higher than only the 2009 figures. Similarly, cancellations per arrival delay minute was 96.8 in May 2010, down from a the 7-year average of 105.2, and higher only than the 2009 figures)So perhaps the ratio indicates a little something is going on, but it's not all that convincing.So what's up? Is this whole thing overblown?Well, if you dive into a handful of the big problem airports, you can see a bit of an up-tick.Overall, at the 35 top airports, May 2010's cancellation to arrival delay ratio was above average for 19 airports, and below for 16 of them. It was the highest over the last 7-years at 5 of them: DEN, FLL, IAD, JFK, MIA, PDX, and TPA.Does that mean anything? Hard to say. Only at JFK was the number really out of whack: 3x the next highest. That signifies a definite behavior change but might be the result of the runway construction, not the tarmac delay rule (although in April 2010, JFK's ratio was right in mid-range, suggesting something did change from April to May). Otherwise, only at TPA was 2010 a real outlier, with a ratio more than 2x the average, caused mostly but a sudden reduction in delayed flights, not an up-tick in cancellations (only 34). ATL would also fall into this category if it wasn't for 2005, where they had a seemingly ridiculous ratio of 0.222. Note by far the worst performing airport according to this metric is ORD, with an average of 0.234 -- More than 2x the overall average. Their peak in May was 2004 with a ratio of 0.355. That means they cancelled an obscene number of flights relative to delays experienced (note these airport-specific figures cannot be compared to overall figures since both arrival and departure cancellations are included for airport-specific analysis, whereas nationally cancellations are only counted once).Are these differences any more than natural fluctuation? It's hard to say, frankly. But there is no discernible pattern except at JFK, with the aforementioned runway problem. Note that if the ratio in May 2010 was the 7-year average, that would have meant 442 fewer flight cancellations in May nationwide -- 130 of those can be directly attributed to JFK. How many tarmac delays did we avoid from those cancellations? We don't know, but almost certainly fewer than 130, let alone 442.All these figures can change as airlines adjust their strategies for dealing with potential tarmac delays, but it just might be possible that simply by creating better plans and paying attention to it a whole lot more, airlines can avoid tarmac delays without canceling many more flights.Yes, you can argue there was a slight up-tick in May 2010, but certainly not the mass-cancellation scenario some airlines predicted.I still don't believe in DOT micromanagement of airlines and rules such as this, but if this conclusion is even half true, it certainly shows you not only the power of DOT intervention, but also the lack of attention that airlines truly paid to tarmac delays in the past. Their year after year claim of doing whatever they can to prevent them ("our hands are tied" "it's out of our control" etc. etc.) might have been perhaps a bit overstated. Never underestimate the power of good old enforcement to drive results. It's a sad lesson, but one hopefully the airlines learn sooner rather than later so they can proactively attack DOT concerns without having a ton of other rules pushed down their throats.
Evan Konwiser is co-founder of FlightCaster. These thoughts are excerpted with permission from his blog.