Measuring and Quantifying Airline Award Redemption: Not all Programs are Created Equal

Scott McCartney covers the IdeaWorks ‘study’ of award availability (.pdf), it points to some interesting results though the methodology of the underlying research is somewhat suspect. And some of the results are so strange that it calls for what’s pretty commonplace in science — the notion that a single study is rarely meaningful, it’s only when results are replicable and pointing in the same direction that we can begin to reasonably draw conclusions.

Still, I certainly applaud efforts to systematically measure and quantify the performance of frequent flyer programs!

A study testing the availability of free seats showed that Southwest Airlines Co. could fulfill 99.3% of requests for award seats requiring standard mileage levels, and Alaska Air Group Inc.’s Alaska Airlines offered choices on 75% of requests. US Airways Group Inc. could fulfil just 10.7%. Delta Air Lines Inc. was among the stingiest, too, with awards requiring the lowest mileage available for only 12.9% of requests made by IdeaWorks Co., a consulting firm.

… IdeaWorks made 6,160 queries at 22 airline websites—280 seat requests at each airline. The firm tested long routes and shorter trips under 2,500 miles in big markets. At most airlines, it had greater success with shorter routes. The requests were made in February and March for travel in June through October. Among carriers outside the U.S., Air Canada, Deutsche Lufthansa AG and Singapore Airlines were among the most generous.

Now, read the study, but there’s no conceivable way that Southwest and Air Canada are meeting customer redemption needs over 90% of the time. Air Canada is actually relatively tight with award availability, though they’ll certainly make seats available for incrementally more points.

And while on the whole Southwest is actually pretty good, there certainly are redemption complaints from members. And don’t forget that this is a bit apples-to-apples. Even if Southwest really is the best, who cares? You’re still redeeming your points to Tulsa or Oakland or wherever. That’s not really the aspirational sort of awards that make these programs in general valuable in the first place.

Lufthansa is pretty good, as the report suggests. But Singapore near the top? No way. And American should certainly be near the top not the bottom. But the study is highly sensitive to route selection bias.

Furthermore, a real limitation of the survey is querying routes on the program websites, which vary widely in quality. And some programs offer partner award availability online while others do not.

Any time you see different results for award availability between Continental and US Airways you know it’s a booking interface issue rather than an availability issue, at least accounting for 95% of the variance.

Sure, web functionality factors into overall member performance but it’s not entirely fair to blame lack of award inventory on a program’s IT problems.

Indeed, Delta concedes its results were bad, precisely because the study involved searching for awards online — they finally admit their website is broken!

But bizarrely also claim that their poor showing is due to the Northwest merger and more miles from the combined programs chasing their seats. That makes no sense since Delta and Northwest were already frequent flyer redemption partners before the merger.

Meanwhile rather than fight back with data, US Airways concedes the results of the survey — which are completely inconsistent with my own pretty extensive experience. Since the beginning of 2010 US Airways has been excellent with domestic award inventory, including transcon routes like Charlotte and Philadelphia to Los Angeles and San francisco, and especially in first class.

Instead of pointing this out (perhaps they don’t even realize?), they concede that their award availability is poor because (1) they release seats closer to departure than the study was examining and (2) they offer the seats “at a different price point” (i.e. for more miles – hah!).

This old canard unforunately gets repeated:

To get seats to vacation destinations, you typically need to book 11 months in advance, when airlines open up flights for reservations.

Now, some seats are released on some flights on some airlines when schedules load into the system — with most carriers 331 days out. But many flights don’t get any award seats at that time!

Airlines want to make awards available for seats that will otherwise go unsold. A year out they don’t necessarily have as good an idea of how many seats that will be, and their knowledge gets better as travel approaches — at 6 months out, 3 months out, 2 weeks out, etc. Many carriers release seats all along the way as they expect seats to remain open, many are much more generous closer to travel than 11 months out.

That doesn’t seem you can’t get awards right away when a schedule loads, each carrier is a bit different in that regard, but the notion that you need to call immediately when the schedule opens and that when you’re unsuccessful it means someone else got to those seats before you did is just wrong.

Ultimately the study, no matter how much it may get wrong in its particulars, points to the idea that not all airline miles are created equal. That’s an important point indeed.

My own currency preferences are first to go for flexibility, you really can’t underestimate the value of American Express Membership Rewards points for instance. (Starwood points are actually more valuable for airline mileage transfers in most cases, but the fact that they take several days to a couple of weeks to transfer makes it a bit difficult maintaining that flexibility at redemption time — you can’t really put an award on hold and then trasnfer points from Starwood).

And for airline programs in particular, I value Air Canada Aeroplan, Continental, US Airways, and American in that order among North American programs, IdeaWorks testing notwithstanding.

About Gary Leff

Gary Leff is one of the foremost experts in the field of miles, points, and frequent business travel - a topic he has covered since 2002. Co-founder of frequent flyer community, emcee of the Freddie Awards, and named one of the "World's Top Travel Experts" by Conde' Nast Traveler (2010-Present) Gary has been a guest on most major news media, profiled in several top print publications, and published broadly on the topic of consumer loyalty. More About Gary »

More articles by Gary Leff »



  1. Maybe your experience has been different. A couple of months ago my gf was trying to fly from SAN to BOS for this coming weekend. USAir didn’t have any seat available (except at higher miles), but I noticed that United did, they showed up on both United and Continentals website. When she called USAir she was told yes United had seats available but it would cost more miles. She ended up booking United one way and American the other way since she had enough miles in her accounts for one ways. That is when I lost faith in USAir’s program and I was so glad when their merger with United fell through.

  2. Where exactly does DL admit their website is broken? That link just points back to your earlier piece! Sure, we all know that DL (and especially their award booking engine) suck, but I have a hard time believing they actually admitted it!

  3. The average non-flyertalk flyer is going to the program’s websites to check for redemption availability. They don’t know the tricks of the trade, and certainly won’t know how to find hidden availability. In that regard, I think it’s entirely fair to have IT problems be a factor in grading award availability.

    These airline miles programs survive partially because the public accumulates the miles based on a perceived ease of redeemability that is more generous than reality. If there is a grading system to make the general public more aware of availability differences, then perhaps it will force the airlines to shape up.

    I am rooting for this study, regardless of the personal opinions I may have of certain results.

  4. @hobo13, the McCartney piece says “says its low standing in the IdeaWorks survey resulted from problems the airline had combining reservation platforms between Delta and Northwest Airlines in the months the survey was conducted.” Now, that says the award availability was an IT issue rather than an inventory issue, though they sorta imply that the problem has been corrected when it hasn’t.

  5. Thanks for your detailed discussion of this topic. I’ve posted on flyertalk the obvious problems with the study (no way is Southwest availability at 99%, and US Airways’ miles are much more valuable than this study would indicate). But the flyertalk community is willing to overlook these mistakes because they’re thrilled that Delta — which truly is miserly — is getting bad press from it. That said, I’m not sure I prefer “junk science” to no science at all. When you tell me that award availability on Southwest is 99%, I’m going to tell you to redo the methodology of your study.

Comments are closed.