Scott McCartney covers the IdeaWorks ‘study’ of award availability (.pdf), it points to some interesting results though the methodology of the underlying research is somewhat suspect. And some of the results are so strange that it calls for what’s pretty commonplace in science — the notion that a single study is rarely meaningful, it’s only when results are replicable and pointing in the same direction that we can begin to reasonably draw conclusions.

Still, I certainly applaud efforts to systematically measure and quantify the performance of frequent flyer programs!

A study testing the availability of free seats showed that Southwest Airlines Co. could fulfill 99.3% of requests for award seats requiring standard mileage levels, and Alaska Air Group Inc.’s Alaska Airlines offered choices on 75% of requests. US Airways Group Inc. could fulfil just 10.7%. Delta Air Lines Inc. was among the stingiest, too, with awards requiring the lowest mileage available for only 12.9% of requests made by IdeaWorks Co., a consulting firm.

… IdeaWorks made 6,160 queries at 22 airline websites—280 seat requests at each airline. The firm tested long routes and shorter trips under 2,500 miles in big markets. At most airlines, it had greater success with shorter routes. The requests were made in February and March for travel in June through October. Among carriers outside the U.S., Air Canada, Deutsche Lufthansa AG and Singapore Airlines were among the most generous.

Now, read the study, but there’s no conceivable way that Southwest and Air Canada are meeting customer redemption needs over 90% of the time. Air Canada is actually relatively tight with award availability, though they’ll certainly make seats available for incrementally more points.

And while on the whole Southwest is actually pretty good, there certainly are redemption complaints from members. And don’t forget that this is a bit apples-to-apples. Even if Southwest really is the best, who cares? You’re still redeeming your points to Tulsa or Oakland or wherever. That’s not really the aspirational sort of awards that make these programs in general valuable in the first place.

Lufthansa is pretty good, as the report suggests. But Singapore near the top? No way. And American should certainly be near the top not the bottom. But the study is highly sensitive to route selection bias.

Furthermore, a real limitation of the survey is querying routes on the program websites, which vary widely in quality. And some programs offer partner award availability online while others do not.

Any time you see different results for award availability between Continental and US Airways you know it’s a booking interface issue rather than an availability issue, at least accounting for 95% of the variance.

Sure, web functionality factors into overall member performance but it’s not entirely fair to blame lack of award inventory on a program’s IT problems.

Indeed, Delta concedes its results were bad, precisely because the study involved searching for awards online — they finally admit their website is broken!

But bizarrely also claim that their poor showing is due to the Northwest merger and more miles from the combined programs chasing their seats. That makes no sense since Delta and Northwest were already frequent flyer redemption partners before the merger.

Meanwhile rather than fight back with data, US Airways concedes the results of the survey — which are completely inconsistent with my own pretty extensive experience. Since the beginning of 2010 US Airways has been excellent with domestic award inventory, including transcon routes like Charlotte and Philadelphia to Los Angeles and San francisco, and especially in first class.

Instead of pointing this out (perhaps they don’t even realize?), they concede that their award availability is poor because (1) they release seats closer to departure than the study was examining and (2) they offer the seats “at a different price point” (i.e. for more miles – hah!).

This old canard unforunately gets repeated:

To get seats to vacation destinations, you typically need to book 11 months in advance, when airlines open up flights for reservations.

Now, some seats are released on some flights on some airlines when schedules load into the system — with most carriers 331 days out. But many flights don’t get any award seats at that time!

Airlines want to make awards available for seats that will otherwise go unsold. A year out they don’t necessarily have as good an idea of how many seats that will be, and their knowledge gets better as travel approaches — at 6 months out, 3 months out, 2 weeks out, etc. Many carriers release seats all along the way as they expect seats to remain open, many are much more generous closer to travel than 11 months out.

That doesn’t seem you can’t get awards right away when a schedule loads, each carrier is a bit different in that regard, but the notion that you need to call immediately when the schedule opens and that when you’re unsuccessful it means someone else got to those seats before you did is just wrong.

Ultimately the study, no matter how much it may get wrong in its particulars, points to the idea that not all airline miles are created equal. That’s an important point indeed.

My own currency preferences are first to go for flexibility, you really can’t underestimate the value of American Express Membership Rewards points for instance. (Starwood points are actually more valuable for airline mileage transfers in most cases, but the fact that they take several days to a couple of weeks to transfer makes it a bit difficult maintaining that flexibility at redemption time — you can’t really put an award on hold and then trasnfer points from Starwood).

And for airline programs in particular, I value Air Canada Aeroplan, Continental, US Airways, and American in that order among North American programs, IdeaWorks testing notwithstanding.