Each year, for five years now, IdeaWorks has published a study (.pdf) on which airlines offer the best award availability and each year Scott McCartney picks it up approvingly in the Wall Street Journal without a recognition of its flaws.
Roughly speaking this year’s piece finds that:
- Overall airline award availability is getting better despite an improving economy (more seats sold).
- Low cost carriers are better for flyers looking for award seats than legacy airlines.
- International airlines are better at offering award seats than US ones.
But the only way it reaches these conclusions is by assuming away virtually everything we know about how these programs actually work.
Nowhere in the study do they account for the value or quality of what the low cost carriers like Southwest Airlines are getting you for your miles.
And while airlines like Singapore do have excellent award availability for members of their own Krisflyer program, there’s no comparison of award chart pricing (Singapore’s chart is more expensive for comparable long haul awards) and no comparison of out of pocket cash costs — like many Asian and European airlines they add fuel surcharges onto award tickets that add several hundred dollars to cost. Just because a seat is available more often doesn’t mean it’s a better value.
More fundamentally, though — and I wrote about the flaws in their study last year and the year before, too (and the year before that) — they don’t seem to make their flawed study materially better, perhaps because it gets reported uncritically as-is.
The results are dead wrong, because the methodology is dead wrong. In fact, consumers will be worse off if they pay any attention to it.
Here’s how they go about their study:
Booking queries for a party of two travelers were made at frequent flier program websites during March 2014. Some airlines require a Saturday night stay for reward travel; all of the queries used date pairings that included a Saturday night stay. While the city pairs varied for each frequent flier program, the travel dates did not. 280 specific dates were selected for survey queries and only reward seat availability for travel on the date specified was recorded; any departure time was acceptable. Furthermore, reward travel had
to be available on the outbound and return dates queried. Overly circuitous routings with long
elapsed travel times and layovers longer than 4 hours were not accepted.
Survey results reflect the availability of saver-style rewards (capacity controlled seats) with two exceptions. For Southwest, Anytime or Wanna Get Away rewards priced up to 25,000 points (roundtrip) qualified as reward travel. For JetBlue, rewards priced at 25,000 points (roundtrip) qualified as reward travel.
The top 10 routes (based upon total seats offered for sale during a 12-month period) longer than 2,500 miles and the top 10 medium-haul routes (251 to 2,500 miles) were selected for each airline. Alaska Airlines was switched to this methodology for 2013 due to the carrier’s increased emphasis on the Hawaii market. Due to a lack of long-haul routes, the top 20 overall routes were queried for
these airlines: Air Asia, GOL, JetBlue, Southwest, and Virgin Australia. Ten top Europe – Palma de Mallorca city pairs (out of 20) were substituted for airberlin to reflect the carrier’s major Mediterranean emphasis on holiday flights.
This seems really problematic, as though it will yield strange results (which it does, as we’ll see in a moment):
- They searched airline websites only. So even though US Airways and United had access in March to the same award space (more or less), United offers online booking of many partners while US Airways does not. Miles in each airline’s programs could access the exact same saver award seats.
- They’re searching different routes for each airline but over the same dates, which ignores the effects of high and low seasons. Since they’re looking at fixed months and days prior to departure for each airline’s most popular routes, airlines whose routes fall into high season during that date range are disadvantaged.
- They’re making subjective judgments about ‘overly circuitous routes’ but not about departure time, consistently offering 6am flights or redeyes counts just as much as offering times many consumers would find more desirable.
- They’re substituting routes. airberlin wins top honors in the study but they’ve chosen to cherry pick routes to compare instead of using the same methodology as applied to other airlines. Low cost carriers were fewer long haul routes get to count award availability on more short haul routes instead, and unsurprisingly low cost carriers fare well.
- They count saver award space only, except when they don’t. Revenue-based programs let you redeem for any seat with points, just requiring more points to do it on expensive flights. They count saver space only for legacy airlines, but count Anytime seats for Southwest (not just ‘Wanna Get Away’ fares). They cap the value of those awards at 25,000 points — not recognizing that 25,000 points with Southwest or or JetBlue is different than 25,000 points with a legacy carrier.
Their methodology also:
- Ignores cost of acquiring the miles. It may be really easy to earn miles with an airline that has several partners and bonuses, so it could even make sense to spend twice as many points. But any seat availability for extra points doesn’t count, except where it does (in favored programs that do well in the survey).
- Ignores the value of a given redemption. Greyhound Road Rewards may give you a free bus trip every 10 trips, and if those bus seats aren’t capacity controlled then they satisfy their riders every time. But that doesn’t make Greyhound Road Rewards a more lucrative, rewarding, satisfying program than United MileagePlus or American AAdvantage which allow you to see the world, in a premium cabin no less (a travel style many would never be able to afford but for the points, but the study looks at coach only).
The conclusion that the low cost carriers with points-based programs where those points are good for any seat is misleading. The methodology focuses only on short haul routes for those carriers, while focusing on long haul routes for other airlines. And since they don’t face the same capacity controls their availability (“100%” for Southwest!) is compared against saver awards only at the legacy carriers.
And the conclusion that non-US airlines are better — while possibly true — is not supported by the approaches taken in the study.
The study — which focuses on online redemption and suggests the most valuable points are those used as currency to buy tickets at market prices — is sponsored by Switchfly which sells online services to frequent flyer programs with a focus on using points as currency.
To be sure, if you want domestic coach redemptions,, then you might well be attracted to Southwest et al. But then that implies other things as well – that you shouldn’t have a points-earning credit card, for instance, since a 2% cash back card will suit you best.
Ultimately, though, the study tries to prove too much with a methodology that doesn’t come close to supporting its conclusions. They go looking for strong value in low cost revenue-based frequent flyer programs and find it, without regard to what ‘value’ for the consumer really is.
Folks following the advice of this study will find themselves with tickets to Florida and Ohio, while the legacy frequent flyer programs that score less well continue to offer greater options to their members.