Each year, for six years now, IdeaWorks has published a study (.pdf) on which airlines offer the best award availability. And each year it is so fatally flawed as to give consumers almost the worst advice possible.

Roughly speaking this year’s piece finds that:

• Overall airline award availability is getting better .
  
• Low cost carriers are better for flyers looking for award seats than legacy airlines

But the way it reaches these conclusions is by assuming away virtually everything we know about how these programs actually work.

Nowhere in the study do they account for the value or quality of what the low cost carriers like Southwest Airlines are getting you for your miles. They simply test whether seats are available, and for revenue-based programs they always are.

And while airlines like Singapore do have excellent award availability for members of their own Krisflyer program, there’s no comparison of award chart pricing and no comparison of out of pocket cash costs — like many Asian and European airlines they add fuel surcharges onto award tickets that add several hundred dollars to cost. Just because a seat is available more often doesn’t mean it’s a better value.

More fundamentally, though — and I wrote about how backward this study is last year and in 2013 and in 2012, too (and 2011) — they don’t seem to make their flawed study materially better, perhaps because it gets reported uncritically as-is. I believe it’s important to document the flaws each year when the study comes out precisely because its conclusions do get repeated, and therefore its erroneous claims need to be countered. (Scott McCartney, for instance — who should know better — gives it Wall Street Journal play each year.)

The results are dead wrong, because the methodology is dead wrong. In fact, consumers will be worse off if they pay any attention to it.

Here’s how they go about their study:

Booking queries for a party of two travelers were made at frequent flier program websites during March 2015. Some airlines require a Saturday night stay for reward travel; all of the queries used date pairings that included a Saturday night stay. While the city pairs varied for each frequent flier program, the travel dates did not. 280 specific dates were selected for survey queries and only reward seat availability for travel on the date specified was recorded; any departure time was acceptable. Furthermore, reward travel had to be available on the outbound and return dates queried. Overly circuitous routings with long elapsed travel times and layovers longer than 4 hours were not accepted.

Survey results reflect the availability of saver-style rewards (capacity controlled seats) with two exceptions. For Southwest, Anytime or Wanna Get Away rewards priced up to 25,000 points (roundtrip) qualified as reward travel. For JetBlue, rewards priced at 25,000 points (roundtrip) qualified as reward travel.

The top 10 routes (based upon total seats offered for sale during a 12-month period) longer than 2,500 miles and the top 10 medium-haul routes (251 to 2,500 miles) were selected for each airline. Due to a lack of long-haul routes, the top 20 overall routes were queried for these airlines: Air Asia, GOL, JetBlue, Southwest, and Virgin Australia. Ten top Europe – Palma de Mallorca city pairs (out of 20 total) were substituted for airberlin to reflect the carrier’s major Mediterranean
emphasis on holiday flights. The selection of Avianca’s 251-2,500 mile routes was changed this year to include markets under 250 miles. .

(Emphasis mine.)

The base methodology is highly flawed. And monkeying with methodology for specific carriers, and finding those carriers do well, is problematic.

• They searched airline websites only. Even where partners have access to the exact same award inventory, their results will vary markedly. This isn’t a study of award availability, it’s a study of website redemption functionality. That’s potentially useful, but it’s different than what its authors claim that it is.

• They’re searching different routes for each airline but over the same dates, which ignores the effects of high and low seasons. Since they’re looking at fixed months and days prior to departure for each airline’s most popular routes, airlines whose routes fall into high season during that date range are disadvantaged.

• They’re making subjective judgments about ‘overly circuitous routes’ but not about departure time, consistently offering 6am flights or redeyes counts just as much as offering times many consumers would find more desirable.

• They’re substituting routes. airberlin wins top honors in the study but they’ve chosen to cherry pick routes to compare instead of using the same methodology as applied to other airlines. They claim it’s necessary because of the carrier’s strategy, but if that were true those flights would break into the top 10 like with every other airline they survey. Low cost carriers with fewer long haul routes get to count award availability on more short haul routes instead, and unsurprisingly low cost carriers fare well.

• They count saver award space only, except when they don’t. Revenue-based programs let you redeem for any seat with points, just requiring more points to do it on expensive flights. They count saver space only for legacy airlines, but count Anytime seats for Southwest (not just ‘Wanna Get Away’ fares). They cap the value of those awards at 25,000 points for JetBlue — not recognizing that 25,000 points with JetBlue is different than 25,000 points with a legacy carrier.

Their methodology also:

• Ignores cost of acquiring the miles. It may be really easy to earn miles with an airline that has several partners and bonuses, so it could even make sense to spend twice as many points. But any seat availability for extra points doesn’t count, except where it does (in favored programs that do well in the survey).

• Ignores the value of a given redemption. Greyhound Road Rewards may give you a free bus trip every 10 trips, and if those bus seats aren’t capacity controlled then they satisfy their riders every time. But that doesn’t make Greyhound Road Rewards a more lucrative, rewarding, satisfying program than United MileagePlus or American AAdvantage which allow you to see the world, in a premium cabin no less (a travel style many would never be able to afford but for the points, but the study looks at coach only).

The methodology leads to this almost self-refuting conclusion:

As in previous years, survey findings indicate frequent fliers are better served by the reward programs at value-oriented airlines. The average among the six value-oriented airlines (Air Asia, airberlin, GOL, JetBlue, Southwest, and Virgin Australia) was nearly 90%, while the more traditional carriers in the survey group registered 69.4%.

JetBlue barely has, and at one point didn’t really even want, a frequent flyer program. They don’t offer international partner redemptions at all. Air Asia suffers the ‘Greyhound Road Rewards problem’ — earn their points and your reward is more flying on Air Asia.

The conclusion that the low cost carriers with points-based programs where those points are good for any seat is misleading. The methodology focuses only on short haul routes for those carriers, while focusing on long haul routes for other airlines. And since they don’t face the same capacity controls their availability (“100%” for Southwest!) is compared against saver awards only at the legacy carriers.

And the conclusion that non-US airlines are better — while possibly true — is not supported by the approaches taken in the study.

The study — which focuses on online redemption and suggests the most valuable points are those used as currency to buy tickets at market prices — is sponsored a company which sells online services to frequent flyer programs with a focus on using points as currency. They sell ‘pay with points’ technology. And the study makes this rather silly claim:

The results indicate the pay-with-points method used by Southwest and JetBlue provides better value than airlines using traditional reward pricing. It’s safe to assume miles and points for these seven carriers provide similar value.

They actually ‘assume’ that JetBlue points are more valuable than United’s and American’s, and that Delta’s points are more valuable than Alaska’s. Like I said, self-refuting.

To be sure, if you want domestic coach redemptions, then you might well be attracted to Southwest et al. But then that implies other things as well – that you shouldn’t have a points-earning credit card, for instance, since a 2% cash back card will suit you best.

Ultimately, though, the study tries to prove too much with a methodology that doesn’t come close to supporting its conclusions. They go looking for strong value in low cost revenue-based frequent flyer programs and find it, without regard to what ‘value’ for the consumer really is.

Folks following the advice of this study will find themselves with tickets to Florida and Ohio, while the legacy frequent flyer programs that score less well continue to offer greater options to their members.

For an industry consultant, the study’s author even evinces a surprising fundamental misunderstanding of when reward seats are made available by airlines:

It’s difficult to discern why airlines choose to decrease the supply of reward seats for their members. For Brazil-based GOL and Air China, perhaps it’s the troubles facing their national economies. The reversal for Turkish is perplexing because the airline is performing so very well in terms of growth, revenue, and profits.

It’s precisely when times are bad and seats are going unsold that there’s excess capacity to release at the saver level, and when revenue and profits are up as a result of full planes at high fares that’s going to be when there’s little if any saver award availability. In other words, they have the mechanics here exactly backward even if their data is meaningful.