Each year, for 7 years now, IdeaWorks has published a study (.pdf) on which airlines offer the best award availability. And each year it is so fatally flawed as to give consumers almost the worst advice possible.
Roughly speaking this year’s piece — just like last year’s — finds that:
- Overall airline award availability is getting better .
- Low cost carriers are better for flyers looking for award seats than legacy airlines
Only the author of this ill-informed study could claim that frequent flyer programs are more rewarding this year than they were before,
“Overall, I think the consumer is being better served than the year before,” says Jay Sorensen, president of IdeaWorks, the consulting firm that conducted the study.
Here’s their top 5 programs for award availability:
The way it reaches these conclusions is by assuming away virtually everything we know about how the programs actually work.
Nowhere in the study do they account for the value or quality of what the low cost carriers like Southwest Airlines are getting you for your miles. They simply test whether seats are available, and for revenue-based programs they almost always are (they accept anything up to 25,000 points from Southwest or JetBlue while they only look at saver-level award space for legacy airlines, failing to recognize that the currencies operate on a different scale).
They also don’t look at the true cost of redemptions, factoring in whether a program adds fuel (or ‘carrier-imposed’) surcharges onto the price of an award. Just because a seat is available more often doesn’t mean it represents strong value.
More fundamentally, though — and I covered these flaws last year and in 2014 and in 2013 and in 2012, too (and 2011) — they don’t seem to make their flawed study materially better, perhaps because it gets reported uncritically as-is.
I believe it’s important to document the flaws each year when the study comes out precisely because its conclusions do get repeated, and therefore its erroneous claims need to be countered. (Scott McCartney, for instance — who I know knows better — gives it Wall Street Journal play each year.)
The results are dead wrong, because the methodology is dead wrong. In fact, consumers will be worse off if they pay any attention to it.
Here’s how they go about their study:
Booking queries for a party of two travelers were made at frequent flier program websites during March 2016. Some airlines require a Saturday night stay for reward travel; all of the queries used date pairings that included a Saturday night stay. While the city pairs varied for each frequent flier program, the travel dates did not. 280 specific dates were selected for survey queries and only reward seat availability for travel on the date specified was recorded; any departure time was acceptable. Furthermore, reward travel had to be available on the outbound and return dates queried. Overly circuitous routings with long elapsed travel times and layovers longer than 4 hours were not accepted.
Survey results reflect the availability of saver-style rewards (capacity controlled seats) with two exceptions. For Southwest, rewards priced up to 25,000 points (roundtrip) qualified as reward travel. For JetBlue, rewards priced at 25,000 points (roundtrip) qualified as reward travel.
The top 10 routes (based upon total seats offered for sale during a 12-month period) longer than 2,500 miles and the top 10 medium-haul routes (251 to 2,500 miles) were selected for each airline. Due to a lack of long-haul routes, the top 20 overall routes were queried for these airlines: airberlin, Air Asia, GOL, JetBlue, Southwest, and Virgin Australia. The selection of Avianca’s 251-2,500 mile routes includes markets under 250 miles. Because surface transport in Colombia compares so poorly, these short markets are among the airline’s most popular for reward travel; the change better aligns consumer preference with the query methodology.
The base methodology of the study is highly flawed. And monkeying with methodology for specific carriers, and finding those carriers do well, is problematic.
Notably, that’s happening with the routes selected for their top performing program airberlin, which also happens to be a featured client on the home page of the company producing the study.
- They searched airline websites only. Even where partners have access to the exact same award inventory, their results will vary markedly. This isn’t a study of award availability, it’s a study of website redemption functionality. That’s potentially useful, and it’s acknowledged by the study, but it suggests a more limited conclusion than is being made here.
- They’re searching different routes for each airline but over the same dates, which ignores the effects of high and low seasons. Since they’re looking at fixed months and days prior to departure for each airline’s most popular routes, airlines whose routes fall into high season during that date range are disadvantaged.
- They’re making subjective judgments about ‘overly circuitous routes’ but not about departure time, consistently offering 6am flights or redeyes counts just as much as offering times many consumers would find more desirable.
- They’re substituting routes. airberlin wins top honors in the study but they’ve chosen to cherry pick routes to compare instead of using the same methodology as airlines receiving lower rankings in the survey. Low cost carriers with fewer long haul routes get to count award availability on more short haul routes instead, and unsurprisingly low cost carriers fare well.
- They count saver award space only, except when they don’t. Revenue-based programs let you redeem for any seat with points, just requiring more points to do it on expensive flights. They count saver space only for legacy airlines, but count Anytime seats for Southwest (not just ‘Wanna Get Away’ fares). They cap the value of those awards at 25,000 points for JetBlue — not recognizing that 25,000 points with JetBlue is different than 25,000 points with a legacy carrier.
Their methodology also:
- Ignores cost of acquiring the miles. It may be really easy to earn miles with an airline that has several partners and bonuses, so it could even make sense to spend twice as many points. But any seat availability for extra points doesn’t count, except where it does (in favored programs that do well in the survey).
- Ignores the value of a given redemption. Greyhound Road Rewards may give you a free bus trip every 10 trips, and if those bus seats aren’t capacity controlled then they satisfy their riders every time. But that doesn’t make Greyhound Road Rewards a more lucrative, rewarding, satisfying program than United MileagePlus or American AAdvantage which allow you to see the world, in a premium cabin no less (a travel style many would never be able to afford but for the points, but the study looks at coach only).
The methodology leads to this almost self-refuting conclusion:
As in previous years, survey findings indicate frequent fliers are better served by the reward programs at value-oriented airlines. The average among the six value-oriented airlines (Air Asia, airberlin, GOL, JetBlue, Southwest, and Virgin Australia) was 87.4%, while the more traditional
carriers in the survey group registered 72.6%
At one point JetBlue didn’t even want a frequent flyer program. They don’t offer international partner redemptions at all. Air Asia suffers the ‘Greyhound Road Rewards problem’ — earn their points and your reward is more flying on Air Asia.
What’s fundamentally misleading is that the survey tests availability for long haul routes on legacy carriers while using short haul routes for the top performers.
To be sure, if you want domestic coach redemptions, then you might well be attracted to Southwest et al. But then that implies other things as well – that you shouldn’t have a points-earning credit card, for instance, since a 2% cash back card will suit you best.
Ultimately, though, the study tries to prove too much with a methodology that doesn’t come close to supporting its conclusions. They go looking for strong value in low cost revenue-based frequent flyer programs and find it, without regard to what ‘value’ for the consumer really is.
Folks following the advice of this study will find themselves with tickets to Florida and Ohio, while the legacy frequent flyer programs that score less well continue to offer greater options to their members.
[…] this the annual "study" by IdeaWorks? If so, you might find this critique interesting: Dangerous New Study Gets Frequent Flyer Programs Exactly Backward – View from the Wing Reply With […]