Each year, for 7 years now, IdeaWorks has published a study (.pdf) on which airlines offer the best award availability. And each year it is so fatally flawed as to give consumers almost the worst advice possible.
Roughly speaking this year’s piece — just like last year’s — finds that:
- Overall airline award availability is getting better .
- Low cost carriers are better for flyers looking for award seats than legacy airlines
Only the author of this ill-informed study could claim that frequent flyer programs are more rewarding this year than they were before,
“Overall, I think the consumer is being better served than the year before,” says Jay Sorensen, president of IdeaWorks, the consulting firm that conducted the study.
Here’s their top 5 programs for award availability:
The way it reaches these conclusions is by assuming away virtually everything we know about how the programs actually work.
Nowhere in the study do they account for the value or quality of what the low cost carriers like Southwest Airlines are getting you for your miles. They simply test whether seats are available, and for revenue-based programs they almost always are (they accept anything up to 25,000 points from Southwest or JetBlue while they only look at saver-level award space for legacy airlines, failing to recognize that the currencies operate on a different scale).
They also don’t look at the true cost of redemptions, factoring in whether a program adds fuel (or ‘carrier-imposed’) surcharges onto the price of an award. Just because a seat is available more often doesn’t mean it represents strong value.
More fundamentally, though — and I covered these flaws last year and in 2014 and in 2013 and in 2012, too (and 2011) — they don’t seem to make their flawed study materially better, perhaps because it gets reported uncritically as-is.
I believe it’s important to document the flaws each year when the study comes out precisely because its conclusions do get repeated, and therefore its erroneous claims need to be countered. (Scott McCartney, for instance — who I know knows better — gives it Wall Street Journal play each year.)
The results are dead wrong, because the methodology is dead wrong. In fact, consumers will be worse off if they pay any attention to it.
Here’s how they go about their study:
Booking queries for a party of two travelers were made at frequent flier program websites during March 2016. Some airlines require a Saturday night stay for reward travel; all of the queries used date pairings that included a Saturday night stay. While the city pairs varied for each frequent flier program, the travel dates did not. 280 specific dates were selected for survey queries and only reward seat availability for travel on the date specified was recorded; any departure time was acceptable. Furthermore, reward travel had to be available on the outbound and return dates queried. Overly circuitous routings with long elapsed travel times and layovers longer than 4 hours were not accepted.
Survey results reflect the availability of saver-style rewards (capacity controlled seats) with two exceptions. For Southwest, rewards priced up to 25,000 points (roundtrip) qualified as reward travel. For JetBlue, rewards priced at 25,000 points (roundtrip) qualified as reward travel.
The top 10 routes (based upon total seats offered for sale during a 12-month period) longer than 2,500 miles and the top 10 medium-haul routes (251 to 2,500 miles) were selected for each airline. Due to a lack of long-haul routes, the top 20 overall routes were queried for these airlines: airberlin, Air Asia, GOL, JetBlue, Southwest, and Virgin Australia. The selection of Avianca’s 251-2,500 mile routes includes markets under 250 miles. Because surface transport in Colombia compares so poorly, these short markets are among the airline’s most popular for reward travel; the change better aligns consumer preference with the query methodology.
The base methodology of the study is highly flawed. And monkeying with methodology for specific carriers, and finding those carriers do well, is problematic.
Notably, that’s happening with the routes selected for their top performing program airberlin, which also happens to be a featured client on the home page of the company producing the study.
In addition:
- They searched airline websites only. Even where partners have access to the exact same award inventory, their results will vary markedly. This isn’t a study of award availability, it’s a study of website redemption functionality. That’s potentially useful, and it’s acknowledged by the study, but it suggests a more limited conclusion than is being made here.
- They’re searching different routes for each airline but over the same dates, which ignores the effects of high and low seasons. Since they’re looking at fixed months and days prior to departure for each airline’s most popular routes, airlines whose routes fall into high season during that date range are disadvantaged.
- They’re making subjective judgments about ‘overly circuitous routes’ but not about departure time, consistently offering 6am flights or redeyes counts just as much as offering times many consumers would find more desirable.
- They’re substituting routes. airberlin wins top honors in the study but they’ve chosen to cherry pick routes to compare instead of using the same methodology as airlines receiving lower rankings in the survey. Low cost carriers with fewer long haul routes get to count award availability on more short haul routes instead, and unsurprisingly low cost carriers fare well.
- They count saver award space only, except when they don’t. Revenue-based programs let you redeem for any seat with points, just requiring more points to do it on expensive flights. They count saver space only for legacy airlines, but count Anytime seats for Southwest (not just ‘Wanna Get Away’ fares). They cap the value of those awards at 25,000 points for JetBlue — not recognizing that 25,000 points with JetBlue is different than 25,000 points with a legacy carrier.
Their methodology also:
- Ignores cost of acquiring the miles. It may be really easy to earn miles with an airline that has several partners and bonuses, so it could even make sense to spend twice as many points. But any seat availability for extra points doesn’t count, except where it does (in favored programs that do well in the survey).
- Ignores the value of a given redemption. Greyhound Road Rewards may give you a free bus trip every 10 trips, and if those bus seats aren’t capacity controlled then they satisfy their riders every time. But that doesn’t make Greyhound Road Rewards a more lucrative, rewarding, satisfying program than United MileagePlus or American AAdvantage which allow you to see the world, in a premium cabin no less (a travel style many would never be able to afford but for the points, but the study looks at coach only).
The methodology leads to this almost self-refuting conclusion:
As in previous years, survey findings indicate frequent fliers are better served by the reward programs at value-oriented airlines. The average among the six value-oriented airlines (Air Asia, airberlin, GOL, JetBlue, Southwest, and Virgin Australia) was 87.4%, while the more traditional
carriers in the survey group registered 72.6%
At one point JetBlue didn’t even want a frequent flyer program. They don’t offer international partner redemptions at all. Air Asia suffers the ‘Greyhound Road Rewards problem’ — earn their points and your reward is more flying on Air Asia.
What’s fundamentally misleading is that the survey tests availability for long haul routes on legacy carriers while using short haul routes for the top performers.
To be sure, if you want domestic coach redemptions, then you might well be attracted to Southwest et al. But then that implies other things as well – that you shouldn’t have a points-earning credit card, for instance, since a 2% cash back card will suit you best.
Ultimately, though, the study tries to prove too much with a methodology that doesn’t come close to supporting its conclusions. They go looking for strong value in low cost revenue-based frequent flyer programs and find it, without regard to what ‘value’ for the consumer really is.
Folks following the advice of this study will find themselves with tickets to Florida and Ohio, while the legacy frequent flyer programs that score less well continue to offer greater options to their members.
Tell it like it is Gary!
Folks following the advice of this study will find themselves with tickets to Florida and Ohio, while the legacy frequent flyer programs that score less well continue to offer greater options to their members.
That does seem to be how people redeem miles, though. A bit dated, but:
Note that AS is a legacy program with lots of high-value redemptions that bloggers salivate and endlessly pump (EK, CX and so on). But what gets redeemed? More than 4 out of 5 rewards end up being on AS/QX metal, which automatically precludes international travel other than to Mexico or SJO. Nineteen out of twenty are redeemed on US-based airlines, which roundly get panned for union workforces and no Dom and caviar.
The bottom line is the blogger aspirational trip of a weekend in Singapore followed by some time at the Park Hyatt Maldives isn’t how most people spend their miles. Which is fine.
Oh, my picture got left out.
http://onemileatatime.img.boardingarea.com/wp-content/uploads/2013/11/Alaska-Investor-Day1.png
Thanks for reiterating these points. It’s nice to have somebody calling BS when these wildly skewed studies are presented to the unwitting public.
It’s almost as if IdeaWorks were trying to curry the favor of airlines. That’s strange, since IdeaWorks is completely objective and doesn’t market its service to airlines. Oh, wait…
Is IdeaWorks even legitimate? Their corporate address is a single-family home, and one of the three emails on their Contact page is a Yahoo address.
You have rightly called this nonsense out for years. Keep it up.
You (and I) have repeatedly communicated to that WSJ guy what crapola these “studies” are. He even once got in an email exchange about how wrong I was about WN, laughably.
At some point, you really have to wonder what actually is going on there.
@eponymous — wonderful insight, and completely agree (“The bottom line is the blogger aspirational trip of a weekend in Singapore followed by some time at the Park Hyatt Maldives isn’t how most people spend their miles. Which is fine.”)
@Gary — you slam every single study out there. Why don’t you just do your own instead of always bitching about what after all are very valuable insights that we otherwise we wouldn’t have?
@Tony — the present study contains “very valuable insights”? That is laughable. The present study is the opposite of face valid, and Gary has pointed out specific methodological errors. Which of Gary’s specific claims seem untenable to you?
That Gary slams every single study out there does not mean Gary is wrong.
Wow, what a blast. And editorially impure. A few observations about the frothy commentary. First, thank you for posting the logos of my clients. Airberlin is at the top of the survey results and Avianca is at the bottom. And what’s the point? I report good and bad results regardless of client status.
But here’s the big disservice to readers. This year’s survey includes a new look at reward payback, which assesses the major US airlines on how much they return in rewards for a dollar spent on base fare. We did a similar report last fall on hotel programs. The article above offers zero mention of this new method of evaluation. I’m truly surprised by this “Fox News” approach to coverage. I guess if it doesn’t fit the narrative, the reader shouldn’t be informed. That is truly surprising, or at best, an oversight.
I chuckled about the criticism of airberlin and “cherry picking” routes. A simple analysis of their route structure would reveal they have few long haul routes that offer daily service. So I’m at a loss as to which long haul routes should be added to the survey. The overwhelming majority of their network (measured by seats flown) is short and medium haul and these destinations represent the largest share for redemption. Thus, for airberlin they represent the focus of the query activity.
In terms of commentary regarding me trying to curry favor with certain airlines, that’s nonsense. As a US based consulting company, it would be in our financial interest to boost the results of American, Delta, and United. I think anyone who has read the 7 years of surveys would readily conclude these carriers have not been top ranked by the survey.
My firm puts two months of hard work into conducting 7,000+ queries to produce this report, and I without hesitation stand behind the integrity of this work. You are invited to question the logic of the methods, but I take extreme exception to anyone questioning my ethics. Just to advise, I have no interest in a back and forth debate here. God knows, there’s too much of that silliness in the media due to the sorry state of presidential politics.
Best regards,
Jay Sorensen
President, IdeaWorksCompany
I apologize for any implication that individual airlines were favored over others. I meant to claim that the methodology is systematically favorable to airlines relative to the reality of saver award availability.
Anyone who has tried to book saver awards for popular routes or for summer travel knows that these awards have never been harder to redeem or more expensive in miles and points when they are available. If the survey says differently, the methodology must be mismatched to customers’ redemption experience.
One specific change is that the survey should include some element of short-notice redemption. Most airlines impose hefty fees for travel less than 3 weeks out. Southwest has no fees but jacks redemption prices up to confiscatory levels when advance purchase cash fares are no longer offered.
There should also be some accounting for huge fees to change or cancel award travel. Southwest ought to be able to show off its lack of award fees, which is a prime differentiator for Rapid Rewards.
@ Jay
Accepting you don’t want a debate about ethics, but you might be interested in getting your facts right and be open to positive critique about your methods and definitions.
Virgin Australia clanger
As an Australian based reader, please may I correct you on your apparent ignorance regarding Virgin Australia. This airline is no longer a value-oriented airline and hasn’t been for some years. It is a full service airline, with a domestic and international business class, a lounge network, in flight catering, etc., indeed sees itself as the equal of rival Qantas in terms of product offering. (Virgin Australia has a separate value-orientated operation called Tiger just as Qantas has Jetstar).
Differential reward levels and rates
By selecting the base reward levels between different airlines you are already in trouble. By way of example, SQ has three levels of award, and competitors Qantas / Emirates / Etihad / Cathay Pacific etc., have one. The cost of the mid-level standard rate (with the website discount of 15%) on SQ is more aligned with the redemption rates for such competitors. But presumably you have omitted SQ standard and only included saver level. SQ is thereby getting a raw deal in your methodology if you take the point of view of the consumer rather than the airline.
Confusing airlines with airline reward schemes and computer only reward access
You have neither measured comparatively airline reward availability nor airline reward scheme reward availability. If the former, you should not include any partner award availability: all those juicy Alaskan and Hawaiian rewards will show on the AA site (if they are included in the popular routes you have chosen). If the latter you need all rewards – JAL rewards won’t show on the QF website and calls need to be made to secure them. In practice you have just measured award space as available online which does not reflect the consumer experience in terms of the true reward inventory at their disposal.
Biased levelling of the playing field
You have forced a Saturday night into your selection process. This favours the airlines which require a Saturday overnight. To collect an unbiased dataset you should have picked dates at random irrespective of the Saturday night requirement.
Economy versus Premium
I may have missed it, but I am assuming you are only including economy rewards? Or have you included any reward available (thus included a premium economy / business / first )? Again, the analysis needs to encompass and differentiate classes of travel, especially as the most value inherent in rewards are at the premium end.
Mis-groupings
Your data may hold insights if the datasets were selected into subsets for more meaningfully comparative analysis: US domestic, international long haul, etc. The overall table requires so many caveats drawing conclusions from it (as you do) is very misleading from the perspective of the consumer.
Interest to the airline of the customer
This report appears to have been written from the perspective of the airline, not the customer. For the customer, you need to look at value (as @ Gary points out). The analysis as summarised holds very little real-world relevant information for the customer!
Ethics and disclosure
I can’t see any problem with @ Gary pointing out some of the airlines are your clients (or for that matter that the report was commissioned by a company supplying airline loyalty expertise to many of the airlines).
However, IMHO, the report as summarised is very much in the mindset of the airline and not the customer – possibly helpful to airlines in promoting their loyalty programs, but of little value (if not misleading) to customers trying to get value out of the system.
…@ Jay…
Just to add….on the selection of the top ten routes…OK for QF domestic where the vast majority of air traffic is on a handful of routes with dozens of daily flights (SYD-MEL,SYD-BNE, etc), but surely ridiculous for an airline the size of AA which offers 1000’s of routes?!
But then prejudicial to QF in judging its reward availability because its popular routes do not necessarily include reward opportunities on its major partner EK, which are extensive and outnumber QF’s international flights by orders of magnitude.
.. and then you select 20 not 10 for Virgin Australia? But you exclude VA’s international flights (which go through cycles of reward availability and non availability)?!
…exempting Avianca and including short hops (<250) surely prejudicial to Avianca since small aircraft (presumably) ply such routes
…the more I think about this study the more holes perforate its very fabric…
…sorry, @ Jay…I hope you delivered to your client's brief because your methods and conclusions are mental from the perspective of the customer.
…disappointing if the media pick up this stuff uncritically and thereby misinform the public about the value and options available to them…
Study seemed fair to me. Gary just values luxury variables more versus just getting a seat to get you somewhere. Which is where I would guess most travelers are interested in Not sure why there is any need to be so harsh and blasting same writer every year over same topic.
@Jay you don’t respond to the substantive criticisms of your study. You take a look at flights at one point in time, when you are making queries, and don’t account for high or low seasonality on the different routes you’re looking at for different airlines.
It would be one thing to say “I’m taking the top 20 routes for each airline” but instead you aren’t even doing that you’re taking a mix of short and long haul routes, but substituting short for long ones for certain carriers and then those are the carriers that are doing overwhelmingly the best.
Then you don’t recognize that the number of points in revenue-based programs isn’t comparable. So you accept anything from Southwest up to 25,000 points. And you’re looking at short haul routes. Your findings are “there are always Southwest short haul flights available for sale at $350 or less when booked in advance.” That’s not a revolutionary finding. Nor does it tell you their program is better than competitor programs.
As far as ‘reward payback’ I did a separate post on your look at hotel programs. I thought it was probably the best piece you had done, though I thought the headline results were something of an overclaim.
http://viewfromthewing.com/2015/10/21/new-study-declares-the-best-loyalty-program-did-it-get-things-right/
We can go into the positives and negatives of your take on reward payback here, but the headline that’s getting coverage – so what was really worth digging into – was the question of which programs are best at award availability and why.
I find it interesting that you say “You are invited to question the logic of the methods” but then don’t justify the methodological issues I’ve taken umrage at. It’s unfortunate though that you conclude “I have no interest in a back and forth debate here.” With regard to your reference to presidential politics, SAD!
Studies like this show up in USA Today, MSNBC, WSJ, etc. and naive people just accept it as true. Many folks that don’t fly much will read this and agree that Southwest is still a discount carrier and that it is best for them.
An experienced flyer will see the flaws and ask “what if I want to fly more than 250 miles without a Saturday stay?”
By the way, props to Jay for posting here. The discussion must not be fun, but it will improve his product. He is clearly diligent in his work. Award availability has so many facets that characterizing the customer’s redemption experience in one or two numbers is impossible. To paraphrase Einstein, the study needs to be as simple as possible but no simpler.
This makes me think that there actually hasn’t been a legitimate study done. A real study would probably have to make 100,000+ queries over a period of 1 year. Is Air Berlin good? Maybe. Try booking 1,2,3,4,5,6,7, and 8 weeks before a holiday weekend. Penalize results if only 6am flights are available. Penalize results if there are taxes/fees. Penalize results in cases of Southwest when it costs more points to redeem the ticket. This binary available/not available study is missing so many factors it’s hard to even know where to begin.
Also it would be nice to make a correlation between redemption rates and current market rate. Example:
Let’s say AA has tickets for some flight for 12.5k. It’s the best redemption rate right now. Every other airline is more expensive. Now go check the price of a similar ticket on Orbitz.com. You can see there are many tickets available for $100.
Now in a different query, for a different time, AA is showing 50k for a redemption. Go check Orbitz.com – Now you can see the average ticket is $400.
However, United has a redemption for 25k in both cases.
So when tickets are cheap, AA is best, and when tickets are hard to get and expensive, United is best.
I think most would say that AA is essentially pointless here. Even when AA is cheapest, you would be better off buying a ticket with cash.
SOOO many ways to make this study better.
to be technical, you can redeem jetblue points for hawaiian airlines flights to japan, though it’s an awful value and depends on your idea of “international partner award”
Just try to get an AA MileSAAver Business or First award to Europe…….and not on ripoff BA.
It might be helpful to visit my website and read the entire press release for more clarity. There is commentary in the postings which indicate that’s not been done. There’s more to this survey than was critiqued in the blog. I’m specifically referring to “Reward Payback” which measures the rate of return associated with spending and accrual. And of course, there are caveats provided regarding the limitations of that method too.
Moreover, the survey does not purport to be the final word, or complete assessment of frequent flier programs. It merely attempts to address the issue of online saver style reward availability. I think the frustration occurs for many of you when the survey is expected to be too many things. And yes, everyone at IdeaWorksCompany works from their homes – – I would never have it any other way.
Over and out from wonderful Milwaukee.
Jay Sorensen
President, IdeaWorksCompany
Peach it to him Gary. It’s just fascinating WSJ would use a one man home based ‘consultancy’ as a basis of a 2 page article.
@Jay Sorensen – I very much appreciate this year that the press release embraces the online limitations this year, which is an improvement over previous years. I think shifting to the payback discussion is really changing the subject, I’m focused here on the availability portion of the assessment and I don’t think you really address the methodological problems with it.
I applaud the effort, but given that it’s reported broadly as being a much more ‘complete assessment’ than it is of award availability I think it’s important to point out that it really doesn’t demonstrate what it’s reported to demonstrate.
As I say, it simply shows for Southwest that you can book short haul advance purchase tickets for less than $350, since you use 25,000 Rapid Rewards points as a standard — not the same as 25,000 miles in a legacy program — and you substitute short haul routes for long ones you’re testing those other programs for. It shouldn’t be any surprise that the programs you substitute routes for wind up at the top of the results?
Thanks, Jay, for commenting — I genuinely appreciate it.
Gary
@ Jay
Yes, some are expecting more than the scope of the report.
No doubt designing such surveys is a complex challenge and any method will have its pros and cons – the ever present trap is in the temptation to over reach in concluding positions. Also, the primary author cannot be held responsible for being quoted out of context or inaccurately or without the associated qualifications by a third party.
As customers / frequent flyers the issue of reward seat availability is one of great interest: many of us are wrestling with the math to work out our best strategies and none work without the airlines “playing fair” in making reward seats available.
To that extent it is great that your client and your agency embark on this exercise.
Hopefully, the move towards an analysis of payback can be explored further if the client is willing in future years.
Good luck with the project!
@ David
Your prejudices against small business and those, which may operate out of a home are misplaced.
Small consultancy businesses can often offer much better value than a large consultancy firm which typically hands the project over to a junior member of staff with some oversight and charges an inflated full rate (i.e. pay $200 per hour and get the work of a $20 hour junior).
Small consultancy businesses operate with low overhead (no expensive offices, payroll and HR and support staff to employ, etc).
Small consultancy businesses can offer the client highly skilled / competent / experienced /specialised individuals to work on a project.
And working from home can be far more productive than sitting in a cube in some client side open plan office, having to interact with gossiping folk, getting dragged into office politics, being encouraged to sit through irrelevant meetings, endless safety briefings, and waste time commuting, etc., etc….
I know this having run my own business for over 20 years supplying to some of the largest Australian and international corporations, in competition with the established consultancy firms.
@ Jay deserves our respect for securing and maintaining headline clients!