Each year, for five years now, IdeaWorks has published a study (.pdf) on which airlines offer the best award availability and each year Scott McCartney picks it up approvingly in the Wall Street Journal without a recognition of its flaws.
Roughly speaking this year’s piece finds that:
- Overall airline award availability is getting better despite an improving economy (more seats sold).
- Low cost carriers are better for flyers looking for award seats than legacy airlines.
- International airlines are better at offering award seats than US ones.
But the only way it reaches these conclusions is by assuming away virtually everything we know about how these programs actually work.
Nowhere in the study do they account for the value or quality of what the low cost carriers like Southwest Airlines are getting you for your miles.
And while airlines like Singapore do have excellent award availability for members of their own Krisflyer program, there’s no comparison of award chart pricing (Singapore’s chart is more expensive for comparable long haul awards) and no comparison of out of pocket cash costs — like many Asian and European airlines they add fuel surcharges onto award tickets that add several hundred dollars to cost. Just because a seat is available more often doesn’t mean it’s a better value.
More fundamentally, though — and I wrote about the flaws in their study last year and the year before, too (and the year before that) — they don’t seem to make their flawed study materially better, perhaps because it gets reported uncritically as-is.
The results are dead wrong, because the methodology is dead wrong. In fact, consumers will be worse off if they pay any attention to it.
Here’s how they go about their study:
Booking queries for a party of two travelers were made at frequent flier program websites during March 2014. Some airlines require a Saturday night stay for reward travel; all of the queries used date pairings that included a Saturday night stay. While the city pairs varied for each frequent flier program, the travel dates did not. 280 specific dates were selected for survey queries and only reward seat availability for travel on the date specified was recorded; any departure time was acceptable. Furthermore, reward travel had
to be available on the outbound and return dates queried. Overly circuitous routings with long
elapsed travel times and layovers longer than 4 hours were not accepted.Survey results reflect the availability of saver-style rewards (capacity controlled seats) with two exceptions. For Southwest, Anytime or Wanna Get Away rewards priced up to 25,000 points (roundtrip) qualified as reward travel. For JetBlue, rewards priced at 25,000 points (roundtrip) qualified as reward travel.
The top 10 routes (based upon total seats offered for sale during a 12-month period) longer than 2,500 miles and the top 10 medium-haul routes (251 to 2,500 miles) were selected for each airline. Alaska Airlines was switched to this methodology for 2013 due to the carrier’s increased emphasis on the Hawaii market. Due to a lack of long-haul routes, the top 20 overall routes were queried for
these airlines: Air Asia, GOL, JetBlue, Southwest, and Virgin Australia. Ten top Europe – Palma de Mallorca city pairs (out of 20) were substituted for airberlin to reflect the carrier’s major Mediterranean emphasis on holiday flights.
(Emphasis mine.)
This seems really problematic, as though it will yield strange results (which it does, as we’ll see in a moment):
- They searched airline websites only. So even though US Airways and United had access in March to the same award space (more or less), United offers online booking of many partners while US Airways does not. Miles in each airline’s programs could access the exact same saver award seats.
- They’re searching different routes for each airline but over the same dates, which ignores the effects of high and low seasons. Since they’re looking at fixed months and days prior to departure for each airline’s most popular routes, airlines whose routes fall into high season during that date range are disadvantaged.
- They’re making subjective judgments about ‘overly circuitous routes’ but not about departure time, consistently offering 6am flights or redeyes counts just as much as offering times many consumers would find more desirable.
- They’re substituting routes. airberlin wins top honors in the study but they’ve chosen to cherry pick routes to compare instead of using the same methodology as applied to other airlines. Low cost carriers were fewer long haul routes get to count award availability on more short haul routes instead, and unsurprisingly low cost carriers fare well.
- They count saver award space only, except when they don’t. Revenue-based programs let you redeem for any seat with points, just requiring more points to do it on expensive flights. They count saver space only for legacy airlines, but count Anytime seats for Southwest (not just ‘Wanna Get Away’ fares). They cap the value of those awards at 25,000 points — not recognizing that 25,000 points with Southwest or or JetBlue is different than 25,000 points with a legacy carrier.
Their methodology also:
- Ignores cost of acquiring the miles. It may be really easy to earn miles with an airline that has several partners and bonuses, so it could even make sense to spend twice as many points. But any seat availability for extra points doesn’t count, except where it does (in favored programs that do well in the survey).
- Ignores the value of a given redemption. Greyhound Road Rewards may give you a free bus trip every 10 trips, and if those bus seats aren’t capacity controlled then they satisfy their riders every time. But that doesn’t make Greyhound Road Rewards a more lucrative, rewarding, satisfying program than United MileagePlus or American AAdvantage which allow you to see the world, in a premium cabin no less (a travel style many would never be able to afford but for the points, but the study looks at coach only).
The conclusion that the low cost carriers with points-based programs where those points are good for any seat is misleading. The methodology focuses only on short haul routes for those carriers, while focusing on long haul routes for other airlines. And since they don’t face the same capacity controls their availability (“100%” for Southwest!) is compared against saver awards only at the legacy carriers.
And the conclusion that non-US airlines are better — while possibly true — is not supported by the approaches taken in the study.
The study — which focuses on online redemption and suggests the most valuable points are those used as currency to buy tickets at market prices — is sponsored by Switchfly which sells online services to frequent flyer programs with a focus on using points as currency.
To be sure, if you want domestic coach redemptions,, then you might well be attracted to Southwest et al. But then that implies other things as well – that you shouldn’t have a points-earning credit card, for instance, since a 2% cash back card will suit you best.
Ultimately, though, the study tries to prove too much with a methodology that doesn’t come close to supporting its conclusions. They go looking for strong value in low cost revenue-based frequent flyer programs and find it, without regard to what ‘value’ for the consumer really is.
Folks following the advice of this study will find themselves with tickets to Florida and Ohio, while the legacy frequent flyer programs that score less well continue to offer greater options to their members.
- You can join the 40,000+ people who see these deals and analysis every day — sign up to receive posts by email (just one e-mail per day) or subscribe to the RSS feed. It’s free. You can also follow me on Twitter for the latest deals. Don’t miss out!
Oh, man, not this crappy study again. How can these people and the ones who praise them stay in business?
This study totally blows, they have a vested interest in advising the airlines on how to structure their frequent flier programs. One has to assume Scott McCartney has some sort of vested interest in this study as well.
And they most likely are searching for the cherished 25,000 mile saver economy award which is one of the poorest mileage redemptions in terms of value. The person whom redeems for a saver 25K coach award instead of buying a ticket at $300, effectively values those miles at 1.2 cents per mile, so a cash back earnings card would be a better value in many cases. Redemptions for domestic saver F awards, especially to Hawaii, or saver international Business or First class rewards are a much better value for redemptions considering their actual retail cost.
One of the first rules in research or designing an experiment to make sure your sampling is random so as not to be “biased”.
That said, why only sample during March? What happens in April, May, June or on other months? What can we conclude between the month-to-month differences?
Same nonsense every year. They may as well evaluate FFP’s to see which is best to get a toaster via points.
Shhhhh! Let them mislead the masses.Reduces competition for CX F to HKG.
Stop whining and let Professor Emeritus Gary Leff Ph.D. bless us with a peer reviewed study. For only 100 Chase affiliate links you have a graduate student for a year that can do the ground work.
Why don’t you start a Kickstarter to fund a better study?
@chitownflyer Even less than 1.2 a mile, because you have the loopback effect of miles earned while flying revenue. Assuming a 5,000 mi transcon, it’s more like 1.0 cents per mile.
This study is better than no study at all, which is the alternative offered by this blog. It has limitations but no structural flaws, and agree with many of its premises, such has having a companion and using standard awards (not inflated double or more ones).
I think this study is far more interesting to the average American than the focus on international premium class redemptions that seems to be the focus here!
@hillrider – is it better, actually? it offers quite a misleading picture for the reasons explained above, while I frequently offer which airlines and routes provide the best availability for all sorts of traveler types to all sorts of destinations.
I don’t need no stinkin’ study to tell me that United miles are the most useful to have if you actually need to get somewhere on an award ticket. The availability is vastly superior to DL and AA these days.
I am warming to WN’s program though — especially after I got my 2 credit cards that netted me lots of points and a free companion pass. For domestic USA travel, it’s a good alternative to finding often-scarce award seats from the legacy carriers.
I think as a point and mile traveller, it is beneficial to have points in multiple programs.
We LOVE Southwest in our family. They have the flights that we need and they are rather family friendly. We also have companion passes so instead of having to pay for 3 tickets, we only have to pay for 2 (soon paying for 2 and getting 4 seats). It’s a huge benefit for us.
Obviously, their points are pretty useless if we want to do international travel (although they are opening more options after the AirTran merger) but that is why we have points and miles in other programs.
I think in general, if you want to have freedom to do a lot of travel, it’s best to put your eggs in multiple baskets.
Right, Erica — using “all” the programs is best (although I think one can safely avoid Skypesos unless one is located in a DL hub city). And if you live near a US/AA hub, Avios is a remarkably useful program (although you still have to contend with US/AA’s unsatisfactory award availability).
This of course suggests that the most useful bank for frequent flyers is Chase, since they feed UA/WN/BA. Ultimate Rewards gives you the flexibility to allocate as necessary.
This study has flaws. But it’s still a more accurate picture of what it looks at than the Freddie or Flyertalk Awards, both of which have been promoted/defended in your blog.
@kokonutz in what way?
This study makes very specific claims, that the methodology doesn’t come close to supporting and that even leads to clearly misleading outcomes.
The Freddies does not. There is one category that’s comparable, Best Redemption Ability. And that recognizes the program ranked highest by those frequent travelers expressing an opinion on the given program. In other words, it does what it says.
I have never written about the Flyertalk Awards to the best of my knowledge.
“*Low cost carriers are better for flyers looking for award seats than legacy airlines.
*International airlines are better at offering award seats than US ones.”
Are those claims really inaccurate? I have my doubts.
I actually don’t say the claims are wrong, I’m not evaluation the conclusions, I’m pointing out that the study in no way supports the conclusions.
This study doesn’t help us at all to know whether the claims are true or false.
Gary,
With regard to
“Their methodology also:
Ignores cost of acquiring the miles. …..
Ignores the value of a given redemption.”
It should ignore those two (which DL SkyPeso loyalists routinely bring up), especially as long as the points/miles from average Aunt Mildred’s flights are mostly earned at not much better than a rate around 1 point/mile (or less) per mile flown and where credit cards are mainly giving her around the same range of airline points/miles per credit-card-charged dollar as others.
Gary, how do you suggest a better study be conducted? I know what I would require for a better study: new government regulation of the programs such that public disclosure requirements are expanded.
@GUWonder – what sort of data are you looking to require disclosure on?
Actually seems like something of a ‘big data’ project that shouldn’t be hard to design with the right computing resources and research assistance (admittedly more than I’ve got as ‘just a guy’ but something a consulting firm with a corporate sponsor should be able to manage).
I lay out the sort of issues in the post above that plague this study’s design, but none of them are insurmountable, they just require a lot more work and data.
My point is that the conclusions of the study seem reasonable to me. The conclusions implied by the Freddies do not.
Do you agree with the Freddies that AA is program of the year? Because that’s what AA is telling me: it is the best program, and the Freddie award proves it. Low information flyers are being screwed if they fall for that ‘finding.’
In this case, the methodology may not be state of the art, but no one will be screwed by following the advice implicit in the findings.
Phone sampling is impossible because agents are inconsistent. And Delta doesn’t have provide direct access to Air France inventory information by means besides its buggy website, so these will always be ‘flawed.’
The study has had a good real outcome. Forcing Delta to address availability issues in a public forum.
@kokonutz Absolutely, American was the best North American program in the year leading up to the Freddies. For sure. Freddies voting ended March 31. Folks are obviously displeased with changes implemented April 8, but any study including this one is a snapshot in time not a guarantee of future performance. I am not sure the April 8 changes materially change which program is best, which is to say that even post-April 8 American AAdvantage has not become worse than United’s MileagePlus or Delta Skymiles.
I think there’s an argument to be made for Alaska Airlines Mileage Plan but it’s a stretch. Certainly not Aeroplan and not Rapid Rewards IMHO.
Oh, and it’s not just frequent fliers who vote in the Freddies. Contestant programs spam ALL of their customers begging for votes. Pa Kettle’s vote based on the one time he was on a plane 2 years ago counts the same as Randy Petersen’s.
People who live in those sorts of methodological glass houses…
Ps, how come you’ve never written about the FT awards? :p
I don’t see how “Pa Kettle” with a mileage account doesn’t have a stake in the program and shouldn’t have a voice.
Again, the difference here is that this study is staking out specific claims that the methods do not at all support.
I don’t know much about the Flyertalk Awards. This one seemed worth writing about because of the regular WSJ pickup, and I thought it deserved some attention.
Turn your methodological analysis on the Freddies with the same intensity you do this study.
Pa kettle doesn’t know squadoosh about the relative merits of a frequency program. Who cares what he thinks, as reflected in his Freddie vote!? Yet his vote carries the same weight as mine, someone with experience with several elite programs in the same year. The reliability of the data is squat as it is based on mostly ignorant opinion rather than hard data and analysis.
In any case, as I say, I feel better about telling folks to follow the advice in this study than I do telling them to use the Freddies as a guide to programs.
My advice these days is to burn AA miles ASAP and if your goal is free seats do what the study says. If your goal is aspirational travel, focus on UA. At least until the aa merger shakes out. The devil you know (and Smisek is a devil indeed) is better than the devil you don’t. AA has proven that it’s willing to make changes without notice or warning. I just don’t trust them right now. 😉
You don’t care what members generally think and that’s cool. Freddies provides an answer to a question you’re not interested in. That’s very different than making a claim wholly unsupported by the data.
If your goal is first class award travel on partners there’s no way that UA is more attractive than AA. But I agree, burn AA miles now rather than holding them. Hopefully UA miles already burned prior to February.
More generally earn and burn within the same general timeframe/under the same redemption chart. That’s been my advice for the last 11 years and remains so today.
Even invalid or unsound studies can provide a valid indication for outcomes related to the matter studied — and in large part that shows up when there is confusion that has people confusing correlation for causation.
While the study has its flaws, it’s better than nothing and the concluding indicators seem to largely align with what I’ve experienced and what I’ve been getting from the airlines’ own gnomes.
This is sort of like the classical notion of theories/models in a scientific methodology regime: the theory/model that better explains the results is better than the non-existent theory/model which explains nothing.
Lol. Let’s ask 10,000 laymen which rocket fuel mixture is best. It’ll answer the question of what people think is in rocket fuel…but be of no practical use whatsoever in launching a rocket!!!
It takes quite a double standard to deride this study yet defend the Freddies methodology.
But your earn and burn strategy is (as ever) sound. ^
“It takes quite a double standard to deride this study yet defend the Freddies methodology. ”
How is a popular vote a methodology? It’s not, it’s a popularity contest.
@kokonutz. Absolutely AAdvantage is the best North American program. Yes, IS, as in present tense. Is it a good program? No it isn’t, but that’s not the point. The point is that MileagePlus does not hold a candle to AA in terms of the value, even though it has a better availability in some instances (but not all).
But I’m sure after AA really devalues, you “will be” right. 🙂
Listen to the WSJ goof on the study (which he seems to have been involved in, somehow):
http://podcast.mktw.net/wsj/audio/20140508/pod-wsjtmit/pod-wsjtmit.mp3
About the 26:00 mark.
The most troubling statistic of the survey is the Avianca percentages……….now that you are really sold on using them it would be nice to see you address those numbers
@JustSaying it’s meaningless, the survey looks at availability for their airline’s own top routes only. (I’ve actually found availability on the various Avianca carriers to be quite good and just booked an award on them a couple of days ago, but as a general matter using their miles is for premium cabin international travel on partners so this study is wholly irrelevant.)
Ok….thanks for talking me off the cliff as I just got approved for his/her Avianca cards……..
Re-reading the comments, it’s telling that even those who seem to want to criticize this post won’t speak up for and defend the methodology of the study.