Scott McCartney writes about this year’s IdeaWorks study of award inventory, but the underlying work isn’t all that useful and I don’t think Scott does a good job of framing why. Some of Scott’s takeaways are true, others true ‘as far as they go’, and still others quite misleading.
Last May Scott covered the study as well.
The problems that I identified with the study last year remain.
- They’re just querying airline websites. As a result you get pretty big disparity between US Airways (‘the worst’) and Continental (‘in the top half’) even though they are dealing with virtually the exact same award availability as Star Alliance carriers. The US Airways website doesn’t show partner availability (yet), the Continental website does.
- Route selection bias. They handpicked 20 routes per airline, remember these are different routes for each airline. This isn’t like comparing American and United out of Chicago on routes where they have similar numbers of seats offered. Or comparing American and United on the JFK – Los Angeles route. This is picking different routes for each airline, and saying that they are somehow equivalent. Now they’ll say that picking ‘the busiest routes for each’ is somehow equivalent, but then you get…
- The Greyhound Road Rewards problem. Greyhound has a frequent rider program, every 10 bus rides gets you… a free bus ride. In other words, hardly an aspiration award that anyone wants. You may fly Delta from Atlanta to Detroit a lot, but then that doesn’t suggest at all that it’s the route you want to redeem your miels on as a reward for all those trips! Which brings us to…
- Partner flights are not considered. US Airways ir quite rightly quoted in the piece saying that their customers love to redeem their miles on international Star Alliance awards, rather than US Airways domestic flights, and they’re penalized for that in this study. Meanwhile, US Airways also suffers because…
- They’re looking primarily at domestic coach awards. Again, the Greyhound Road Rewards problem, this is a great thing for folks who want another bus trip or who want to earn a free Southwest Airlines ticket as a reward for flying Southwest Airlines (the ‘low cost carriers’ are said to fare especially well in the study). But if that’s not your goal, then the study tells you nothing that is useful to you. Meanwhile, in addition to premium cabin international partner awards, US Airways suffers on its own metal because they tend to have excellent, outstanding award availability in first class on their own flights. But the study isn’t looking at this. Now maybe I don’t want to redeem for US Airways first class from Los Angeles to Charlotte, but it’s sure great when you want to fly from a US Airways city to an international gateway, the first class domestic seat is basically ‘thrown in’ and US Airways can get you there — something that’s frequently not the case with United or Continental (or Delta).
- Calendar bias. They picked fourteen dates between June and October and check those dates during March and April. But different airlines make award seats available at different times, some tend to release seats early when schedules open, others release them much closer to departure. I give Delta Skymiles a really hard time on this blog for their award availability (and that of many of its partners), and rightly so I think, but Delta’s award availability is pretty decent last minute, their revenue management folks tend to hold back releasing award seats until late, very conservative not wanting to give up seats for awards that they might sell. I’d argue they are too conservative and that it speaks poorly of their abilities to predict loads if they really believe they need to do it. But it says little about award availability to check during a specific defined set of dates rather than to search broader sets of dates during a braoder period of time.
I’m also a bit skeptical of the methodology here when they (a) pick GOL as one of the airlines to test and (b) give GOL a 100%. Funny, they don’t include Brazilian carrier TAM in the equation, how did GOL weasle itself in and especially with such an unlikely result?
Now, I suppose this study is useful if it describes how you would actually use your miles:
- You want to fly domestic coach, and on your own airline’s ‘busiest routes’
- You will only look on an airline’s website, you will not do your own research or even call the airline
- You make your bookings in March for summer travel
If that’s the case, perhaps the results are valid, at least as far as they go, but I’m still skeptical about some of the outlier data points like GOL at 100% availability, Southwest at 99.3%, Singapore 90.7%, and Air Canada over 80%. These results almost have to be attributable to biases in the research design noted above.
But for anyone who wants to travel in international premium cabins, on partner airlines, or checks for award space either early or last minute, this study tells you absoltely nothing. And other than ‘someone is trying hard’ I really don’t get the continued news angle here.
Nice analysis ! I am, however, concerned about these “airliens” mentioned in point #6. (sorry- delete me !) 🙂
Ok, ok, typo fixed…
Good analysis!
No way is SWA 99.3%.
Gary,
Do you ever reach out to these author’s directly to dispute their methodology? Do they ever reply to the points you raise? I think it would be interesting to see Scott or Nate Silver or some of these other folk’s responses.
ANY such study that doesn’t conclude Delta being dead last (just above 0%) is a useless piece of garbage.
nice thoughts
perhaps you could offer your own insights as to whose the best and what for.
I agree that international J/F awards are all I care about and all i care is who has them available and how i can find them and get them
best early released inventory
best last minute inventory (?DL according to your last post?)
best use of USair miles (if they are more useful than the article makes them out to be)
any other bests?
I don’t know why you would want to argue with the writer. If these types of stories are written for the masses, wouldn’t it be better to continue to let the masses think that US miles are worthless and WN is great?
In theory, this should drive more people to fly WN, opening up inventory on your preferred airlines, no? If it drives more people to open a WN credit card, your preferred airline may offer better incentives to sign up for their credit card. It goes on and on….
In the words of the great talking head reporter in the movie “Airplane,” I say “Let ’em crash!”
@Andrew well Christopher Elliott has been known to reply in the comments here, Nate Silver’s blog linked to my criicism but didn’t answer it, usually folks just ignore…
I think you are spot on and agree with you that there methodology is very perplexing. Seems like the people designing the study didn’t have a good grasp of what I think most of us want to use our miles for – International Premium Cabins.
As a United Mileage Plus member, I know I will never sit in one of Singapore Airlines next generation First or Business seats – only in economy. And who would choose GOL as their airline to South America?
VVanderlust.
But now isn’t Southwest more like a points based system like the Chase Sapphire card than a frequent flier program, so comparing “Anytime” availability is like just buying a ticket using 1 point per cent. Obviously there is almost always availability.
There’s yet another problem with the methodology: the methodology accepted any single flight on the requested dates as a success. Now, if you’re looking to fly premium class overseas, that may be sufficient, but for those of us who DO use our miles for domestic coach trips, it’s not. If I’m taking a quick last minute weekend trip, there’s often a big difference to me whether my flight is at 6 pm or 6 am.
Whatever the methodology, todays wall street journal had a huge article about the results. Airlines pay attention to this and according to the article Delta specifically changed their availability to improve their Ideawroks results. If the Ideaworks methodology becomes the defacto standard of rating airlines, then airlines will game the system to do well on their survey.
The focus for so long has been on earning miles. Very little attention has been paid to the spending of those same miles, with the noted exception of the frequent flier community. The ability to redeem is just as important as the ability to earn, as the readers of this blog are well aware. The kettles probably have limited experience redeeming and that experience, most likely, hasn’t been good. Hence, McCartney may strike a nerve! The general populace will read and digest the soundbites put forth by McCartney. The airlines, in turn, appear to be responding to those soundbites. Gary so eloquently points out the flaws of the methodology and hence the erroneous conclusions it reaches.
The important point is, regardless of flaws, the airlines WILL RESPOND if enough attention is brought to the issue. Maybe other airlines will open up more inventory as they realize seats will not be filled. Maybe Delta will become less conservative regarding the timing of when they open inventory. Airlines will adapt if enough focus, however flawed, is brought to the issue.
I applaud the study and the focus on the issue. We just need to get to McCartney to help him adjust his study to address its inherent weaknesses.
As I think about this – there are so many variables – what would be a valid sampling method to determine availability and rate rank the carriers? If as some of the posters here suggest – this study is becoming a standard which carriers are responding to – should we not consider developing a more sound methodology and (perhaps collectively) test the carriers and report the results?
This “study” is sad.
Gary’s points are good, but there is more to it. I’m a WN guy and the comment above about the WN/JetBlue programs, etc. being “cash-like” is the key to this.
WN has no blackouts now and you are buying a seat with your points under 2.0, not looking for award availability. Thus, if there is a seat for sale, there is availability.
100%!
They don’t tell you that it might be $500 each way, cashing in at the highest ratio of 12 2.0 points per dollar, on the route you select.
It’s as if UA/AA/US/DL guaranteed you last seat award availability 100% of the time, but it would cost you 300K miles for a domestic RT.
Pretty meaningless, and if this reporter guy does not understand that, he’s useless.
Delta Airlines is always the best for award availability 99 per cent of the time. JD Powers told me so :):):)
So watch out for the other 1 percent
Say no to inaccurate flawed studies eeeeek!
My suggestion to all stockpile Sky Pesos!
Yikes wouldn’t it be wonderful if a really qualified company could crunch the data/analytics correctly
The study also doesn’t take into account the cost of acquiring airline currencies by various means (not just flying). This cost varies drastically. It is quite easy to pile on cheap DL and US miles by using a plethora of promotions, for example, but not nearly as easy with UA/CO, AA, and others.
Gary,
I wanted to address some of your points. I think if you look more closely you’ll see some incorrect assumptions.
The IdeaWorks study looks at the top 10 long-haul routes for each airline, many of them international markets, and the top 10 shorter markets, based on airline seat capacity. I think that’s a very fair way to compare airline availability. You aren’t picking routes where carriers have limited service; you are giving them their best chance to perform — the routes where they have the most seats. The point of the study, after all, is to rank airlines–to compare their availability of awards. Picking the routes where each airline flies the most customers make sense. Picking the standard-level coach award is a fundamentally sound way of comparing programs and measuring how accessible a program is.
You are correct that the IdeaWorks study queries airline Web sites and this puts US Airways at a disadvantage since it doesn’t offer partner awards on the Web site. There may be some top US Airways routes where partners overlap. I think the lack of partner awards available on the Web site is, in fact, an availability issue. US Airways offered other reasons for its admittedly limited availability.
Measuring award availability is difficult, but the IdeaWorks study offers a solid way to fairly compare programs. The study makes no attempt to compare the attractiveness of destinations, the availability of upgrades, etc. It only looks at how airlines behave on their busiest routes on the most basic award. And it has had an impact — Delta felt pressured to increase availability, for example. It’s solid research done by a firm that knows airline FFPs intimately.
–Scott McCartney
@Scott thank you so much for your thoughts!