US News is out with its new 2015 ranking of the best frequent flyer programs.

And though their top two picks are plausible you really should ignore this ranking, and I hope no one cites it.

The rest of the results are:
6. Virgin America Elevate
7. Frontier Early Returns
8. United MileagePlus
9. Delta SkyMiles
10. Spirit FREE Spirit

After I critiqued last year’s effort I had long chats with the person behind the rankings about ways to get at the question. I hoped that with the resources they seemed to have behind it, which are far more than I’m well-positioned to muster, they’d be able to come up with a substantially better product this year.

I do think there are improvements that have been made, and while some of the results are plausible I don’t think the methodology supports drawing firm conclusions from those results at all.

What’s Plausible in the Ranking?

Alaska Airlines has an outstanding frequent flyer program. They have great partnerships, and one-way awards that allow for stopovers. Mileage Plan miles are one of my favorite currencies.

American AAdvantage has an excellent program as well. And Spirit Airlines has a very poor one overall.

I found what they liked and didn’t like about American rather odd, though. A notable deficiency is that you can only earn miles for one seat per flight?

Some of the Conclusions are Absurd

There’s no universe in which Frontier Early Returns — which is trying its hardest to become the next Spirit — offers a better program than Delta SkyMiles (let alone United). They charge last minute booking fees for the vast majority of award bookings. They don’t have partners, or mileage upgrades, or a strong route network.

Somehow Frontier rates well because of its elite program, though.

HawaiianMiles is a relatively poor program relative to other US offerings. To say it’s better than United’s strains credulity. For all of my criticisms of United the airline, and for all of my hand wringing over their exorbitant costs for international first class awards, they offer excellent award availability to most destinations in the world via the Star Alliance and — at least for business class (and economy) — at reasonable prices.

JetBlue’s program isn’t better than United’s or Delta’s either. That’s just… weird.

Their narrative doesn’t get anything wrong about Delta though.

What Sort of Methodology Gets You There?

The methodology they’re using remains hackneyed at best.

45% of an airline’s score was “Ease of Earning Free Round-Trip Flight” which had both an earning and a redemption component.

For ease of earning they only looked at miles earned from flying.

For ease of redemption they looked at:

• Only 12 routes for each airline
  
• No redemptions on an airline’s partners
  
• Different routes for each airline they’re ranking, so it’s not apples-to-apples in terms of where a program gets you easily
  
• Searched made only in July 2015 (so the number of months before high season for different destinations serviced by different airlines will be… different)

25% of an airline’s score was “Additional Benefits” which mixes together how easy it is to get elite benefits, whether or not points expire, and what additional redemption options you have for miles beyond free flights.

They built-in a determination that programs with minimum spend requirements for elite status receive a lower score. While that will be a preference for some members, it’s not obviously a factor related to program quality (and remember that both Untied and Delta will let you spend on their credit cards to escape this requirement, though United caps the status you can earn that way to their Platinum level).

10% of an airline’s score was “Network Coverage” and while I do think we can quibble about the relative importance of a program being able to get you anywhere in the world on its planes and those of its partners with miles, this does seem like a legitimate element to include.

10% of an airline’s score was “Award Flight Availability” which seems like an important category. It also appears to be duplicating the ‘ease of redemption’ piece in their first category, since they’re apparently looking at the same 12 routes which differ by airline. They also do not look at partner airline availability as part of this analysis.

5% of an airline’s score was “Number of Daily Flights” so the bigger an airline’s domestic route network, the better it scores in this small contribution to the overall total score. I’m not sure how this matters, since they’re already capturing ‘network coverage’ on the earn side and ‘award flight availability’ which is much more important than whether a flight exists on the redemption side.

5% of an airline’s score was “Airline Quality Rating” which has nothing whatsoever to do with the frequent flyer program, or partner airlines, but rather is a function of the airline‘s “on-time arrivals, lost or mishandled baggage rates, involuntary denied boarding rates and consumer complaint rates.” In other words, it’s an operational score and not even about quality inflight product which one might imagine was a part of the analysis from the category title.

At least Airline Quality – which is separate from the value of the frequent flyer program – is down from 20% of the total in last year’s survey. Indeed, in last year’s survey 60% of the scoring was about the airline and not the loyalty program. So there’s some measure of improvement in this year’s effort.

Update: One Mile at a Time says “ultimately not more right or wrong than any other type of methodology, given that everyone is looking for something else.”

I strongly disagree because the survey is not even measuring what it purports to measure. You can disagree with the relative importance of different factors of a program. That’s why even Delta SkyMiles and Spirit’s FREE Spirit program will have fans! But when you design a ranking of frequent flyer programs based on factors unrelated to the program, and compare award availability based on a small sampling of routes that’s different for each airline, I do not think you can say this is as good as anything else even if there’s not a silver bullet perfect approach.