Delta’s operations suffered a significant IT lapse today. Ultimately hundreds of flights were cancelled and many more than that were delayed inconveniencing travelers across the globe.
On the one hand, this is hardly a new phenomenon. Technology programs have led to major issues for all of the 4 largest US airlines over the past 13 months.
- Two and a half weeks ago Southwest was forced to dump significant portions of its schedule for three days.
- American had its operations comprised by IT issues last September.
- And United’s operations melted down last July after IT issues. And that came after United suffered computer outages the month prior, not to mention in February 2014, November 2012, and August 2012. And of course United’s computer system integration in March 2012 created operational chaos as agents unfamiliar with the new systems had difficulty with missing reservations and telephone hold times backed up for hours.
On the other hand, Delta’s core selling proposition is its operational reliability. Delta filed to trademark the phrase ‘the on-time machine’ a claim American Airlines used to make in its late 1980s advertising.
Delta went so far as to stop interlining with American rather than accepting each others’ distressed passengers at industry rates — because Delta is so reliable they don’t need another airline’s network, and don’t want to help out a competitor. They continued to interline with United, under an agreement that has United paying more.
Even Delta’s early explanation reeked of hubris. They blamed a ‘power outage’ which many took to suggest was an external event, outside of their control. Several readers (and some media that contacted me) wondered ‘why an Atlanta power outage would cause this, and wouldn’t Delta have backup power?’
It turns out that the explanation, while technically true (their systems lost power) was misleading as Georgia Power made clear on Twitter.
Delta should be commended for getting their systems back up — first to get passengers checked in, and then to get flights out — quickly considering that the incident occurred around 2:30am Eastern time. They brought their operation back to the skies and did so safely.
But not only weren’t they candid about the problem, they stuck it to their customers with their approach to a travel waiver.
They allowed passengers who were supposed to travel today and today only to reschedule flights without penalty or additional fare only as long as they’re willing to fly this week.
- There will be residual effects of today’s outage felt in Delta’s operation tomorrow, as some aircraft remain out of place and as crew rest requirements as a result of tonight’s delayed flights push other flights to leave late tomorrow. But customers traveling on Tuesday get no reprieve to reschedule their trips.
- Even though having some folks traveling tomorrow push their trips to the future would give Delta extra seats needed to get stranded passengers from today’s outage to their destinations more quickly. (There’s little slack in a system running at about 85% capacity already.)
- Business travelers starting their week on a Monday can’t just start on Thursday or Friday. Meetings need to be rescheduled, and even leaving on Tuesday means losing of day of their week. A reasonable waiver would give them a do-over next week or the following.
Customers who were supposed to travel today can still have change fees waived if rebooking for travel beyond Friday however difference in fare will still be collected.
Anyone whose flights were delayed today should consult the benefits of the credit card used to purchase their tickets.
- Most premium cards and certainly most rewards cards offer coverage
- This will usually give you up to $500 towards the cost of hotel, meals, and related expenses you incur as a result of a flight delay
- Most cards require you to pay the full cost of a ticket on a card to qualify for coverage. Some (like Citi Prestige and Chase Sapphire Preferred) only require partial payment — which means award tickets are clearly covered, albeit only up to the amount charged.
- The length of delay required will vary as well, from a 4 hour delay [Citi Prestige] to an overnight delay or a 12 hour delay.
Ultimately while Delta’s CEO recorded an apology video and the airline updated its flight cancellation stats online throughout the day, it should have been more forthcoming with its customers about what happened (not started the day deflecting blame) and should have been more flexible with its customers as well (a travel waiver that extended to customers scheduled to fly on Tuesday, with a longer period in which to rebook travel).
Update: Monday night Delta extended their travel waiver to passengers scheduled to fly on Tuesday, still requiring that they fly by Friday to avoid paying an increase in fare.
$200 vouchers for pax with 3+
HR delay or cancelation. http://news.delta.com/delta-offers-compensation-customers-affected-systemwide-outage
I think it should be noted that NONE of the American Express Cards, including the Platinum, Premier Rewards Gold, Delta, etc. carry trip interruption or delay protection as included offerings. You must pay EXTRA to have trip cancellation or delay coverage by purchasing their premium protection option. I found this out recently (the hard way) and now will make sure I never put airfare on an AMEX ever again.
Pretty sure the Amex SPG card carries just the type of protection you’re talking about…
@Wayner Unfortunately it does not. Just like every other AMEX card, it has Travel Accident Insurance (In case you hurt yourself) and Global Assist (In case you need HELP while traveling, but that is at your expense), but not actual delay or cancellation protection.
Prestige kicks in after 3 hour delay.
Also, Chase Sapphire will cover up to $500 even if you just charged taxes for an award ticket to the card if delay is 12+ hours or overnight.
Delta is being more than a bit tight fisted with this. Georgia Power reminds us that ONLY Delta was affected and the failure was due to their own equipment. Delta’s rescheduling and accommodation permit is tight and is worthless if PAX cannot make contact. No response to tweets, emails and phone calls. I have to wonder if Delta is not trying to turn their own screw up into a minor, very short profit, at the expense of their customers. If Delta had half a brain, they would be reaching out to the affected customers and proactively offering solutions. Do I really care? No. For entirely unrelated reasons, I no longer fly Delta unless there is NO other choice. Currently, they are even worse that the long history horribly miserable service that apparently began over at United some years ago. At least for U.S. domestic travel, three of the four legacy majors simply do not get it and the majority of the international long hauls and not a lot better. At least on international routes, we generally have more options. As Lilly Tomlin (Earnestine) used to say, referring to the telephone company, “We don’t care; we don’t have to!)”
Typical of Delta was to always blame someone else for their problems. Well, it looks like the Power company already destroyed that argument.
It’s clearly the fault of the Middle East airlines. Isn’t that obvious?
Delta CEOs deflecting blame: It’s what they do.
Stuck in Rome at the airport two extra days on what was to be only a change of planes. Utter chaos at the airport with little or no explanations or directions. A rumor here, a half answer from a representative there (when you could find one). “Take the shuttle to a hotel”. Where’s the hotel? Which shuttle? Where do we find it? Help lines that went unanswered at $2 a minute. Delta Counters closed at the airport.
It was about 8 hours until we got thru to Delta and rebooked.
Why should Delta allow passengers who were scheduled to travel on a day with no known interruptions to rebook their flights at no charge?
DOESNT EVER LEAVE THE AIRPORT
DRIVING EVERY LOYAL TRAVELER AWAY
@Christian – because Delta says it still needs to cancel 100+ flights on Tuesday (and possibly beyond) and that’s entirely unfair to those affected pax.
DL interlining with UA saved me arse from what could have been a disastrous situation and am really glad this DL IT glitch happened on Monday!!!!
On Saturday I was booked to do a mileage run to MEL (where I am now) to earn the 20K miles that I needed in order to make million miler. I was to do:
LGA-DEN-LAX-MEL
We left LGA on time and arrived at DEN, only to get a text message that my flight to LAX was delayed and it would not arrive at LAX until after my LAX to MEL flight had departed, so I needed to see UA CS at DEN for available options. UA had no flights that would get me to LAX on time for the MEL flight. American? Zilch. DL? Yes!!!
I was booked on the DL flight (first time in over a decade!) and even managed to get a seat in Y Comfort when I told the DL agent that my flight on UA was in first class (an upgrade). He said he had no first available but he could do Y Comfort. Kool.
I made it to LAX, boarded the UA B787-900 in BF for MEL, where I am now enjoying my complimentary suite upgrade at Hilton Melbourne South Wharf.
Back in the US on Sunday as a UA 1MM and *A Gold for life 😉
Cheers, mates!
Delta’s explanation yesterday smelled fishy. It would take something on the order of a bomb or major earthquake (in a non-earthquake area) to impact properly hardened IT systems (or the internet backbone in a region). If Delta had proper infrastructure, a power failure should have triggered a UPS with at least 6 hours of runtime and a (set of) backup generator(s) capable of running for at least 2 days with onsite fuel, with fuel contracts guaranteeing additional fuel out to a week in emergencies. And even in a small datacenter, multiple redundant UPSes and generators would all have to simultaneously fail to trigger an outage — multiple primary systems, multiple backup systems. Serious datacenters are actually designed to survive nearby explosions, even. And usually attached to multiple municipal power grids as well.
Really, a power outage is just not something that can happen in modern IT outside of a major catastrophic event. This just reeks of “we deferred maintenance on critical systems to save money and lost that bet badly”. Looking forward to reading the real story when it comes out.
And for that matter, Delta’s IT footprint is absolutely big enough to build in geodiversity — why didn’t their redundant systems running in SLC or MSP or DTW kick in and take over, eh? 😉
Didn’t Gary coin the term “SkyPesos?”
@Nik +10000000000000000000000
@civet That’s fine for those passengers that do get affected, when they get affected. Gary sounded like Delta should have offered it upfront to all customers on Monday and Tuesday, which seems to make no sense.
Here’s what appears to be an actual answer, via the Minnesota Star Tribune:
The airline’s computer systems failed around 1:30 a.m. Central time, Delta said, because of a “power outage in Atlanta.” Georgia Power, the electric company that supports Delta, said something went wrong with the airline’s switchgear, which can be likened to a fuse box.
Delta concluded by the end of the day that “following the power loss, some critical systems and network equipment didn’t switch over to Delta’s backup systems.”
So, yeah, this is a failure to test and validate resilience and disaster recovery plans. Also known as a failure to spend money. I’d expect the CIO or CTO’s resignation over this one. This is beyond amateur hour — absolutely inexcusable for a $10MM/year business; let alone one of Delta’s scale.
Further, the switchgear should not be a single point of failure. So if it really was, that’s an equally astonishing system design failure. And 6 hours of downtime for an outage like this? Did they have to go fabricate a new switch?
Easy for an armchair analyst to sit back and tell DL what they should have done. Meanwhile, how many airlines have you actually worked for? What is your experience in disaster recovery & business continuity?
@Josh G: I’ve taken multiple businesses through many Cat 3, Cat 4 hurricanes, fires, lightning strikes, and tornadoes. Hurricane flooding out the entire 1st floor of a building (including power & communications rooms), local ILECs completely DOA after a hurricane, UPS melts down (literally), flames shooting out of the rack? Been there, done that for all of that and more — the show went on like nothing was wrong. It can be done, and it can be done on the cheap. My setups are kludgy and ugly to outsiders, but the abuse they can take is unreal.
So… given that… Yes, Delta screwed up. BIG TIME. Where was their redundancy? Where are their monitoring systems? Where are their IT staff at that hour? Where are their alternate data centers, if any? Why aren’t their systems segmented/partitioned into smaller sub-systems which can exist solo for a period of time? I can’t help but think they never ran through what-if scenarios.
Reminds me of the data centers in NJ that flooded out during Hurricane Sandy when the floor drains started flowing backward. What did they expect the drains to do when water was rising? Why did they even have them in there to begin with? Same for a few hospitals who put generators in basements.
@Josh G – what specifically do you disagree with here, you think they should have lied and blamed the power company? Easy for you to say “Delta knows what it’s doing” but I am making substantive and specific arguments and you aren’t.
@JoshG I operated a small datacenter for 8 years on a shoestring budget, including participating in the build. I consulted on BC/DR planning and design for a $1B company. We actually lost our automatic transfer switch at one point. The show went on.
But that’s beside the point. This is really basic stuff. And it has nothing to do with being an airline — IT is IT, and keeping the power on has nothing whatsoever to do with the loads and services running on your systems.