Concur & TMC Clients In Recovery Mode After Major Service Outage
Concur and the users of its Compleat travel fulfillment system are in recovery mode after a service outage that lasted more than 24 hours and resulted in a massive queue of millions of unprocessed PNRs.
Compleat is billed as “a customizable travel fulfillment system that automates PNR processes, eliminating manual tasks so agents can get back to serving customers.” It counts many major TMCs among its customers.
The problems began early Wednesday, and as of 8 p.m. Thursday night, the Concur Open Current Status Dashboard was still showing partial outages for all of its services through the North American data centre, including: expense; travel; invoice; request; imaging; analysis/intelligence and mobile. As of 5 a.m. this morning, Concur was reporting normal functioning of the product.
“At approximately 12:30 US central time, the faulty switch hardware was replaced, testing was completed and the Compleat service was restored,” said Concur E.V.P. Supplier & TMC Services Mike Koetting in a statement to customers.
“There is a considerable backlog of transaction processing and Concur will be closely monitoring performance. After the transaction volume has returned to normal and we are confident of system performance our attention will turn to a forensic analysis of the outage and go-forward improvement actions.”
Concur’s Twitter feed was a long list of apologies yesterday, telling users it was aware of problems and working on a fix. “On behalf of Concur, allow me to apologize for the disruption to your business,” Koetting wrote to users. “Historically Compleat has delivered very high service levels; we’ve never experienced an outage even approaching this severity or duration. Although unprecedented, we recognize that any unplanned service interruption is unacceptable and will be working hard to regain your confidence.”
The outage created a buzz at the GBTA Canada Conference in Toronto this week, but despite Open Jaw’s efforts, no one wanted to speak on the record about the situation.
The staff at Concur’s booth at GBTA Canada said they were not aware of any outage. As well, Concur’s PR department told Open Jaw yesterday that it was not aware of any outage.
Open Jaw contacted Nancy Tudorache, Director Operations – Canada for GBTA. Her response:
“Thank you for reaching out to me. I won't be able to comment on this issue at this time. “
Koetting said the outage “appears to have been caused by a simultaneous hardware failure of the primary and backup data switches at the data center where Compleat is hosted.”
There were problems in advance of the full system outage. The service status dashboard shows that a full outage was listed for the expense functionality 09APR, as well as partial outages on 17, 18 and 20APR.
Concur has been under fire from customers of late for other service issues. As Business Travel News reported 28MAR, “customer criticism has ranged from ‘terrible’ product to confusing interface.”
At the recent Concur Fusion conference, executives of the company founded by CEO Steve Singh heard difficult feedback, the BTN story said. "Sometimes it's a little challenging to look in the mirror when you're telling us how we're doing," Concur President Elena Donio said about criticism of the company.
The company says it grew its employee base by 30% in the past year, with 1/3 of those new hires working in service and support.
"Our support team has grown significantly over the last year, which has allowed our employees to specialize in specific areas of functionality so you have more educated service, more in-depth service faster and with a more positive consistent experience," Donio said.