Personalized SAP Concur Open Updates

Personalized up-to-the-minute service availability and performance information

  1. OPI-6047043 : US2 | EU2 | APJ1 : Travel : Root Cause Analysis

    In the US2, EU2, and APJ1 Data Centers, completing travel bookings for air and hotel in the New Concur Travel using the Travelport GDS service through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have experienced an issue not returning any result when doing air and hotel search. Root cause: Our partner Travelport has confirmed the issue was caused by an expired certificate on their end and will provide SAP Concur with and RCA and their identified preventive actions. Corrective Actions: N/A

  1. OPI-6038696 : US2 : Expense | Travel | Invoice | Request | Imaging | Analysis/Intelligence : Root Cause Analysis

    Impact as Reported: In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app are below expected levels. Affected users may experience an error, "Sorry something went wrong" when attempting to login. Root cause: A code defect triggered by the selection of specific localization options resulted in an unexpected increase in the payload size for data stored in a cache used by a Next Gen Travel backend service. While the condition persisted the network capacity to the caching layer became saturated and triggered traffic throttling, with a cascading latency impact that eventually resulted in the disruption of search and booking functionalities. Due to a monitoring gap the network saturation was not initially detected by the incident response team, delaying the identification of the root cause of the problem while the investigation was focusing on issues on other parts of the system which were eventually found to be a result of the degradation on the caching layer. Once the actual problem was identified the caching layer was flushed to remove the oversized data sets and the application tier recycled and scaled up to restore service. Corrective Actions The following corrective and preventative actions have been identified: Monitoring and alerting has been enhanced to full visibility of the caching layer, including network metrics The caching infrastructure is undergoing a sharding process to increase resiliency to unexpected abnormal data loads. Data throttling is being implemented at the application layer to reduce the possibility and mitigate the impact of similar code defects. The containers scaling configuration for the service in question is being tuned to better respond to scenarios where the service needs to be restarted under load. The code defect is being remediated.

  1. OPI-6043545 : US2 | EU2 | APJ1 : Travel : Root Cause Analysis

    Impact as Reported: In the US2, EU2, and APJ1 Data Centers, completing travel bookings for air, car, and hotel in the new Concur Travel experience using the Sabre, Amadeus and Travelport GDS services via the SAP Concur website and mobile app were performing below expected levels. Affected users may have seen search results; however, the results are being returned from the Concur Travel instead of the new Concur Travel, which may impact booking accuracy and availability. Root cause: A defect introduced in the rules deployment pipeline for the Concur Feature Toggle service resulted in the rules data to be corrupted, leading to routing issues for users of new Concur Travel. Once the issue was identified by the incident response team a fix was applied to the impacted pipeline and the rules were redeployed to fix the data corruption and restore service. Corrective Actions: The following corrective and preventive actions have been identified: The validation procedure for deployment pipeline changes is being reviewed and revised to address the testing gap the allowed the defect not to be detected in the pre-release environment. The release process is being revised to allow phased deployment by environment and reduce the potential impact for escaped defect. The service architecture is being reviewed to identify opportunities to decouple functionalities to simplify maintenance and testing.

  1. OPI-6048626 : US2 : Travel : Issue Resolved

    In the US2 Data Center, travel bookings for air, car, and hotel services in the New Concur Travel experience via the SAP Concur website and mobile app were performing below expected levels. Affected users may have experienced errors when attempting to book travel in the New Concur Travel. The Incident Response Team (IRT) performed a system task on web services and temporarily paused an automated background process to help restore service. The team is also working to implement measures to help prevent this issue from recurring. We have verified that service performance is stable and will now resolve this incident. An investigation into the root cause is underway, and a root cause analysis report will be provided once it is complete.

  1. OPI-6048626 : US2 : Travel : Issue Status Update

    In the US2 Data Center, travel bookings for air, car, and hotel services in the New Concur Travel experience via the SAP Concur website and mobile app were performing below expected levels. Affected users may have experienced errors when attempting to book travel in the New Concur Travel. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.