Personalized SAP Concur Open Updates

Personalized up-to-the-minute service availability and performance information

  1. OPI-6296995 : US2 : Expense | Travel | Invoice | Request | Analysis/Intelligence : Issue Resolved

    We resolved a functional disruption in the US2 Data Center that affected access to Expense, Travel, Request, Invoice and Analysis/Intelligence services. The majority of US2 customers may have experienced error “Sorry, Concur is currently unavailable” when attempting to sign-in, or an error 500 after successfully signing in and clicking Expense, Travel, Request, Invoice or Reporting tabs. Affected users may have also encountered a home page displaying tiles indicating outages for the impacted services. Executing tasks within each service, such as creating an expense report or searching for travel, may have also resulted in various errors. Workaround: No workaround was identified during service disruption. The Incident Response Team (IRT) executed a configuration change on a database server tier to restore service. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

  1. OPI-6296995 : US2 : Expense | Travel | Invoice | Request | Analysis/Intelligence : Issue Status Update

    A functional disruption was identified in the US2 Data Center affecting access to Expense, Travel, Request, Invoice and Analysis/Intelligence services. The majority of US2 customers may have experienced error “Sorry, Concur is currently unavailable” when attempting to sign-in, or an error 500 after successfully signing in and clicking Expense, Travel, Request, Invoice or Reporting tabs. Affected users may have also encountered a home page displaying tiles indicating outages for the impacted services. Executing tasks within each service, such as creating an expense report or searching for travel, may have also resulted in various errors. Workaround: No workaround was identified during service disruption. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

  1. OPI-6296995 : US2 : Expense | Travel | Invoice | Request | Analysis/Intelligence : Issue Identified

    We are working to determine the user impact and will provide information as it becomes available.

  1. OPI-6296866 : US2 : Expense | Travel | Invoice | Request | Imaging | Analysis/Intelligence : Issue Identified and Resolved

    We identified and resolved a service degradation affecting user logins in the US2 data center. All customers hosted on a particular server were affected. During the time of impact, users attempting to sign in using either Single Sign-On (SSO) or a username and password may have encountered the following error message: 'Sorry, something went wrong.' As a result, users were unable to access Concur and would have experienced errors with various services during the downtime. Users that were already logged in during the time of impact were able to continue with their work. Unfortunately, no workaround was available during the period of impact. As the Incident Response Team (IRT) was gathering, the service self recovered. We have verified that service performance is stable and will now resolve this incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

  1. OPI-6287698 : US2 : Travel : Root Cause Analysis

    Impact as Reported: We identified a functional disruption impacting New Concur Travel car searches via the Sabre, Amadeus, and Travelport+ Global Distribution Systems (GDS) in the US2 Data Center. When users entered their search criteria and selected Search, the error message, “An unexpected internal error occurred. Please try again.”, was displayed. The error occurred regardless of the city pairs or travel dates selected and prevented successful car bookings. Air and hotel searches were not affected. Root Cause: The incident was caused by high concurrency against a profile data retrieval process that is executed as part of real-time profile synchronization. Under elevated demand, the process experienced database lock contention on temporary data structures, causing requests to block one another and resulting in a buildup of queued sessions. This degraded response times for dependent services and led to customer-facing failures.Analysis determined that the issue primarily affected requests configured to bypass cached profile data and instead perform a real-time profile refresh before profile retrieval. This design, intended to ensure the most current profile information is returned, significantly increased the number of concurrent executions of the affected database process. The analysis concluded that customer impact occurred because dependent services timed out waiting for profile responses; those profile responses were delayed by blocked database sessions; the blocking was caused by locking behavior within the database process under high concurrency; and the high concurrency was driven by real-time profile synchronization requests. The condition self-resolved as request volume and contention levels decreased. Preventive Actions: The following preventive and corrective actions were identified: Remove locking-prone temporary data structures from the affected database process to eliminate unnecessary contention under high concurrency. Reduce unnecessary executions of the profile retrieval process during synchronization workflows where the data is not required. Decrease reliance on real-time profile synchronization for profile retrieval scenarios that can be served without an immediate refresh. Improve monitoring and alerting for database blocking, lock contention, and concurrency trends to enable earlier detection and response. Incident Mitigation: No immediate corrective action was required during the event, as service recovered automatically once contention subsided. The actions above have been identified to reduce the likelihood and impact of recurrence.