SAP Concur Real Time Status

19 September 2023 | 4:54 pm

In the US2 Data Center, logging in through the SAP Concur website and SAP Concur mobile app was below expected level. Affected users may have experienced an error message “Sorry, Concur is currently unavailable.” when attempting to login. The engineering team has determined that a process to deploy load-balanced worker nodes in stages to an internal API gateway service did not allow the newly-deployed nodes to fully come online before accepting traffic, causing these nodes to enter a failed state while serving connections. This resulted in a cascading failure as load shifted within the gateway service, causing core services handling authentication functions to return errors. Service alarms triggered a rollback action, recovering service. Corrective Actions: The gateway service team will implement changes to the node deployment process which will pre-warm the application services running on new instances to ensure they are ready to accept traffic without failure.

19 September 2023 | 4:40 pm

In the US2 Data Center, accessing Concur Services through the SAP Concur website and SAP Concur mobile app was below expected level. Affected users may have found tabs were missing or were unavailable after login. The engineering team has determined that the cause of the incident was a code performance issue impacting a core database system which serves as the reference datastore for user profiles. The performance issue was latent (not newly introduced) and triggered by heavy system load. The issue resulted in extremely poor query performance for user profile data within core system services, resulting in application errors after authentication and missing product feature tabs enabled via profile attributes. After determining the issue could not be resolved via online resource tuning and removing blocking application connections, the incident team restarted the database engine, which resulted in recovery. Corrective Actions: -Via log and trace analysis following the incident, the database vendor recommended adjustments to table statistics sampling configuration to optimize query plan performance for the impacted data tables involved in the incident. This change has been implemented and verified. -Three separate code optimizations have been identified for performance improvement based on traces collected during the incident. These will be deployed to the system once fully tested and validated. -Additional database vendor-supplied diagnostics have been deployed to the environment to assist database. administrators in further data collection for analysis. -The operational playbook for recovery of this core database system is under review based on learnings from this incident and will be updated to optimize system recovery time.

19 September 2023 | 1:28 am

In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have received the following error message when attempting to log in, "Sorry, Concur is currently unavailable". Additionally, errors or latency may have been experienced while attempting to perform tasks. As a workaround, users experiencing errors may be able to continue successfully by re-trying. The Incident Response Team (IRT) worked with network vendor during the investigation. The network vendor Team acknowledged and confirmed incident resolution. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

18 September 2023 | 7:05 pm

In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have received the following error message when attempting to login, "Sorry, Concur is currently unavailable". Additionally, errors or latency may have been experienced while attempting to perform tasks. As a workaround, users experiencing errors may be able to continue successfully by re-trying. Services have returned to normal operating levels. The Incident Response Team (IRT) will remain engaged with the vendor, to ensure the stability of this incident. Once verified as stable with the vendor, a communication reflecting this verification with the vendor will be sent, and the incident will be resolved.

18 September 2023 | 5:59 pm

In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have received the following error message when attempting to login, "Sorry, Concur is currently unavailable". Additionally, errors or latency may have been experienced while attempting to perform tasks. As a workaround, multiple attempts to perform tasks may have provided successful results. We have observed a new disruption to our services during the monitoring period. The Incident Response Team (IRT) has engaged additional for investigation. Our next scheduled update will be in one hour or sooner, should a significant change occur.