Concur Support & User Guides

27 January 2023 | 4:10 pm

In the US2 and EU2 Data Centers, logins through SAP Concur website and SAP Concur mobile app were below expected level. Users encountered extreme latency, followed by a "502 Bad Gateway" error when attempting to login and use Concur. Additionally, in the US2 Data Center for a period following resolution of the primary incident, access to the Concur Travel Request service through the SAP Concur website and SAP Concur mobile app was unavailable. Affected users may have encountered the following error after clicking the Travel Request tab, "Sorry but something went wrong with this task." SAP Concur engineering teams have determined the root cause of the incident to be a logical defect in Concur-designed automation to maintain PKI certificate signing chains within network infrastructure used to proxy and process traffic calls from Concur application tiers to external systems. The logical defect was the result of an unexpected interaction with 3rd-party PKI management software run internally by Concur, which permitted a successful certificate renewal whose result was a valid certificate whose expiration exceeded the expiration of dependent certificates within its signing chain. It was believed from a design perspective based on vendor documentation, this result would not be possible; in 8+ years of operation of this tool by Concur there was no past experience of this scenario. Due to the nature of the failure, designed compensating controls in-place within various systems in the Concur environment also did not pre-detect and pre-mitigate the failure. Further, engineering teams were required to quickly re-factor tooling during the progress of the incident to work around the invalid scenario presented by the PKI tool, causing delayed incident resolution. As a result of the above, various internal service tiers began to fail as calls to dependent services failed due to extended certificate chain validation and the presence within the chain of an expired cert, leading to the observed impact. Corrective Actions: The incident was mitigated by engineering a forced-renewal of the expired intermediate certificate and then using the existing automation to re-generate and replace the entire PKI chain on the impacted network devices, restoring services. Teams then immediately validated that no other related systems had pending possible impacts, to prevent re-occurrence while preventive fixes are in progress. Further in the US2 Data Center, a processing tier for the Travel Request function was found to have not self-recovered automatically after the primary issue was mitigated; the incident response team initiated a service restart of that service tier resulting in recovery. Additionally the following fix items have been identified for prevention: - The automation for PKI maintenance which was impacted by the unexpected condition in the 3rd-party PKI solution is being updated to explicitly test and account for the condition that led to this incident, and automatically handle the workflow to renew any invalid intermediate certificates in the renewal chain. The team is additionally in contact with the product vendor to clarify the observed contradictory behavior from the product. - Monitoring code to pre-detect and alert on upcoming PKI expirations within key infrastructure is being updated to explicitly test and alarm on the entire signing chain used for all leaf certificates in use. - A code artifact used by service tiers which handle serving the user interface was found to be incorrectly referenced via external location for this resource, resulting in unexpected impact as a result of the primary failure. The product code is being corrected to reference an internal location for this resource. -The team responsible for the service tier identified to have been unable to self-recover following recovery of the primary failure is developing and will deploy a mitigation to the failure-loop condition which required additional mitigation during the incident. -As a result of observations of the internal engineering teams during the course of this incident, our incident management process has been updated to better facilitate communication between technical teams to improve our overall incident management.

20 January 2023 | 3:35 pm

In the US2 Data Center, logins through SAP Concur website and SAP Concur mobile app were below expected level. Additionally, access to the Concur Travel Request service was unavailable. Affected users may have intermittently encountered extreme latency, followed by a 502 Bad Gateway error when attempting to login. Affected Travel Request users may have encountered the following error after clicking the Travel Request tab, "502 Bad Gateway". The Incident Response Team (IRT) identified an issue within a networking configuration and took action restoring logins. The IRT restarted an application server tier to restore access to the Concur Travel Request service. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.