Waking up to a P1 incident! Users are not able to see any icons after they authenticate to Citrix Store Front/Web Interface. Enumeration is failing.
Contents
The environment
- Citrix StoreFront/Web Interface hosted on Windows 2008 R2
- Zone Data Collectors of XenApp 6.5 Farm hosted on Windows 2008 R2 Servers.
- Each ZDC has an issued certificate which is showing a valid path and is correctly configured in the Citrix SSL Relay Configuration Tool
- XML Service Transport is set to SSL relay
What has changed?
An intermediate certificate renewal has taken place overnight. However, when you look at certificates which are depending on the new certificate, they show valid certification path and no errors.
What’s the error?
Looking at the Storefront Event Logs; the error we see associated with users authenticating Looking like this
Event ID: 30029
Source: Citrix Web Interface
Site path: C:\inetpub\wwwroot\Citrix\XenApp.
An SSL connection could not be established: You have not chosen to trust the issuer of the server’s security certificate,<CertificateName>.
This message was reported from the Citrix XML Service at address .
The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services.
Troubleshooting
- A workaround was put in place to use HTTP transport protocol instead of SSL Relay while we troubleshoot the issue.
- The issue was reproducible, but what made it hard to resolve is the fact that the certificate chain on each of the ZDCs was showing valid path and no errors, however, the handshake between the StoreFront Server and the ZDC/XML service was failing.
- Wireshark captures on both sides showed that the connection was failing right after the initial “Hello” handshake. There was no clear error message which could indicate the reason behind this failure.
- When comparing the new intermediate certificate with the old one, we noticed that the new certificate has a different Signature Algorithm which doesn’t match the signature of the rest of the chain. The new certificate had RSASSA-PSS instead of matching the existing SHA265RSA. The Hash Algorithm was matching though.
- Although using mixed algorithm is supported in general, the client (in this case the Citrix Web Interface Server) didn’t know/understand the new algorithm.
Solution and How to Prevent in The Future
Re-issuing the intermediate certificate with the correct matching signature algorithm resolved the problem.
I noticed that we could have found out what was wrong sooner if we were able to see the properties and the attributes of each certificate in the chain in a formatted table. The way Windown MMC Certificates Snap-in shows the certificates in the chain doesn’t allow for easy comparison, hence, I have written a PowerShell script that can look for a certificate you specify and get all the certificates in the chain of trust and allow you to display them in a table for comparison. The Script can be found here Get and Enumerate Certificate Chains Remotely Using PowerShell
References
https://tools.ietf.org/html/rfc5280