Contents
Summary
I have been working recently on building a new Citrix Xendesktop site, with RDSH being Windows Server 2016 Standard, and Ivanti (Appsense) as a Workspace Manager, and Citrix PVS 7.15 LTSR for image provisioning. In this article, I am discussing a problem we faced with slow logon and logoff time when compared with Out-of-the-Box (fresh) installation. In this case, Windows Server 2016 hangs for minutes on “Please wait for the user profile service”
Environment
Windows Server 2016 Standard.
Ivanti User Workspace Manager 10.1 (FR3).
Citrix XenDesktop 7.15 LTSR CU1.
VMWare 5.5 Build 4180647.
VMWare Tools with VMware NSX File Introspection filter driver (vsepflt) – Formerly known as VMWare Vshield.
Problem
User logons would get stuck at “Waiting for the user profile service” for up to two minutes followed by a black screen for up to 1 minute before loading the Desktop icons.
Similar behavior is also seen upon logoff.
Event Logs don’t show any signs of errors or warnings that are relevant to this behavior.
Solution
Trying to decipher the problem, I started by disabling Appsense services on the RDSH Windows Server 2016 which we are launching the sessions on
[code language=”powershell”]
Get-Service *appsens* | Stop-Service -Force
[/code]
Subsequent logons went straight in without any issues, so I disabled services one at the time and found that disabling “AppSense Application Manager Agent” alone would stop the problem.
So, I moved my doubts to the AM config itself, so I tested using an empty AM (Application Manager) config and that partially resolved the issue: The “waiting for the user profiles service” stage was not hanging anymore but the desktop would still take up to minute to load while waiting on a black screen.
Ivanti support suggested to try and remove the VMware vsepflt filter driver. We were able to confirm that the issue is indeed caused by vsepflt by restoring the regular AM config we use and unloading the driver from the target RDSH server in question
- From an elevated command prompt run: fltmc
- Confirm that vsepflt is in the list
- To unload the driver run: fltmc unload vsepflt
- Attempt to recreate the issue.
After doing the above the problem was solved, but that would technically leave us without an Antivirus solution, so doing that was not an option in our case.
In case you don’t need VMware vsepflt filter driver in your environment
If VMware NSX File Introspection filter driver (vsepflt) is not being used in your environment then this could be a no brainer solution for you. All is needed is to uninstall this componenet from the VM VMware tools
You may refer to VMWare Documentation to automate removing the component
If uninstalling the driver is not practical in your environment, you could prevent it from loading by following the below steps:
- Click Start > Run, type regedit, and press Enter. The Registry Editor window opens.
- Navigate to:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\vsepflt - Locate the StartDWORD.
- Right-click Start and click Modify.
- Change the value to 4 and click OK.
- Click File > Exit to close the Registry Editor.
In case you DO need VMware vsepflt filter driver in your environment
The alternative workaround is to disable “Change file’s ownership when it is overwritten or renamed “ feature in Ivanti Application Manager configuration.
Click on Trusted Owners in your AM Config
Un-tick “change a file’s ownership when it is overwritten or renamed
Save the config and deploy it to endpoints servers.
Doing that resolved the logon/logoff wait problem and the whole login process is now finishing in under 15 seconds.
Although this workaround could potentially cause a security risk, the risk was minimum in the environment we are working on as users don’t have rights to overwrite any binaries.
Root Cause
The root cause according to Ivanti KB is that vsepflt.sys is not handeling further I/O requests when file rename is pending. This is causing a deadlock which delays the logon and logoff process.