Infrastructure
VMware vSphere, VMware Horizon, Cerner Millennium through RHO
Problem
Medical staff was suffering from either inconsistent performance or complete inability to access USB peripheral devices inside their Cerner sessions.
Challenge
A large health system with over 15 hospitals and hundreds of clinics across the U.S. provide VDI desktops to their medical staff via VMware Horizon View, and from those VDI sessions the medical staff accesses Cerner Millennium. Cerner Millennium is being delivered by Cerner’s remote hosting option (RHO) which delivers access internally via Citrix. This configuration is how 90% of their staff access Cerner Millennium. This is commonly referred to as a “Double Hop” configuration as there is one session to the on-premises VDI environment and another session to Cerner’s remote hosted environment in Kansas City (Cerner RHO).
Because of the complexity of the environment and the lack of visibility in the hosted Cerner sessions, it is extremely difficult to troubleshoot end-user performance issues. It becomes even more challenging when the issue is with connected peripheral devices where root cause could be within the hosted Cerner instance, on-premises VDI session with VMware Horizon, an issue with the device itself, or even end-user behavior. The medical staff had major inconsistencies with performance when leveraging USB based devices (scanners, e-signature pads, printers, etc.) and would have to log off and back on several times before they had access to the USB peripheral inside of the Cerner sessions. These issues frustrated the staff and caused endless finger-pointing between Cerner and the health system infrastructure team on what technology element was causing the performance issues. There was no clear resolution in sight.
“We have numerous tools that look into specific elements in our environment and each of these tools indicated that there were no issues. Unfortunately, our medical staff told a different story about their inability to logon or experiencing slow session performance times. We felt helpless given our inability to isolate root cause and solve the issues.” – Health System Director of IT
Isolating the Root Cause
The health system decided to deploy Goliath into Cerner RHO to get visibility into the end-user sessions and their overall performance experience while accessing Cerner Millennium applications from the on-premises VDI environment. Cerner’s RHO team and the health system’s IT team were able collaborate together and isolate the affected end-user sessions and drill into the ICA/HDX channel. The ICA/HDX details in Goliath show ICA channel usage to and from the end-user’s endpoint to the user’s virtual machine or session host server. This is a key metric when analyzing the end-user experience as it can indicate if ICA bandwidth usage was actually being used by the scanner and other USB peripherals within the Citrix session being run within Cerner RHO.
Understanding ICA Channels
In order to deliver a session to the end user and retrieve input, Citrix breaks each data type into separate distinct virtual channels (Figure 1). These virtual channels not only show the bandwidth used but also show the direction the bandwidth is going. Evaluating this information effectively gives IT the ability to determine the behavior of the Citrix session based on the bandwidth that is being used in the session.
Figure 1: Shows the available ICA channels in which bandwidth can be detected for any Citrix session.
What this health system was able to find is that for all of the affected sessions, there was no ICA traffic from the VDI to the Cerner servers with regard to the USB peripheral related traffic (TWAIN, COMM, USB etc.), but there was traffic being sent back from Cerner to the VDI (Figure 2).
Figure 2: Shows the bandwidth used inside of the session, the session bandwidth direction is from the Cerner Virtual Apps server perspective meaning that output TWAIN and USB bandwidth is from the Cerner server to the VDI. The presence of output bandwidth and the lack of input bandwidth shows that the issues were not from Cerner because if they were, there would not be any bandwidth present (input or output).
This was objective evidence to show that the issue was not with the Cerner session but actually with USB Device Redirection from the Horizon VDI environment. The on-premises team had to investigate further into the issues and ultimately ended up re-configuring USB device redirection in the environment and were able to confirm with Goliath as well as the medical staff that the issues were resolved.
“We were able to rule out a vast majority of the environment that we were focusing resources into and redirect those resources into the VMware Horizon instance which we now know was where the root cause of the issue existed.” – Health System Systems Engineer
Conclusion
The complexity of this “Double Hop” configuration has historically made isolating where the failure points are extremely challenging, especially in a hosted application like Cerner. Goliath’s purpose-built EHR module for troubleshooting end-user performance issues related to Cerner helped this health system isolate the root cause of the issue by fostering data driven conversations between the on-premises teams and the Cerner team, providing empirical evidence that the issue was not because of Cerner Millennium being delivered by Citrix. The ability to drill down into an end-user performance issue and quickly determine root cause is invaluable to this health system’s IT team who is focusing on delivering high performing applications and services to their medical staff treating patients on a daily basis.