Deploying Citrix StoreFront for 100,000 users

Citrix

I’ve previously been involved in several large projects to deploy Citrix StoreFront for a companies with a global presence for 100,000 users or more. Across these projects I gathered quite a few notes from the field to use when designing StoreFront services for a large number of users.

This blog post will outline the design decisions as well as the features we enabled and disabled to provide the required experience. I will try and condense the detail down to blog post friendly size but it still might end up being a bit of a read.

Current Environment

Our assumptions around an environment utilise the following example details:

60 XenApp/XenDesktop farms and sites with varying versions
Global user bases with datacentres across many continents
A desired support for around 500 login actions per minute per server from users during high usage or failure scenario’s.

Design
The StoreFront design layout itself is fairly run of the mill for anyone used to it the only difference being theres a heck of a lot of it. The theoretical company already utilised other none Citrix load balancing devices (for the purpose the of the blog lets take a common assumption that they are using F5 type features) so we had to use those for load balancing duties along with various new NetScalers for the AGEE features. We load balanced XML for the XenApp and XenDesktop simplifying and reducing the required StoreFront farm configurations.

Across the globe there are 7 locations where the StoreFront servers would reside.

2x Europe locations
1x America’s location
3x Asia locations
1x African location

Server virtual hardware sizing

For the virtual hardware design we took the leading practices from Citrix and made two standard hardware specifications based upon the expected usage for the server group.

The small server was given 2vCPU’s and 8GB of RAM which during normal operation would support 500 logins per minute or 30,000 logins per hour.
While the large server was allocated 8vCPU’s and 16GB’s of RAM which would support around 1,300 logins per minute or 80,000 logins per hour.

Based upon the anticipated BAU logins plus failover from F5 GTM if required we required 26 StoreFront servers across the globe and a number of server groups as outlined below.

Now this is of course dependant on the host hardware being performant but during our load testing we comfortably got the small servers up to around 2,000 logins per minute before the stress began to show which for us was a great result.

In regards to the subscription database, Citrix recommends allocating a minimum of 5MB disk space plus 8MB for every 1000 application subscriptions. Now these numbers are not something you would normally even consider but due to the number of possible users this suddenly becomes a concern.

Now your mileage may vary but we calculated an average of 10 application subscriptions per user which in our largest server group which we anticipated around 58,000 staff this works out as the following:-

58,000 staff x 10 applications = 580,000 subscriptions

580,000 subscriptions / 1000 = 580 8MB chunks

580 chunks x 8MB = 4.640GB

Allocating an additional 4.640GB per virtual machine for the subscription store is not a huge amount in the scheme of things but still worth considering when designing for scale.

Server Groups
To service the users across the globe we utilised several server groups all the server groups were built with a minimum of an N+1 configuration for expected server load and were backed up with NetScalers and F5 GSLB to provide a single url with the desired failover and location awareness features.

The server groups at there largest (which was in the Europe) had 6 large servers within the group and the smallest were out in Asia with 2 small servers within the group.

Subscription Synchronisation
However there were two locations near Asia requiring a separate server group for each location but the staff could log into either one during a normal working day as they had a localised F5 GTM GSLB requirement so this presented a different challenge.

Users logging into lets say Asia1 server group would subscribe to Word and Excel and then logout, but during their next login they might actually hit the Asia2 server group which did not share the same application subscription database so users would have to subscribe to their applications again there as well.

Citrix StoreFront however does allow you to sync the database of an identically named store with one in the in another server group. The article HERE describes how to configure this but it basically boils down to making sure port 808 is open between the servers in the two groups, adding in computer names to the CitrixSubscriptionSyncUsers group in each server and then running a couple of powershell commands.

it is quite important to note that during the initial database sync between server groups the entire database is replicated, however subsequent sync’s are delta files so while the initial sync has a danger of being a good bit of traffic going down the wire that will be it for painful data transfer.

XenApp/XenDesktop enumeration

So theres no getting away from it 60 separate sites and farms is an eye watering amount of infrastructure to try and amalgamate behind StoreFront. The possible load it would place during logon would normally be quite excessive as each time a person logs on the delivery controllers for each farm would be queried for applications available to the user.

In this instance to resolve and ease the issue we had to utilise two customisations.

User Farm Mapping
XML brokering

By default User Farm Mapping is meant to direct users to particular farms based on user groups and act either as primary and failover type configuration or as a method of presenting a single icon for duplicate applications across farms and sites.

We used UFM for neither of those purposes really just purely for the directing users based on group memberships with a very light smattering of aggregation. This had the advantage of fantastically reducing the query traffic sent to the sites and farms as only the farms the user has a group membership linked to is queried during a logon. (easily verified utilising wireshark or similar tool).

I’ve already written a quite extensive blog post on how to set up and configure user farm mapping which you can view HERE so I’m not going to repeat the basics. However the basic framework of the configuration is as below.

<userFarmMapping name="UserMapping1">
 <groups>
 <group name="group-hk" sid="*************" />
 </groups>
 <equivalentFarmSets>
 <equivalentFarmSet name="Farm1" loadBalanceMode="LoadBalanced" aggregationGroup="None">
 <primaryFarmRefs>
 <farm name="XenApp" />
 </primaryFarmRefs>
 <backupFarmRefs>
 </backupFarmRefs>
 </equivalentFarmSet>
 </equivalentFarmSets>
 </userFarmMapping>
<userFarmMapping name="UserMapping2">
 <groups>
 <group name="group-hk" sid="*************" />
 </groups>
 <equivalentFarmSets>
 <equivalentFarmSet name="Farm2" loadBalanceMode="LoadBalanced" aggregationGroup="None">
 <primaryFarmRefs>
 <farm name="XenDesktop" />
 </primaryFarmRefs>
 <backupFarmRefs>
 </backupFarmRefs>
 </equivalentFarmSet>
 </equivalentFarmSets>
 </userFarmMapping>
<userFarmMapping name="UserMapping3">
 <groups>
 <group name="group-Europe" sid="*************" />
 </groups>
 <equivalentFarmSets>
 <equivalentFarmSet name="Farm3" loadBalanceMode="LoadBalanced" aggregationGroup="None">
 <primaryFarmRefs>
 <farm name="XenApp-Eurpe" />
 </primaryFarmRefs>
 <backupFarmRefs>
 </backupFarmRefs>
 </equivalentFarmSet>
 </equivalentFarmSets>
 </userFarmMapping>

So with this snippet of code looking through UserMapping1-3 if I was a user of group-hk when I logged in mapping 1 and 2 would apply and queries to those farms would be sent but not to the farm in mapping 3, as I’m not a member of group-UK.

Its worth noting that within our design from the old infrastructure upto 5 user groups would apply to some of the farms so within our StoreFront configuration we had around 100 separate user farm mapping code blocks, which during testing there was no noticeable performance issues or errors due to the extremely extensive UFM configuration.

StoreFront Server domain mismatch and XML brokering
Another complication of the design is that as the solution was global the StoreFront servers sat in a different domain to those of the majority of the users logging in and backend XenApp/Xendesktop servers.

There is an issue/feature which presents itself when using SSO from within receiver. When logging in with SSO from a desktop your group memberships are gathered from your endpoint and sent through to IIS on StoreFront which is in Domain1 for example. Now if your user groups that you have utilised for user farm mapping or just even application access are domain local groups in Domain2 (where the user account and XA/XD servers are) these are not visible to the StoreFront servers IIS instance which results in no apps being displayed.

This does not occur using standard browser access as the authentication method is different.
In order to resolve this issue  the Citrix article outlined here which changes the enumeration of the groups to SID’s from IIS to the Citrix XML service is required.

Automation is key (sort of)
Now for everyone’s sanity during these example project’s we tried to automate the build process as much as possible however it was troubled slightly by some less than desirable powershell cmdlet’s. As a consultant I make extensive use of the Microsoft Deployment Toolkit (never leave home without it) so we already had the basic StoreFront install in the bag however what it didn’t do was set up the stores or additional farms so I fleshed out the script to include all the farms that we needed to add along with the initial configuration.
The basic install of Citrix StoreFront is the below batch script.

@ECHO OFF
REM Set variables
SET SOURCE=%~dp0
SET SOURCE=%SOURCE:~0,-1%
IF NOT DEFINED LOGDIR SET
LOGDIR=%SystemRoot%\TEMP
ECHO – Installing Citrix StoreFront
START /WAIT CitrixStoreFront-x64.exe
-silent
ECHO – Disable IIS Logging to reduce
excessive logging (enable for troubleshooting)
%windir%\system32\inetsrv\appcmd.exe set
config /section:httpLogging /dontLog:True

Then we switched over to powershell for the rest of the configuration which could be invoked from the batch file by using “powershell.exe -exec bypass -file c:\script\location”.

# Set variables
# SET SOURCE=%~dp0
# SET SOURCE=%SOURCE:~0,-1%
# IF NOT DEFINED LOGDIR SET LOGDIR=%SystemRoot%\TEMP

Echo – Importing Citrix Powershell Modules

cd ‘c:\program files\citrix\receiver storefront\scripts’
./importmodules.ps1

# ./c:\program files\Citrix\Receiver\Scripts\importmodules.ps1

ECHO – Configuring Citrix StoreFront

set-dsinitialconfiguration -hostbaseurl https://LB.company.com -farmname “farm1” -port 80 -transporttype HTTP -sslrelayport 443 -servers @(“xenapp01.company.com”) -loadbalance $false -farmtype XenDesktop -storevirtualpath /Citrix/Company -webreceivervirtualpath /Citrix/CompanyWeb

ECHO – Configuring additional Farms

set-dsxdfarm -farmname “farm2” -servers xenapp02.company.com -port 80 -transporttype HTTP

The set-dsxdfarm option can be replicated as many times as you need to script the addition of the XenDesktop sites. This command however has a MASSIVE issue in StoreFront 3.0.1, there is no way to specify what type of delivery controller you are configuring so running the above command will insert every item in as a XenDesktop controller rather than allowing you to chose XenApp or VIAB (lol). Really hope that gets sorted soon.

As I created the entries for the script using excel and formula’s it was still way quicker and less error prone to do it this way and then go into the c:\Inetpub\wwwroot\Citrix\Store\Web.conf and alter the farm types manually for the XenApp farms than it was to do the whole thing manually.

What in actual fact we ended up doing was copy the entire farm configuration ( signified by the <farmsets></farmsets> tags)from a completed server group and then paste them into the first server within a server group once the initial install and configuration had been completed. This could then be propagated to the other servers within the farm and configured 60 farms within StoreFront by a simple copy and paste action, not exactly slick but it was quick and easy and repeatable!

Prettification!

Within the environments there were two major use cases for StoreFront one being standard fat client access generally through external or internal access and the second was for thin client access. I say thin client but in actual fact most were running a highly locked down version of Windows on a standard desktop but it did the job. We created two stores for these use cases so each store could be modified to suite the use case.

For the thin clients the client wanted as little option as possible for customisation of clicky stuff so in order to do that we configured the following:-

Full Screen launch

Disable session reconnect

Disabled Subscriptions

Configured SSO (single sign on)

Additionally the GUI of Storefront was highly customised to match the clients look and feel, now this I’ll freely admit was fairly new to me at the start of the project but boy did I pick up a lot in this project. I’m still not a css/js code junky but with Chrome developer mode/notepad and luck on my side I know enough to get by. I’m in the process of creating a blog post that explains a bit more on how you can change the UI of Receiver for Web so hopefully someone will find that useful in the future.

Full Screen Launch

Desktops have the ability to launch within a Window which in a thin client use case is often not desired. You can ensure that the Desktops are always launched in full screen mode by editing the Default.ICA file located in c:\inetpub\wwwroot\StoreName\App_Data\default.ica. Then within the Application section of the file insert the below line.

DesktopViewer-ForceFullScreenStartup=true

Disable Session Reconnect

By default, when staff authenticate to a StoreFront instance any disconnected applications or desktops attempt a reconnect option allowing staff to continue where they left off from before. Within the example clients environment this is undesired so must be disabled. This can be configured within the web.conf file located in c:\inetpub\wwwroot\Citrix\StoreName\.

Now search the config file for the allowSessionReconnect=”true” statement and change to false.

Disable Subscriptions 

A major part of the StoreFront feature base is the ability to subscribe to your available applications which add’s them to the Favourites folder (also see THIS post on correcting the Americanism’s within StoreFront), for the thin client store the client did not want this. Happily though this is simple and requires no text hacking about in web.conf’s and the like. All you need to do is open up the StoreFront GUI and select the correct store. Then on the right hand window select Disable User Subscriptions.

subs

Configuring Single Sign On 

Within the thick client use case SSO is pretty much required for every client let alone this one. This required a bit of a major upheaval though as their desktops were still utilising the now deprecated Online plugin version 12 and when you think about the number of staff etc we are touching with this configuration a simple client upgrade suddenly becomes a bit of a task luckily though that is down to the internal IT department to sort out!  We advised them that an upgrade would be in order and  configured single sign on for a group of test machines in order for the thin clients to login effectively with their local windows credentials. This of course was affected by the XML Broker change we had to make (as mentioned above) in order for remote domain local group memberships to be enumerated correctly on the endpoints. Configuring Single Sign On has not really changed that much in recent history and there are plenty of step by steps to configure that if needs be (See here for the process which talks about it along side Desktop Lock).

Little Critters!

During our design and deployment phase we did come across a bug which menaced us utilising user farm mapping. We worked with Citrix engineering who provided us a hot fix for it. The bug in question presents a Cannot Complete Your Request error when a user logs in to a StoreFront store that has user farm mapping configured for certain circumstances. Additionally within the StoreFronts event viewer a log is created that states “an item with the same key has already been added”

This came about with one or two of the technical users within the pilot phase because they were assigned twice to a static delivery group within a single XenDesktop site. Where normally without user farm mapping the user would be presented with two desktops with the same name (except with the 1/2 monitor ) user farm mapping had a bug where it could not parse the duplicate entries it saw within the responses from the XenDesktop brokers. There is a Bug filed for StoreFront 3.0.1 within Citrix for this (BUG0622730) so if you come across this then refer to that bug and hopefully they’ll send you the private hot fix. I understood that they were looking to implement this in latter versions of StoreFront but I’ve not had the time to test if this has been resolved in StoreFront 3.5.

Summary

I think I’ll leave that here for this post I’ve tried to include the major points of interest from the project’s in delivering Citrix StoreFront to a large number of user and by and large it was a simple process with the exception of creating the user farm mapping configuration that was extraordinarily tedious but worked very well. Some elements of user farm mapping are now including in the GUI however this will only really suit aggregation or failover of resources for something more complicated you’ll still need to start crafting text configuration and its worth noting that as soon as you do that the StoreFront GUI locks out the ability to manage the controllers, no doubt fearing that someone will change something in the GUI that will mess up the ufm configuration royally.

I was pleasantly surprised how well the virtual hardware withstood the increase until failure test’s which pushed the logon’s per second way past the sizing guidelines given by Citrix and that the extensive user farm mapping did not cause any delay in login actions, however I will be watching the powershell and automation options for StoreFront closely as that did cause us some undesired manual steps which would have been far better if automation was an option.

This blog post isn’t meant to be any sort of design guide for anyone rather than some pointers for the major decisions we made during the design and deployment although this by no means outlines all of the design just some of the more interesting elements. Anyway thanks for reading and as always comments welcome.

Author: Dale Scriven

4 thoughts on “Deploying Citrix StoreFront for 100,000 users

  1. Excellent article. Thanks for your time 🙂 . I cant wait to read your next blog about how you can change the UI of Receiver for Web 🙂 . thanks once again . keep writing

  2. Thanks for your time for writing this great blog! I am very curious to your blog about customizing Receiver for Web.

  3. Dale, I can confirm that the bug ‘BUG0622730’ around duplicate responses for aggregated resources has been fixed in the currently shipping version of storefront.

Leave a Reply to dudley delderfieldCancel reply