FIM Optimization & Performance Best Practices

To start out, follow your virtualization platform’s best practices. Best practices generally result in better performance and stability. The following links will guide you to best practices recommendations:

• Performance Best Practices for VMware vSphere™ 5.5 (Performance Best Practices for VMware vSphere 5.5)
• Performance Best Practices for VMware vSphere™ 5.0 (Performance Best Practices for VMware vSphere™ 5.0)
• Windows Server 2012 Hyper-V Best Practices (In Easy Checklist Form) (Windows Server 2012 Hyper-V Best Practices)

• Integration services should be installed on each VM Guest
• Integration services should be the correct version for the hypervisor

• Over-committing virtual processors (assigning more processor cores to the guest than are can be made consistently available due to other process usage) can create latency in fulfilling requests in performance systems. For instance, having four (4) virtual machines with four (4) processors when there are only eight (8) cores.

If you need to over-commit processors, please read the next item (CPU Ready Time) for how to detect a stalled condition.

• Review the CPU Ready Time (VMWare) or Virtual Processor\CPU Wait Time Per Dispatch (Hyper-V) on a regular basis to ensure that the solution is not stalled waiting for processor time
  • This post provides the following guidance on the meaning of the CPU Ready Time counter:

• • The Hyper-V Equivalent to VMware’s CPU Ready Time (Kehayias, Hyper-V Equivalent to VMware CPU Ready Time, 2013

• The following articles provide additional information on the subject and are good reads:
  • CPU Over-commitment and its Impact on SQL Server Performance on VMware (Klee, 2012)
  • SQL SERVER – vCPUs – How Many Are Too Many CPU for SQL Server Virtualization ? – Notes from the Field #003 (SQL SERVER – vCPUs – How Many Are Too Many CPU for SQL Server Virtualization? – Notes from the Field #003)

• Overcommitting memory is a useful virtualization capability when used properly. Overcommitting should be avoided in situations where VM guests routinely have extended periods where they need the RAM for application purposes. However, SQL server is an application that is likely to use memory for extended periods of time and may experience performance penalties due to swapping.

Properly configuring the antivirus client can dramatically reduce the time that resources are locked while real-time scanning is occurring. Use the following configuration information to improve performance by reducing wait times for locked resources:

The following recommendations for configuring the antivirus client for the Hyper-V role are taken from WINDOWS ANTIVIRUS EXCLUSION RECOMMENDATIONS (Helmick, 2014).

• Vmms.exe
• Vmwp.exe

• AllVHD,VHDX,AVHD,VSV and ISO files
  • *.vhd
  • *.vhdx
  • *.avhd
  • *.vsv
  • *.iso

• Default virtual machine configuration directory: C:\ProgramData\Microsoft\Windows\Hyper-V
• Custom virtual machine configuration directories
• Default virtual hard disk drive directory: C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks
• Custom virtual hard disk drive directories
• Custom replication data directories, if you are using Hyper-V Replica
• Snapshot directories
• C:\Clusterstorage and all subdirectories (if using Live Migration together with Cluster Shared Volumes)

Note: The paths shown below are default paths and may have been modified in your installation.

The following recommendations for configuring the antivirus client for the SQL application are taken from WINDOWS ANTIVIRUS EXCLUSION RECOMMENDATIONS (Helmick, 2014).

• %ProgramFiles%\Microsoft SQL Server\MSSQL11.<Instance Name>\MSSQL\Binn\SQLServr.exe

• %ProgramFiles%\Microsoft SQL Server\MSRS11.<Instance Name>\Reporting Services\ReportServer\Bin\ReportingServicesService.exe

• %ProgramFiles%\Microsoft SQL Server\MSAS11.<Instance Name>\OLAP\Bin\MSMDSrv.exe

In the event you are using a Windows cluster, the following best practices should be used (Antivirus software that is not cluster-aware may cause problems with Cluster Services, 2012)

• Antivirus software must be cluster aware
• Exclude the following locations:
  • Q:\ (Quorum drive)
  • C:\Windows\Cluster
  • Temp directory of the SQL cluster’s service account

• SQL Server data files
  • *.mdf
  • *.ldf
  • *.ndf

• SQL Server backup files
  • *.bak
  • *.trn

• Full-Text catalog files locations: (File Locations for Default and Named Instances ofSQL Server)
  • Default instance: Program Files\Microsoft SQL Server\MSSQL\FTDATA
  • Named instance: Program Files\Microsoft SQL Server\MSSQL$instancename\FTDATA

• Trace files
  • *.trc – these files can be generated either when you configure profiler tracing manually or when you enable C2 auditing for the server.

• SQL audit files (forSQL Server 2008 or later versions)
  • *.sqlaudit

• SQL query files
  • *.sql

• Analysis Services data
  • Default location is C:\Program Files\Microsoft SQL Server\MSSQL.X\OLAP\Data.

• Analysis Services temporary files
  • Default location is C:\Program Files\Microsoft SQL Server\MSSQL.X\OLAP\Data.

• Analysis Services backup files
  • Default location is C:\Program Files\Microsoft SQL Server\MSSQL.X\OLAP\Backup

• Analysis Services log files
  • Default location is C:\Program Files\Microsoft SQL Server\MSSQL.X\OLAP\Log

• Filestream data files (SQL 2008 and later versions)
• Remote Blob Storage files (SQL 2008 and later versions)
• Reporting Services temporary files and Logs (RSTempFiles and LogFiles)

Note: The paths shown above are default paths and may have been modified in your installation.

The following recommendations for configuring the antivirus client for the IIS role are taken from WINDOWS ANTIVIRUS EXCLUSION RECOMMENDATIONS (Helmick, 2014).

• %systemroot%\system32\inetsrv\w3wp.exe
• %systemroot%\SysWOW64\inetsrv\w3wp.exe

• IIS compression directory: %SystemDrive%\inetpub\temp\IIS Temporary Compressed Files

The following recommendations for configuring the antivirus client for the SharePoint application are taken from WINDOWS ANTIVIRUS EXCLUSION RECOMMENDATIONS (Helmick, 2014).

• Drive:\Program Files\Common Files\Microsoft Shared\Web Server Extensions
• If you do not want to exclude the whole Web Server Extensions folder fromantivirus scanning, you can exclude only the following two folders:
  • * Drive:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\Logs
  • * Drive:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\Data\Applications

• Note: If you use a specific account forSharePoint services or application pools identities, you may also have to exclude the following folders:
  • * Drive:\Users\ServiceAccount\AppData\Local\Temp
  • * Drive:\Users\Default\AppData\Local\Temp

The following directory exclusions can be used to minimize the scanning of solution files during FIM’s processing:

• Exclude the FIM Binaries: C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\
• Exclude any clients needed for management agents:
  • Notes Client: E:\Notes
  • Oracle Binaries: E:\Oracle

• Exclude the FIM Binaries: C:\Program Files\Microsoft Forefront Identity Manager\2010\Service

The proper configuration and optimization of the SQL Server is critical for achieving the best possible performance of FIM. If performance problems are occurring, I would validate the SQL configuration first. The items listed below are not in any particular tuning priority order. Each item should be considered when validating the SQL server’s configuration.

• Follow the best practices mentioned in the tuning of the SQL server’s antivirus client listed in the previous section.

• All disk partitions should be validated for disk alignment offset issues prior to performing any installs. This is especially true for all partitions created by a Windows OS prior to Windows 2008. Disk Partition Alignment Best Practices for SQL Server (May & Lee, 2009) offers prescriptive guidance for how to detect and fix this issue. Even though this is written for SQL 2008, the guidance still applies for newer versions of SQL.
• Underneath logical disk partitions are performance impacting items such as RAID block size and physical hard drive sector size. Guidance varies between RAID vendors and hard drive manufacturers. Check with the appropriate vendor sources to ensure that the physical implementation of your logical disk space is optimized.

• Disk I/O per second (IOPs) should be carefully thought through and properly engineered. Do not assume that the SAN offers unlimited IO or that the SAN’s configuration is ideal for what the design requires.
• The underlying disk subsystem and RAID levels are extremely important in optimizing disk performance. This statement is particularly true when using physical servers with locally attached storage. SAN vendors typically provide tuning guidance for the SAN configuration to optimally support applications such as SQL server.
• A SQL server has different IO usage patterns. These usage patterns can benefit from using different performance tiers of disks. SQL SERVER – Virtualized SQL Server Performance and Storage System – Notes from the Field #013  suggests separating SQL components onto different virtual disks with different storage characteristics. (e.g. SQL database needs fast disk, whereas SQL Backup doesn’t need the fast disk but it needs lots of disk space).
• The following counters are useful in detecting diskIO issues:
  • Logical Disk – Avg. Disk Queue Length > 1 indicates a sustained backlog of IO requests
  • Logical Disk – Split I/O – useful in detecting fragmentation
  • Logical Disk – %Disk Time >= 80%

• Defragment both the Virtual Host and the Virtual Guest’s disks to reduce fragmentation
• Use a commercial disk defragmentation utility that can defragment large files. The default OS defragmentation software is not capable of performing large file defragmentation (> 64MB) which can leave large files such as a database spread across non-contiguous space.
• Physical Disk (Virtual Host)
  • Physical disks can become fragmented when large files such as Virtual Hard Drives (VHDs) are expanded.
  • Pre-size the virtual disk files to their intended size. Do not allow the VHDs to auto-grow.
  • Defragment the physical volumes if the virtual hard drives are resized to maintain contiguous disk usage.
  • Do not place virtual disks on system partitions

• Virtual Disk
  • Schedule defragmentation during off hours when backups are not occurring.
  • Limit the number of virtual disks for simultaneous defragmentation to reduce IO load on disk subsystem.
  • Defragment the volumes prior to install, after install, and after pre-sizing the database
  • Avoid defrag on VM Guests that have a Snapshot. Defragging can make the snapshot grow.
  • Page files should be placed onto their own virtual disk to reduce system partition fragmentation

• Database file fragmentation
  • Data base files are typically larger files that are placed on physical or logical drive volumes and cannot be easily defragmented using the operating system defragmentation tools. If these files grow, it is possible for the databases themselves to be split into non-contiguous disk space thereby slowing performance. If this has happened, it is recommended that a commercial grade defragmentation software be used to reduce the fragmentation. To reduce the occurrence of fragmentation caused by database growth, see “Pre-sizing the databases” for more information.

• Pre-size the FIM databases with the initial size from lab environment. Pre-sizing the database to the expected size reduces future disk fragmentation when other files exist on the same volume. For instance, if there are two databases on the same volume, when one database grows it may need to utilize non-contiguous space for the growth. Pre-sizing allows for the databases to reach their intended size without needing to expand into non-contiguous space.
• Ideally a pre-sized database shouldn’t grow, but events often conspire that prevent us from staying within the boundaries set during the initial sizing. It is recommended that the DBA allow growth by larger blocks (500 MB) to avoid constantly growing the database during processing cycles. This practice also helps to reduce disk fragmentation as well.
• The following commands can be used to expand the size of the databases & log files as well as change the growth patterns:

• Consult with the DBA as to the appropriate size and number of Temp DB files that should be added to the solution. A general rule of thumb is one Temp DB per processor core provided that the disk subsystem can efficiently support the IO requests of multiple files. The idea is to maximize the throughput to the Temp DB to increase the performance of the application.
• Temp DBs should be placed on a fast disk as it is heavily used by the FIM Service
• Standardize the size of the default Temp DB to the size of the additional temp DBs that will be created. Remember to size according to typical size and growth patterns.

• Add additional Temp DBs per the DBA’s recommendation.

• In a SQL cluster, use Local Disks for Temp DBs (new feature in SQL 2012)

SQL fragmentation is when indexes and catalogs become heavily fragmented. SQL uses this to boost the performance of queries. To keep SQL performing optimally, it is good practice to rebuild indexes and full text catalogs on a routine basis.

• Index fragmentation
  • Create a weekly maintenance plan to Rebuild Indexes and Update Statistics for all User databases.

• Full Text Catalog fragmentation
  • Schedule the Full Text Catalog for reorganization during off peak hours. The following article provides guidance on how to set this up: How to Automate the Rebuilding of FIM Full Text Indexes (Bradley, 2014)
  • The following article discusses how to determine the level fragmentation of the Full Text Catalogs: Full Text Fragmentation (iFTS)

• Ensure all FIM components are running the same version levels across the solution. (FIM Synchronization Engine, FIM Service, FIM Client, etc…)

• Install and run the FIM Best Practice Analyzer (FIM Best Practice Analyzer for FIM 2010 R2, n.d.)

• Ensure all attributes that are used in “Joins” or metaverse searches are properly indexed in the metaverse configuration for the attribute.
• Remove indexing from any metaverse attributes that are not active using this in Joins or metaverse searches.
• Evaluate “out of band” coding activities. Calls made to systems inside of the FIM provisioning code to external resources (AD, SQL database) are latency expensive. Even small delays (10 milliseconds) add up when performed on thousands of objects.  Make judicious use of “out of band” activities. Consider the use of custom workflows to deliver the functionality made in the out of band activity.
• If external resources (e.g. SQL database) must be called during an “out of band” coding activity, then the setup and takedown of connections to these external resources needs to be done in the “Initialize” and “Terminate” methods for the extension, and not the “Provision” or “Flow” methods. For code that is not thread safe, pooling or caching should be implemented.
• Evaluate the performance of “in bound” logic. Poorly tuned .NET code (even if it never leaves the process context of the sync engine) has a big impact. Even simple stuff like using string flags (“Y”/”N”) vs Boolean values can make a big difference in the overlying performance.
• If there are a large number of valid dis-connectors use the filter feature Declared (Import Filter) in place of the filter feature Declared. This can dramatically speed up synchronization times as the objects do not need to be processed during the cycle. More information about this topic can be found here (Drewes, 2012).

• Remove SharePoint Foundation Search service
• Enable only the needed MPRs for the solution. Disable MPRs that are not needed for the solution, such as an “All Powerful MPR” that may have been created for troubleshooting FIM permissions issue as these MPRs will create multiple requests for each transaction.
• Remove any unnecessary Sets from the solution
• Ensure that there are not any circular MPRs by ensuring that the MPR filters are as narrowly defined as possible to avoid overlap which may result in a circular loop
• Ensure that client update requests only update attributes that are actually changing. For instance, setting an attribute to the value that it already is will kick off MPRs that likely don’t need to run.
• Ensure that the minimum number of requests that are needed to solve the business problem are used (combine multiple attribute updates to the same object into a single request).
• Combine multiple attribute updates into single requests. This means that the built in function evaluator should be avoided when updating multiple attributes on the same object. Instead use the WAL, another 3rd party activity, or write your own.
• Consider use of FIM Service partitions to spread the request load across multiple instances of the FIM Service
• For bulk operations, consider the use of a web services client to talk to the FIM Service directly instead of the PowerShell cmdlets

• In some cases, it makes sense to separate the synchronization engine and FIM Service databases onto separate servers. It should be noted that data from one database is flowing into or out of the other database. If resource constraints occur on one database, it will cause performance issues with the other database.
• SCSM databases should ideally be placed on a different SQL server from the FIM Service database due to potential resource constraints issues as mentioned in the FIM Sync/FIM Service database discussion above.
• If virtualizing the servers, you need to pay special attention to the underlying hardware. For instance, if you have three separate SQL servers on the same physical host or using the same disks, you might not get the performance you expect due to resource constraints.

Antivirus software that is not cluster-aware may cause problems with Cluster Services. (2012, August 17). Retrieved from https://support.microsoft.com/kb/250355

Bradley, B. (2014, Jan 1). How to Automate the Rebuilding of FIM Full Text Indexes. Retrieved from https://social.technet.microsoft.com/wiki/contents/articles/5701.how-to-automate-the-rebuilding-of-fim-full-text-indexes.aspx

Drewes, F. (2012, Jan 8). Pre-Import Filtering of AD objects in FIM 2010 R2. Retrieved from https://social.technet.microsoft.com/wiki/contents/articles/6600.pre-import-filtering-of-ad-objects-in-fim-2010-r2.aspx

File Locations for Default and Named Instances of SQL Server. (n.d.). Retrieved from https://msdn.microsoft.com/en-us/library/ms143547%28v=sql.110%29.aspx

FIM Best Practice Analyzer for FIM 2010 R2. (n.d.). Retrieved from https://technet.microsoft.com/en-us/library/jj203402%28v=ws.10%29.aspx

Helmick, B. (2014, January 16). Comprehensive AV Exclusion document. Retrieved from https://sdrv.ms/LES7gB

Kehayias, J. (2012, November 29). CPU Ready Time in VMware and How to Interpret its Real Meaning. Retrieved from https://www.sqlskills.com/blogs/jonathan/cpu-ready-time-in-vmware-and-how-to-interpret-its-real-meaning/

Kehayias, J. (2013, November 13). Hyper-V Equivalent to VMware CPU Ready Time. Retrieved from https://www.sqlskills.com/blogs/jonathan/hyper-v-equivalent-to-vmware-cpu-ready-time/

Klee, D. (2012, December 12 17). CPU Overcommitment and Its Impact on SQL Server Performance on VMware. Retrieved from https://www.davidklee.net/2012/12/17/cpu-overcommitment-and-its-impact-on-sql-server-performance-on-vmware/

May, J., & Lee, D. (2009, May). Disk Partition Alignment Best Practices for SQL Server. Retrieved from https://msdn.microsoft.com/en-us/library/dd758814.aspx

Performance Best Practices for VMware vSphere 5.5. (n.d.). Retrieved from https://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.5.pdf

Performance Best Practices for VMware vSphere™ 5.0. (n.d.). Retrieved from https://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf

Pinal, D. (2014, January). SQL SERVER – Virtualized SQL Server Performance and Storage System – Notes from the Field #013. Retrieved from https://blog.sqlauthority.com/2014/01/30/sql-server-virtualized-sql-server-performance-and-storage-system-notes-from-the-field-013/

SQL SERVER – vCPUs – How Many Are Too Many CPU for SQL Server Virtualization ? – Notes from the Field #003. (n.d.). Retrieved from https://blog.sqlauthority.com/2013/11/21/sql-server-vcpus-how-many-are-too-many-cpu-for-sql-server-virtualization-notes-from-the-field-003/

Windows Server 2012 Hyper-V Best Practices. (n.d.). Retrieved from https://blogs.technet.com/b/askpfeplat/archive/2013/03/10/windows-server-2012-hyper-v-best-practices-in-easy-checklist-form.aspx

SQL SERVER – Virtualized SQL Server Performance and Storage System – Notes from the Field #013 (Pinal, 2014)

Thank you to Rex Wheeler and Sami Van Vliet for their assistance with this guide.