After working with Microsoft Team Foundation Server since 2004 (remember trying to install that first beta?) I have learnt a lot about TFS. I am also fortunate enough to be part of two virtual teams who are the absolute experts on TFS. The first virtual team is the Microsoft Premier Field Engineers specialising in Application Lifecycle Management – all things TFS included. This group consists of the deep experts in the field, assisting customers to get it right. I have some truly brilliant colleagues in the group who astound me with their knowledge and insight. They understand the hardware, how memory works, how SQL operates, the detail of web service performance and the kernel of the operating systems. I often rather keep quiet, standing in awe of their genius!
From this group I have learnt a lot of small changes that can make a huge difference to the performance of Team Foundation Server. When we talk about TFS, it is important to remember that TFS is not a single application or even a set of applications. TFS is a complex solution using several technologies that are stacked together to provide a solution that is truly great!
So when we talk about improving the performance of TFS we need to look at SQL Server, IIS, SharePoint, Analysis Services, Reporting Services and network.
I will write another time about capacity planning for TFS. It is not as straightforward as calculating your number of users and plotting it onto the correct number and sizes of servers. More on that another day…
You already have a TFS environment up and running (or currently it is crawling). How can you gain some performance gains with relatively little expense and effort?
Reasons for performance degradation:
- High pages/sec reading - this means you are using virtual memory instead of RAM. Virtual memory is memory on your hard disc assigned to the Operating System to use as working memory. Ideally your pages/sec reading should be less than 20 – less than 10 is best. Your maximum shouldn’t peak over a 100 too often. This is the ideal scenario. Numbers higher than this and you will experience poorer performance simply because accessing virtual memory means the heads of your hard drive have to move. Solution - add more memory.
- High pages/sec reading while Available memory is also high – Before you add more memory, ensure that your machine’s virtual memory setting is set to System Managed Size. If the virtual memory is not system managed, you might not be using all your RAM and still have high usage of virtual memory. I don’t know or understand why. I also don’t know whether this just applies to older versions of Windows Server, but in practice I have seen this more than once!
- Hard drive contention – If any or all of the following occurs you have hard drive contention: less than 15% idle time for the physical disc interface, read and write responsiveness is slow – say higher than 25 milliseconds and the physical disc queue length becomes greater than 4. Solution – Provide separate hard drives for the SQL TempDb and the TFS databases. Also provide separate hard drives for the IIS content and the IIS logs. Both of these actions will improve the performance of your TFS environment if performance was affected by hard drive contention.
- Build server, Application Tier and Data Tier on single server - this has never been a recommended deployment. If this is what your environment looks like, expect poor performance when you are running a build. Your poor server is suffering – even if you are only 10 users. Deploy your Build servers on separate machines. Ensure that these machines comply with the appropriate hardware specifications for your team’s build habits and the size of your builds.