Why does Killing Floor 2 open so slowly?

I play alot of Killing Floor 2 (KF2) on my PC, it’s a pretty fun game and generally runs quite well. I have noticed, however, that compared to most other games I play it takes quite a long time to load. It takes around 3-4 minutes, in fact, which is obscenely long as far as I’m concerned. During one of the many times I was twiddling my thumbs waiting for the game to load, I started wondering what the hell was taking so long.

After a little Googling it was clear I’m not the only one experiencing this frustration and it’s not limited to only  KF2. One of the worst offenders appeared to be GTA Online and it seemed it was taking some players ~6 minutes to load. ‘t0st’, an avid GTA Online player, who I assume was also fueled by pure frustration, investigated GTA Online's behaviour and developed a solution/fix to the lengthy delay he/she/they were experiencing. More importantly to this article,  t0st details the methodology and tools to develop this fix used on this website (https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/). 

Not feeling the need to reinvent the wheel, I figured I would utilise t0st’s methodology and apply it to KF2.

Now, before we get into the nuts and bolts, a few point to bear in mind:

My PC specs - it’s a reasonably old machine, but then again KF2 is an older game so the horsepower available should be more than sufficient.
I5-4690
AMD R9 380 
16GB of RAM
160GB SSD (INTEL SSDSA1M160G2LE) OS + Steam installed in program files
4TB HDD (HGST HDN724040ALE640) Steam library folder

My install location for KF2 is on a mechanical Hard Drive (very important factor as you will read below). It’s a 7200RPM drive, but it is spinning rust and put to shame by even an entry level SSD in several important ways (to this problem).

Additionally, I am applying the startup parameter “-nostartupmovies” via Steam to KF2, which as you might have guessed prevents the various intro movies being played every time you load the game (sweet mercy!). This does cut the loading time down, but not surprisingly only by around the length of these movies. Note that all testing conducted below is with this parameter applied, so your KF2 start up times may differ.

-- Task Manager

The first stop on this journey is one most users will be familiar with, the venerable Task Manager. Task Manager is a tool built into Windows which provides a range of insights into the functioning of a computer and individual programs, albeit at quite a high level. Opening KF2 whilst running Task Manager shows several interesting things. Firstly, it appears that while loading KF2 has some periods of high CPU use on, but also extended periods of little or low CPU usage. This implies the problem is likely not one borne from a slow or outdated CPU. Incidentally, it also appears this is quite different to t0st’s experience, given excess/inefficient CPU processes seemed to be driving the GTA Online problem. So, more digging is required.

Task Manager also reports that loading KF2 is resulting in a high level of disk activity (shown in the screenshot below), however the transfer rate (i.e. amount of data being moved around) never seems to be that high (almost always below 25MB/s) and certainly never approaches the theoretical limit of my HDD (~130MB/s sequential read speed).

For reasons that will become clearer below I timed KF2’s load time on my PC, it takes 198 seconds. Consider this a ‘baseline’ time to be improved upon.

-- Luke Stackwalker

Luke stackwalker samples the stack to see what is running. It then gives statistics about what functions are running. If you load the Symbols from the Microsoft servers you can get information about the Windows functions that are being called in the program you are debugging. Let’s give Luke Stackwalker (http://lukestackwalker.sourceforge.net/)  a go because that is one of the major reasons for this project. So I fired up Luke Stackwalker and it gave this: 

This shows the majority of the calls sampled where zwReadFile.This is to be expected based on what we saw with Task Manager. So what is zwReadFile? Well, it’s part of the Kernel mode drivers in Windows, under the wdm.h header section. https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-zwreadfile. Essentially, it is used to read files into memory.

-- Process Monitor

We need to see what zwReadFile is doing. The whole install of Killing Floor 2 is 69.2 GB and 91,327 Files, 511 Folders. So there are a lot of files that could be touched. I used Process Monitor ( https://docs.microsoft.com/en-us/sysinternals/downloads/procmon ) with a bit of filtering and discovered some nice little features called File Summary and Process Timeline. 

File Summary shows what Killing Floor 2 is doing with the file system. Process Timeline gives the timeline for the File system reads, writes and other actions like processing.

We will ignore the C drive because only 1.5 seconds of time is spent there. This is insignificant for our purposes. The Important information is ~17169 file opens and ~2.6GB of data is read into memory or on average 0.15MB per file. 

-- Guesstimation of the cause

We have lots of files being read but only ~2.6GB of data being read in total. This seems very much like a latency problem. The average access time on my hard drive is ~10-15ms (https://www.anandtech.com/show/8743/hgst-deskstar-nas-4-tb-review/3). Assuming we have to seek every new file and that they are not one after another - 17169 * 10 / 1000 =  171 Seconds of seek time. The drive will average about 130MB/s transfer sequentially.  Assuming all drive reads are sequential another 2600/130= 20 seconds of transfer time. This gives us a total of 191 second which is pretty close to the 198 above.  Some of the seeking or transfer might have been a bit slower?

-- Guesstimation of the Solution 

Based on this we can guess the performance of a Killing Floor 2 if installed on a SSD. In theory the INTEL SSDSA1M160G2LE has a seek time of  75 µs for reads ( https://ark.intel.com/content/www/us/en/ark/products/56565/intel-ssd-320-series-160gb-2-5in-sata-3gb-s-25nm-mlc.html ) So we should get 17169 * 75 / 1000000 = ~1.3 seconds. The drive will average about 250MB/s transfer sequentially. This gives 2.6GB / 250 = ~11 seconds of transfer time. It is unlikely that it will take < 15 seconds to load as there are processes being run on the CPU as well, but in theory it should be under < 30 seconds. 

-- SSD Test

I changed Killing Floor 2 to run on my SSD. The setup is the same as above but now: 160GB SSD (INTEL SSDSA1M160G2LE) OS + Steam installed in program files + Steam library folder. So after a lot of clean up and moving stuff around. Time to open Killing Floor 2 on an SSD is 80-90 seconds. More disk usage can be observed and more CPU usage can be observed in a more compressed time frame. It looks like it is doing nothing for 40-60 seconds. This is less pronounced in the previous test on the HDD. Maybe there is some kind of timeout/errors happening? 

-- The Re-install

Thinking maybe this was a problem with my installation i completely reinstalled Steam and Killing Floor 2. No change still 90 seconds.

-- Steam

With a little bit more looking at the other processes running I noticed that steam.exe was part of the cause of another delay. This can be seen when putting the two graphs together as per below. The graphics are both of the Process Timeline of the respective programs. Killing Floor 2 pauses for a bit of time while Steam does some kind of process. This lasts for 40 seconds as seen in the section circled below.

I did a bit of looking at various forums around the internet to see if anyone else has had this problem. I discovered a few that say to disable “Enable the Steam Overlay while in-game”. I tried it out and this cuts 30-50 seconds off load time. 

--The correction and re-test

After the discovery of the 30-40 second Steam issue, the “Guesstimation of the cause” section is inaccurate. A retest gives 166 seconds on the HDD with “Enable the Steam Overlay while in-game” disabled. The seek time of the HDD is still a major issue however. This consequently makes the hard drive performance more efficient than first thought. This makes sense because there will be multiple files that are one after another being loaded and no seek time will be required to read the next file.

--The Fix

What can you do:
The easiest way to fix seek times is to install and run KF2 on an ssd. 
Assuming you don’t use Steam Overlay, disable it, IF you see high CPU usage during the loading of Killing Floor 2 by the steam.exe process. 

What could the developers of Killing Floor 2 do:
The developers could implement a caching system. The files that need to be cached could all be put into a single file ~3GB. After all what’s another 3 GB in a 69GB + install. This file could then be read sequentially. 3000/130MB/s = ~25. This would require writing the caching process which creates and loads the cache. Then it would require rewriting all the code that loads data across what is multiple threads(by the looks of it) to be changed to first check the cache to see if the data is required is there, if so use it and if not load it like normal. How hard would this be to do? Probably reasonably hard and it would require a lot of testing. 

What should the developers of Killing Floor 2 do:
Change the recommended system requirement to be ssd storage.

-- TL;DR

I hate excessive game loading times. In my case KF2 takes 3-4 minutes to load. This is too long.

Based on my testing most of this delay seemed to be caused by seek time on my HDD. Although a decent chunk of the time (20%) was caused by a delay introduced from Steam Overlay related processes.

If you want faster loading ,run KF2 (and your other games) from an SSD.

If you don’t use Steam Overlay you may consider disabling it for KF2 (and possibly for other games) if you notice it is excessively utilsing your CPU and adding to your load times.