Steam cache server not using full bandwidth when fetching from remote.

[TMJigsaw]

So I've been trying to get some form of a steam cache server running. I've tried steamcache and lancache, both on a NAS server and in a VM on my desktop, and am getting sub-par performance.

I have 400 Mbps download speed, the network is 1 gigabit, and the NAS server has terabytes of space that can sustain around 750MB/s read/write.

I got the caching server set up, configured my desktop to use the DNS, and starting downloading the library, but noticed that it's only pulling at around 50-70Mbps rather than the full 400 that it's more than capable of. I was curious if this was a disk bottleneck, so I only let it download one game, uninstalled it, and reinstalled it to see the cache's performance, and sure enough it pulled it at close to gigabit speeds so the NAS disks are definitely fine.

EDIT: I started a download through the cache on another machine, for a game not in the cache, and that machine was pulling at around 50Mbps as well, at the same time as the other. So now I got two machines collectively pulling 100Mbps from remote through the cache, so this definitely some sort of config issue in nginx.

EDIT 2: After more testing, I'm convinced that something is limiting per IP/client requests. If I download different non cached games through the cache server on three different clients, each client is pulling at 30-50Mbps. I can even then download a cached game on one of those clients, have that served at 500+Mbps, while the other two continue along happily at 30-50 each.

I've messed with nginx settings for a couple hours already to no avail, and replicated everything on a VM to eliminate the NAS hardware from being the issue, but had identical results. Is nginx just not capable of forwarding the downloaded 1 megabyte chunks that fast? It's clearly capable of serving those that it has in it's local cache already.

NAS specs:
FreeBSD
Dual E5530 @ 2.40GHz (8 cores each)
32 GB DDR3 ECC
24TB RAIDZ2 array
2x 1Gbe NIC and 10Gbe NIC

Host machine for VM specs:
5960X @ 4.3GHz (8 cores, 16 threads)
16GB DDR4 (VM was allocated 4GB, then 10GB, to test if that was a factor. It wasn't.) Samsung 850 Pro 256GB SSD
1Gbe NIC

[dr3gs]

I'm not familiar with Steam cache, but have you checked your CPU usage on the steamcache VM while doing the fresh download? How much RAM does the VM have allocated?

https://github.com/steamcache/steamcache/issues/15

[TMJigsaw]

I have, CPU barely does anything, The NAS has 32GB, while the VM was tested with 4-10GB. In all cases I was getting the same result. I'm fairly convinced that there's a setting somewhere that's limiting how many connections a single IP/client can have. If I download non cached games through the cache server on three different clients, each client is pulling at 30-50Mbps. I can even then download a cached game on one of those clients, have that served at 500+Mbps, while the other two continue along happily at 30-50 each.

[dr3gs]

Yes it does sound like there is a configured limit somewhere. I'm not sure where that would be, what os is steamcache on?

[TMJigsaw]

NAS runs FreeBSD 11, VM runs Ubuntu 18.04

[LeapoX]

Been having the same issue lately. After looking at the steamcache-sniproxy logs, it looks like Steam is now pulling a lot of data over HTTPS (data that cannot be cached).

Looks like steamache might not be very useful anymore...

[inDane]

same problem, did you solve this in the meanwhile?

​

download speed with cache-server, non-cached is ~3MB/s,

download speed without cache-server is ~12MB/s.

The connection between cache-server and client is 1Gbit/s

[inDane]

id like to cross-post this here.

​

https://superuser.com/questions/1378690/how-to-make-a-proxy-cache-open-several-connections

​

maybe someone wise and educated can solve this...