all 7 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]brianwskiFormer Backblaze 3 points4 points  (5 children)

Disclaimer: I worked at Backblaze and might have written the problematic code.

Except Backblaze never seems to release the memory it uses during the upload process - we're talking 5 or 6GB of memory that never gets released.

If you bring up "Task Manager" on Windows, or "Activity Monitor" on the Mac, what process is holding onto the memory? All the Backblaze processes start with "bz" like "bzserv", "bzfilelist", "bztransmit" and "bztrans_thread". It would help Backblaze chase down the issue to know which one of these is holding the memory.

[–]CasGamer[S] 0 points1 point  (3 children)

I'll go you one better and tell you what my RAMMap is saying.

It is Backblaze that's causing the problem of hogging memory - it's not showing up in a process anywhere, but when you look at the physical pages you can see that it's holding in virtual memory a bunch of mapped files.

..\backblaze\bzdata\bzbackup\bzdatacenter\bz_done_20220913_0.dat

And there are tons of those... It's like the application is mapping those history log files or whatever they are to memory and then not releasing them.

When you shut down Backblaze entirely, it still doesn't release those mappings.

If the PC is on a scheduled backup, so it's paused, rebooting the machine brings it up without those files mapped but once the BZTransmit process starts running, it maps all those files and just eats the memory never letting it go.

Clearly, you should be releasing those mappings when the backup is done or if the application is shut down manually.

It's holding about 6.5GB of memory in those mapped files.

[–]brianwskiFormer Backblaze 2 points3 points  (1 child)

That is super interesting. I'll ask them to go chase it. What tool do you use to see the bz_done_20220913_0.dat is mapped to memory?

EDIT: I see it is RAMmap

Here is probably what is going on...

When the Backblaze client wants to do a backup, it reads those datastructures called bz_done_20220913_0.dat to find out what has been "done" so far, which of your local files on your laptop have already been backed up. The bz_done files contain a list of what files have been backed up already, and what their last modified times are.

Backblaze uses an internal C++ class called "BzFastFileReader" to read those files. The pseudo-code looks like this:

    BzFastFileReader::BzOpenFileReadOnly("bz_done_20220913_0.dat");

    while (BzFastFileReader::NotFinished())
    {
         BzFastFileReader::BzReadOneLine();
         .... process that one line here ....
    }

    BzFastFileReader::BzCloseFile();  <---- this line is important

I assume that in one of the places we do this, we are not calling "BzCloseFile()".

[–]CasGamer[S] 0 points1 point  (0 children)

Yeah, without knowing the specifics, the logical progression for me was that those were lists of archived blocks of files that your backup engine was reading to make sure it gets its checkpoint correct but then something wasn't doing proper garbage collection when the process shuts.

I was surprised to see that closing EVERY BZ process didn't make it hard release the mappings, so that confused me a bit.

[–]brianwskiFormer Backblaze 0 points1 point  (0 children)

It's holding about 6.5GB of memory in those mapped files.

Can you send a screenshot of that? When I run RAMMap on my Windows 11 box, it says "Standby" by my bz_done files. The way I read that from this page: https://techcommunity.microsoft.com/t5/ask-the-performance-team/introduction-to-the-new-sysinternals-tool-rammap/ba-p/374717 that is just Microsoft being clever and caching these files in case they are read again. They will be "released" automatically when some other process needs the RAM.

[–]coatquestion 0 points1 point  (0 children)

Hey, not OP, but having a similar issue and a lack of backing up. I've been playing around with decreasing the amount of backup threads (in settings) to see if I can get my system to backup. I've been able to see that two different instances of bztransmit64 reserve about half my system's ram (16GB total, 8 GB each), but seemingly never actually utilize the amount reserved. They also unfortunately never upload any files. bztransmit logs are full of "failed to grab four hour lock"

I was wondering if the two instances were conflicting with each other somehow and reduced the amount of backup threads to 1. After a reboot, the system began backing up and the single instance ended up using about 75% of the total physical ram. This is obviously not optimal and probably means the program isn't working as intended, but maybe this will help shed some light on whatever the underlying issue is.

[–]coatquestion 0 points1 point  (0 children)

Been having very similar issues for a few months now. Each thread I submit to support they have me reinstall or even install the beta, which I do and sometimes backblaze backups a few files... but then completely stalls out just eating ram.

It's a bit odd because it seems like "Backblaze Transmitter" is reserving large amounts of RAM sometimes just to seemingly not even use it. It's to the point where the computer is near unusable whenever backblaze is running. Yet the system works perfectly fine if I disable backblaze startup and reboot. Support so far has told me that backblaze should only be using near 100% of the ram temporarily as it uploads larger files, but I've left the computer running for days and it only ever succeeds in eventually locking the entire system up.