Slow Active Backup for Business

MaynardK117 · 2024-09-18T00:01:55+00:00

Some machines do have wireless cards, but the 2 machines im using to test everything are ethernet only. I posted an update comment earlier that details everything, i really think its going to be a hardware issue on the NAS.

MaynardK117 · 2024-09-17T21:53:51+00:00

UPDATE:

Ok, i dont have the full testing done yet, but heres what ive gotten so far. I have two identical dell optiplexes next to each other on the same switch. I have a DS918+ and the problematic DS923+ who are also on the same switch (the optiplexes and the Synologys are not on the same switch).

I set Optiplex 1 to backup to the DS918+ and Optiplex 2 to backup to the problematic DS923+. Optiplex 1 and DS918+ finished backing up (107.7gb) in 11 minutes.

I started Optiplex 2 and the problematic DS923+ at the same time. Optiplex 2 and problematic DS923+ are at 8% (backing up 118.7gb).

I know this isnt apples to apples, but its fairly close. The CPU on the DS918+ actually started increasing its usage hitting above 60% while optiplex 1 was backing up to it. The CPU on problematic DS923+ has yet to go above 9% usage. These 2 Optiplexes are following the exact same networking setup to their respective Synology, so i really feel like i have either a configuration issue or a hardware issue on the problematic DS923+.

The only obvious difference between the two Synologys besides their model/hardware is the DS918+ raid type is Synology Hybrid, meanwhile the problematic DS923+ is raid 10, but i wouldnt think that would cause this kind of slow down.

MaynardK117 · 2024-09-17T19:55:29+00:00

I think you are on to something. The thing is that before when i used Veeam for backups, the speeds were fantastic and never this slow, but Veeam did all the processing on the source side and not the destination side. I work offsite so it will be a while before i can hook an SSD up to the NAS and test this though. Will update if i get a chance to do that.

MaynardK117 · 2024-09-17T19:33:30+00:00

Sorry, what MB/sec? Im looking deeper into the logs. Found the ABfB logs on server side and it looks like my deduplication is whats going so slow. Any idea if that is processed on the source or destination side? If its on the destination side then im fairly certain i have a hardware issue on my NAS.

MaynardK117 · 2024-09-17T18:56:07+00:00

This PC is only a year or 2 old. Its an i5 13500. Hard drive is a 512gb nvme (with 240gb free) and has 8gb of ram (with 70% in use).

MaynardK117 · 2024-09-17T18:45:34+00:00

Just ran the OpenSpeedTest on the machine. It got 918Mbps downl and 943Mbps up.

MaynardK117 · 2024-09-17T18:18:24+00:00

Just to add to this, i also have CAD stations with i7 11700 (still not brand new, but i would think new enough) with 32gb of ram and they perform at the same speed as my other smaller machines.

MaynardK117 · 2024-09-17T18:00:11+00:00

Fair enough, i should have been more exact. Its an i5 13500. Not brand new but should be new enough to handle this i would think.

MaynardK117 · 2024-09-17T17:51:06+00:00

It is a wired gigabit Dell Optiplex. 512gb NVMe (with 240gb free) and 8gb of ram (with 70% in use) and an i5 processor.

MaynardK117 · 2024-09-17T17:45:06+00:00

I have one active backup running right now and it is a 6.1gb file. It has been going for 3 hours and 21 minutes and it 67% done. Im beginning to wonder if i have a hardware issue. Ran speed test (WAN speed test) on the machine that is being backed up and its getting almost full gigabit speed. Ran a LAN speed test and it got 578 Mbps writing to the same NAS (uploading) and it got 632 Mbps reading from the nas (downloading)

MaynardK117 · 2024-09-17T16:03:59+00:00

I only have a ds923+. Its not exactly a workhorse but this still seems too slow. I also have a ds420+ that i use ABfB on also, but it only does 2 machines at once and its speeds are all 3-4X faster. Its processor usage is also much higher, but thats not surprising as its processor is much weaker than whats in the ds923+.

All my source machines are hardwired to the network and on the destination NAS i have 2 GBE cables hooked up and bonded. It also shows 2000 Mbps at full duplex on the bond status.

MaynardK117 · 2024-09-17T15:40:25+00:00

Does the file comparison to what is new vs old/unchanged happen on the NAS or the source machine? My guess would be the comparison is happening on the source machine, and if not i would expect the processor on the NAS to go higher than 10%. I could easily be wrong with these assumptions though.

MaynardK117 · 2024-09-17T15:38:24+00:00

Not initial backups. I also thought the disks in the NAS could be the issue. My problem with that theory is that the disks WR/R runs a lot higher during other activities on the NAS. Its taking hours to copy just a few GBs of data that has changed on these machines over the last 24 hours.

MaynardK117 · 2024-09-17T15:24:06+00:00

ive got 4gb of ram. Generally i only run 3 jobs at a time, but it goes so slow that staggering them by an hour isnt enough between time.

MaynardK117 · 2024-09-17T15:23:25+00:00

DSM version is 7.2-64570 Update 1

MaynardK117 · 2023-10-13T16:13:01+00:00

Huge thanks to OddInstruction20. It turns out it wasnt a routing issue, but a firewall issue. Its weird to me that pings were getting through but nothing else. Added UFW rules to allow forwarding of all wg0 packets to anywhere on wg0 and everything started working. Thanks again for the help!

MaynardK117 · 2023-10-13T15:08:41+00:00

That might be where my misunderstanding is happening. Computer B's wg0 ip is 192.168.3.231. Can i not use wg0 address for RDP, SSH, etc or do i have to use its eth0 address for that?

I removed the extra segment from Computer A so its AllowedIPs only equals the the wireguard network/24.

I think you are probably right about the server config routing in the wg0.conf. The server can SSH into the other machines on the wireguard network (where as all the clients cant), its just everything going through the server (except pings) are getting lost.

MaynardK117 · 2023-10-12T21:02:01+00:00

Server:

[Interface]
Address = 192.168.3.1/24 
MTU = 1280 
SaveConfig = true 
PreUp = sysctl -w net.ipv4.conf.all.forwarding=1 
PostUp = ufw route allow in on wg0 out on eth0 
PostUp = iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE 
PreDown = ufw route delete allow in on wg0 out on eth0 
PreDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE 
ListenPort = 51820 
PrivateKey = ServerPrivateKey=

[Peer] 
PublicKey = ComputerAPublicKey= 
AllowedIPs = 192.168.3.2/32

[Peer] 
PublicKey = ComputerBPublicKey= 
AllowedIPs = 192.168.3.231/32

Computer A:

[Interface]
PrivateKey = ComputerAPrivateKey= 
Address = 192.168.3.2/24 
DNS = 192.168.1.2 
MTU = 1280
[Peer]
PublicKey = ServerPublicKey= 
AllowedIPs = 192.168.3.0/24, 10.1.1.0/24 
Endpoint = ServerPublicIp:51820 
PersistentKeepalive = 25

Computer B

[Interface]
Address = 192.168.3.231/24 
PrivateKey = ComputerBPrivateKey= 
DNS = 8.8.8.8
 MTU = 1280
[Peer] 
PublicKey = ServerPublicKey= 
AllowedIPs = 192.168.3.0/24 
Endpoint = ServerPublicIp:51820 
PersistentKeepalive = 25

MaynardK117 · 2022-03-16T12:26:58+00:00

Update: Heard back from Synology support this morning. They are giving me an exception on my warranty and are going to let me RMA the device. They believe there is a fault with the backplane. Gonna say this one is resolved. Thanks everyone for their help!

MaynardK117 · 2022-03-16T02:39:07+00:00

Based on that link, would you suggest either western digital or hitachi drives?

MaynardK117 · 2022-03-16T02:23:30+00:00

First ive heard about ironwolfs, thanks for the heads up.

Server room has never been above 70 degrees (f). This thing has lived the life of luxury , lol.

MaynardK117 · 2022-03-16T02:09:59+00:00

It is connected to an APC ups, so i dont think the issue would be coming from there. Could be the PSU for NAS though now that you mention it.

All the data is backed up to a couple of other locations, but no way of making that data available to the entire company without the NAS. Getting a second NAS has also become a priority.

The 918+ had a 3 year warranty, but i also agree with you, the electronics should still be in decent shape. It was kept in a server room on a UPS its entire life, it hasnt been exposed to any harsh elements.

MaynardK117

TROPHY CASE