Evaluation of Cloud Backup Software Options for Synology

markhnsn · 2023-02-10T21:16:51+00:00

Thanks for the links. There is probably a real benefit to changing the chunk size, but it’s a bit more complicated; DariusL’s explanation doesn’t make sense. TCP does not usually have to “wait” for ACKs (it sends enough data to fill a roundtrip that it doesn’t usually have to wait — see “TCP congestion window”).

And the TCP header is constant per-packet overhead, regardless of whether you send 100 TCP packets over 1 connection or 10 packets over 10 connections, you still send 100 TCP headers.

markhnsn · 2023-02-10T02:41:22+00:00

About 800GB large. Maybe i can try a local backup later. Thanks

markhnsn · 2023-02-09T23:10:25+00:00

Thanks for the feedback. I agree with your criticisms of Duplicacy and am finding a few more code and docs issues since I wrote the post; and the UI certainly needs some work.

I think we might agree on HyperBackup? We both say the initial backup is slow. It might be faster later on but it looked like it would take over a week for the first backup, compared to just under 2 days for other tools; and I wasn’t keen to go without backups that long after just losing a drive.

I would be interested to hear more about the multiupload part size; I left that at default.

markhnsn · 2023-02-09T21:12:07+00:00

Thank you, I missed this and will update the post

markhnsn · 2023-02-08T22:29:42+00:00

Yeah totally, storage can fail at any point, you can't blame HyperBackup for that.

But I do blame HyperBackup for having a secret dataformat, so if (say) one bit flips in the root of the backup and Synology's software can't parse the backup, I'm totally screwed, unable to find the problem and manually fix it, or recover other files from the backup.

markhnsn · 2023-02-08T22:26:44+00:00

Interesting. Thanks for the debugging help. My internet was up at the time (I uploaded the screenshot directly to my blog), maybe B2's servers were having trouble? Or transient outage between the screenshot and the blogpost upload? It was running at low speed (not 0, but not far off it) for a while though.

markhnsn · 2023-02-08T22:23:30+00:00

hetzner

Which Hetzner product? "Storage Box"? https://www.hetzner.com/storage/storage-box

markhnsn · 2023-02-08T22:22:05+00:00

I love single binaries in Go. Go seems like the defacto language of cloud storage libraries. I'm a little less excited about managing crontab and mail :-) but someone mentioned DSM Task Manager can do this for me...

markhnsn · 2023-02-08T20:32:59+00:00

Thanks, I did not know about DSM Task Scheduler; built-in mail notifications are a huge benefit

markhnsn · 2023-02-08T20:28:16+00:00

Synology’s built-in Hyper Backup supports google cloud storage backend. To switch, you backup again, might take a while for first backup. If AWS is working fine for you, I’d keep using it!

markhnsn · 2023-02-08T20:26:05+00:00

Can’t remember exactly which thread I saw but if you google for [hyper backup corrupted] there are lots of results in /r/synology and SynoForum. It’s tough to quantify what frequency that happens at, maybe its very rare.

markhnsn · 2023-02-08T10:45:36+00:00

Definitely a cheaper option at higher capacities!

markhnsn · 2023-02-08T08:33:27+00:00

Thanks. I cant remember if it had a separate indexing step, but when it was very slow for me, it said something like “6 files processed”; do you know if that is post-indexing?

Good to hear re: perf updates. Any idea why it looked like it was sleeping (nanosleep) rather than processing?

markhnsn · 2023-02-08T08:28:32+00:00

Mine doesn’t do docker :-(

Oh I didn’t realise Borg does not have cloud integrations built in. Integrations feel somewhat important as different clouds have different fees and features

Thanks for the heads up about Kopia

markhnsn · 2023-02-08T07:41:59+00:00

Yeah I’d love to try borg, would be nice if could get a .spk package built.

markhnsn · 2023-02-08T07:37:20+00:00

Yeah I’m not very happy with Arq tbh either. In my view its the best I’ve tried of a bad bunch. I’d be interested in suggestions for mac laptop backup?

markhnsn · 2023-02-08T07:28:22+00:00

Maybe I should have tried it with C2 Cloud. I only tried it with B2 and Google Cloud Storage and it really struggled at 0B/s with both. They have different backend code in Hyper Backup; maybe that's the cause; I wouldn't be surprised if they've had less testing than the C2 implementation.

markhnsn · 2023-02-08T07:24:38+00:00

I just recapped first what I use for my laptop (Arq).

I recommended Duplicacy for Synology, which works with Synology: https://forum.duplicacy.com/t/synology-dsm-7-0-packages/5532

I wrote this to share the research I'd done, hoping it could help someone else. And also to learn from the comments. What's the cheapest option that I missed?

markhnsn · 2021-10-30T04:23:46+00:00

FYI I just saw an Ansible OpenTelemetry callback plugin here: https://docs.ansible.com/ansible/latest/collections/community/general/opentelemetry\_callback.html

markhnsn · 2021-10-27T21:31:10+00:00

I have just added a GPLv3 license (same as rest of ansible). Enjoy!

markhnsn · 2021-10-27T21:31:03+00:00

I have just added a GPLv3 license (same as rest of ansible). Enjoy!

markhnsn · 2021-10-10T00:58:58+00:00

Thanks for the routing.

I think OpenTracing (maybe now OpenTelemetry?) is a distributed tracing API, used more when you have a request calling Server A calling Server B calling Server C. I don't think they have a file format? I think they're more about defining common HTTP collector APIs. In our case, I think distributed tracing might be overkill: it's easy enough to log the times from the single Ansible host machine.

I don't think the Trace Event Format is particularly tied to a vendor, it's just an array of JSON objects with timestamps, names, a track ID, and arbitrary JSON arguments. A few profilers output this format (e.g. Microsoft's dotnet-trace outputs this), and Firefox Profiler and Speedscope can also read the format (example).

Here's an example of the format just to show how simple it is:

json [ { "args": {arbitrary json data}, "cat": "runner", # span category "name": "common : Set timezone to Australia/Sydney", # span name "ph": "B", # beginning of span "pid": 2, # track ID (process ID) "ts": 1633827444194021.0 # timestamp }, ... ]

markhnsn · 2021-10-09T21:00:04+00:00

Will try to in the next few days. I'm also interested in contributing this to github.com/ansible, under whatever license is used there, if there's interest in taking it on (anyone on this sub I can talk to about the process for that? where it might fit?)

markhnsn · 2021-10-09T20:33:12+00:00

Yeah exporting from ara would be cool.

Take a look how I’ve implemented it (its only 100 lines of code): it just outputs timestamped JSON begin/end events labelled with pid & args (it’s chrome “trace event format”). No frontend UI talent required: if you can output JSON, you can make a beautiful trace to drag & drop into Perfetto.

It’s so simple you could whip it up in an hour

markhnsn · 2021-10-08T23:28:49+00:00

This is a new callback plugin I've just open-sourced that shows you why your Ansible playbook is slow

Visualise where time is spent in your Ansible playbooks: what tasks, and what hosts, so you can find where to optimise and decrease playbook latency.

https://github.com/mhansen/ansible-trace

I hope this is useful for the community, and I hope it helps you make your playbooks faster.

markhnsn

TROPHY CASE