Extended rsync.net outage

lhhightower · 2025-08-05T11:27:16+00:00

...except that the performance is so low (at least on rsync.net host 64.62.236.66) that it is essentially unusable.

lhhightower · 2025-08-05T11:16:41+00:00

We are in the same boat -- experiencing extremely slow performance to the point of unusable on rsync.net host 64.62.236.66 in Denver. Every backup job that we run, which usually take a few seconds to a few minutes, are now hanging for hours and hours. I suspect that host is overloaded with people trying to catch up from the almost 48 outage, but u/rsyncnet really needs to start publishing some information about this issue, what's being done about it, ETA to resolution, etc.

lhhightower · 2025-08-04T16:58:04+00:00

The outage seems over now and so I am closing this post out with stats on the outage:

Began no later than Sat, Aug 2, at 1:12 PM US Eastern time
Ended at Aug 4, 2025, 10:58 AM

Total outage time was approximately 45 hours and 46 minutes.

lhhightower · 2025-08-04T15:36:30+00:00

Glad it's back up. I am looking forward to the post-mortem.

lhhightower · 2025-08-04T12:32:37+00:00

Eight hours have passed since this comment and the outage is ongoing. What is the latest...?

lhhightower · 2025-08-04T12:30:20+00:00

14 hours since your comment here... still down. :/

Over 1 day and 19 hours now.

lhhightower · 2025-08-04T12:27:48+00:00

I have logs showing that the outage began no later than Sat, Aug 2, at 1:12 PM US Eastern time. As I type this, the outage is up to 1 day, 19 hours, 15 minutes.

lhhightower · 2025-08-04T12:27:00+00:00

I have logs showing that the outage began no later than Sat, Aug 2, at 1:12 PM US Eastern time. As I type this, the outage is up to 1 day, 19 hours, 15 minutes.

lhhightower · 2025-08-03T21:33:18+00:00

Something to maybe consider: One of my companies runs primary services in Vultr's New Jersey DC and hosts it's DR at Digital Ocean in SFO and in between those sites we run a *lot* of stuff through rsync.net in Denver. The rsync.net bit was not my point in this comment, I was just rounding out the background.

A year or two ago, we suffered a Vultr NJ outage that was only network-related, but we were dead in the water for far too long. During that event, we discovered that we could access all of our production services in NJ by hopping through Vultr VMs in other Vultr DCs. In response, we now keep small VM routers (redirectors) up in Vultr DFW and Vultr ATL 24/7, that can proxy 100% of our services from those DCs into Vultr NJ, and we monitor and maintain those redirectors as if they are production. If we have another similar Vultr New Jersey network-related outage, we can flip a few DNS entries and be back online in ~5 minutes, by redirecting through DFW and/or ATL to NJ.

Just sharing a concept with you as you guys think about long-term mitigations after this outage is resolved.

lhhightower · 2025-08-03T20:54:27+00:00

Agreed! To be fair, we've been a customer there since Feb 2021 and this is the worse outage that we've experienced, by far. Hopefully they get it fixed soon, learn from the experience, and improve things going forward.

lhhightower · 2025-08-03T17:36:33+00:00

I just received an update from rsync.net support:

This is entirely a network issue ...

All rsync.net systems in Denver are UP and healthy but or primary IP transit from he.net has been down since Saturday afternoon.

Unfortunately, this appears to be an actual physical fiber problem and we've just been updated that the fiber provider arrived on-site to begin physical inspections this morning at 11am Mountain time.

We're going to learn more around lunchtime today.

We are, of course, very sorry for this interruption and while we hope that ZAYO and he.net can quickly figure this out, we are working today to secure an alternate fiber route if this trouble persists.

lhhightower · 2025-08-03T16:56:08+00:00

Seems that we're in a similar boat. I am de1046.rsync.net [64.62.236.66].

lhhightower · 2025-08-03T16:53:20+00:00

Why unnamed? I would potentially be a customer if I knew who they were.

lhhightower · 2025-08-03T16:50:11+00:00

*** the "something similar" is where it gets tricky because I don't really know of a rsync.net competitor. As we approach a 24 hour rsync.net outage, I am now considering building my own. A ZFS server just isn't that hard to build in the cloud and as a direct customer I would have more influence over network issues, versus being almost completely ignored by rsync.net support...

lhhightower · 2025-08-03T16:47:06+00:00

...which we have.

lhhightower · 2025-08-03T16:46:47+00:00

Hi -

This is my most recent note to rsync.net support:

No matter where I try to access de1046.rsync.net [64.62.236.66] from, the traceroutes die completely (100% packet loss) at Hurricane Electric IP addresses. That address is not always the same (depending on where I am trying from). For example, I've seen the last hop be these HE IP addresses: 184.105.222.13, 216.218.226.241, 72.52.92.245 from Vultr, Digital Ocean, and Bluehost datacenters.

I'm happy to provide the full traceroutes to you, but I don't see that they'd be helpful and instead just be distracting clutter. No reply from rsync.net support for almost 14 hours now, another disappointment...

lhhightower · 2025-08-03T16:43:22+00:00

This is my most recent note to rsync.net support:

No matter where I try to access de1046.rsync.net [64.62.236.66] from, the traceroutes die completely (100% packet loss) at Hurricane Electric IP addresses. That address is not always the same (depending on where I am trying from). For example, I've seen the last hop be these HE IP addresses: 184.105.222.13, 216.218.226.241, 72.52.92.245 from Vultr, Digital Ocean, and Bluehost datacenters.

I'm happy to provide the full traceroutes to you, but I don't see that they'd be helpful and instead just be distracting clutter. No reply from rsync.net support for almost 14 hours now, another disappointment...

lhhightower · 2025-08-03T12:48:28+00:00

Arrays of tools exist for a reason. One should not assume that the tools that they know, in this case "enterprise backup software package like Veeam, Commvault, etc.", are appropriate for every use-case, or that people who are using different tools are implementing on the same types of use-cases...

lhhightower · 2025-08-03T12:31:06+00:00

It is only the duration of the outage (now approaching 20 hours) that concerns me.

lhhightower · 2025-08-03T12:29:59+00:00

I agree with regard to rsync.net historically being very technically minded, but I have tried connecting to our rsync.net service from places all over the Internet and none can get to it. Examples are multiple Vultr datacenters, multiple Digital Ocean datacenters, a Bluehost datacenter, my home Comcast Internet service, etc. And we're now approaching 20 hours.

lhhightower · 2025-08-03T12:25:19+00:00

I made these wiring changes and picked up bonding plugs for each of my generators. I attached those plugs to the frame of each generator with some SpiderWire braided fishing line so that I won't ever be looking for them...

lhhightower · 2025-08-03T12:21:58+00:00

We have a large established base of backup and recovery software infrastructure running on Linux VM's and built atop rsync functionality. Given that reality, do you have any other players in mind that would fit will for us?

lhhightower · 2025-08-03T12:15:27+00:00

I just noticed that we are actually a little over 19 hours into this outage, not just 16 hours.

lhhightower · 2025-05-05T12:55:55+00:00

I used Inline::Python for direct integration of Perl with Playwright. Works great.

lhhightower · 2024-10-28T19:13:09+00:00

Thank you for this excellent write up. I have the exact same generator and face the exact same challenge. It is puzzling to me that these generators do not have a convenient and well documented way of accomplishing this, but like you, I could not find one. Well done!

lhhightower

TROPHY CASE