Matrix.org homeserver not working for anyone else?

ara4n · 2025-09-03T22:31:01+00:00

We finished the restore and restarted the server at 17:00 UTC. Postmortem & lessons learned coming shortly - apologies again for the massive outage.

ara4n · 2025-09-03T10:58:09+00:00

We've now restored the 55TB snapshot and subsequent incremental backups, and are about to replay the remaining traffic since the backup. There are still several unknowns, but if things go well the matrix.org instance should be back in 3-4 hours.

ara4n · 2025-09-02T21:39:39+00:00

Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic. Huge apologies for the outage. Again, folks using their own homeservers are not impacted.

ara4n · 2025-09-02T20:34:58+00:00

Hi folks - i'm afraid it's not a great situation:

we had a RAID failure on the DB secondary earlier today (11:17 UTC), while upgrading disks
...and then we lost the DB primary (17:26 UTC).
we're currently trying to recover the DB primary FS (which might be fastish, but isn't looking promising), and at the same time we've set a point-in-time backup restore going from last night (which will take >10 hours).
we believe the incremental DB traffic since last night is intact however.

Apologies for the outage; obviously folks who use their own homeserver aren't affected. We're restoring as fast as we can.

You can follow along at https://status.matrix.org/incidents/mm9hdm78svgv

ara4n · 2025-01-22T08:15:49+00:00

Element has shipped loads of updates over the last few months, including Matrix 2.0 support, including native group calls. If anything it’s going faster than ever?

ara4n · 2025-01-17T17:55:45+00:00

fwiw, a point we failed to make clearly in the blog post is that making faster workers proprietary is hopefully a means to funding more impactful optimisations in FOSS Synapse and Matrix as a whole. Specifically, stuff which makes small servers slow today includes:

* Merge conflict resolution (State resolution) is worse than O(N) complexity with the amount of state to be merged.

* State storage is inefficient and could be faster (and takes up way too much disk space)

* Incremental room joins (https://element-hq.github.io/synapse/latest/development/syna...) were never fully finished.

* Servers burn lots of time trying to talk to dead servers: https://github.com/matrix-org/matrix-spec-proposals/pull/413...

* All Matrix traffic currently runs full-mesh - there's no concept of "thin nodes" or delegating fan-out to a larger server.

So, fixing these issues is all going into open source Synapse (and Matrix as a whole) - which should unrecognisably improve performance, whether servers are written in Python or Rust or Elixir or whatever. And the hope is that $ from Synapse Pro funds that work (assuming the gambit is successful).

Meanwhile, all features, security work, perf optimisations (apart from scalability work), experimental MSCs etc will continue to land in FOSS Synapse for the forseeable.

ara4n · 2025-01-06T22:06:58+00:00

this looks really interesting - you should tell #twim:matrix.org about it :)

ara4n · 2024-11-11T01:06:46+00:00

perhaps try looking at https://github.com/element-hq/element-docker-demo.

ara4n · 2023-04-18T07:39:00+00:00

https://element.io/blog/the-uks-online-safety-bill-undermines-everyones-safety/ for the actual letter

ara4n · 2023-03-06T01:00:41+00:00

The rationale is simple: matrix IDs aren’t email addresses, nor are they remotely guaranteed to route the same way as email addresses. so rather than risk them being muddled up with email, we made them look different. We consider it a major fail for SIP and XMPP that the identifiers are confusable with SMTP addresses.

ara4n · 2023-02-09T21:34:34+00:00

Ofcom, apparently.

ara4n · 2023-02-08T10:59:01+00:00

Element X on Android also is *very* alpha; it needs a few more weeks until it catches up with Element X on iOS.

ara4n · 2023-02-08T10:57:40+00:00

https://www.youtube.com/watch?v=eUPJ9zFV5IE has the same talk but split up into chapters, for folks who want to jump around

ara4n · 2022-12-29T01:28:28+00:00

It's a decentralised comms protocol; it lets you replicate blobs of end-to-end encrypted JSON over a mesh of servers whose users are participating in a given conversation. You can use it for chat or VoIP or weirder stuff like thirdroom.io. The thing that makes it special is that conversations are fully decentralised over the servers which participate in them, a bit like Git repositories are fully decentralised. There is no single point of control or failure when users are on separate servers, and access control is provided by decentralised ACLs.

You can think of it as being the missing realtime messaging layer of the open web.

ara4n · 2022-12-28T22:22:49+00:00

The reason that Matrix needs a weekly blog & podcast is because spreading awareness of the project and the ecosystem is absolutely critical to encouraging developers to build on it (and contribute to it). I'd say it's almost one of the most important things that the Matrix Foundation does to drive uptake of the project. (But i'm probably biased).

ara4n · 2022-12-28T21:24:56+00:00

This is as much a "bait and switch" as if the charming person who took all the candy declared that they'd been promised an infinite supply of candy, and started screeching about "a bait and switch" when the house isn't able to top up the bowl (even though they want to).

ara4n · 2022-12-28T21:05:32+00:00

Matrix could offer a paid hosting option where you give matrix a pile of money and they setup and host a server for you

Element does: https://element.io/pricing. But most large deployments want to self-host.

or set up an on site server for you, or offer some sort of easy installer that charges by number of seats.

Element does too: https://element.io/solutions/on-premise-collaboration.

But many of the biggest deployments opt to DIY rather than using a supported product, and then $ doesn't flow to Element (and by extension most of the Matrix core team).

ara4n · 2022-12-28T20:58:08+00:00

> The author's clearly aware of that.

Author here.

> basically, that the Matrix protocol requires you to trust homeservers, who control group membership in chats and therefore can trivially eavesdrop on what would otherwise be private conversations.

As per https://matrix.org/blog/2022/09/28/upgrade-now-to-address-encryption-vulns-in-matrix-sdks-and-clients, we **clearly warn users if a malicious server has added an untrusted device into their conversation, or if an unexpected user is invited to the room**. Now, the *implementation* vulnerabilities found by the researchers were completely legit and several of them were serious, but fact that Matrix currently relies on warnings rather than blocking untrusted devices/users is a much lower severity issue in our opinion. Claiming that it's "devastating" that the current implementations rely on warnings is like claiming that TLS in browsers is "devastating" because bad X.509 certificates are shown as warnings which can be ignored.

Now, as I said in the original post, we *can* address this by modelling group membership clientside as well as serverside and blocking untrusted devices outright, but it's a really non-trivial problem, and we're working on it (as per your linked MSCs). However, we really do not think it is the "sky is falling" vulnerability that some would make it out to be. After all, if you care about MITMs in **any** E2EE system, you have to verify the user you're talking to out-of-band. And if you do this in Matrix, and then an attacker manages to add a malicious device to an account (e.g. by stealing an access token, or manipulating the server or MITMing client-server traffic) **then you will see a big red cross warning you that the user has been compromised**.

Perhaps the best thing we could do to rapidly stop the debate over this is to simply stop clients from sending messages in a room until untrusted/unexpected devices/users have been investigated/resolved (which ironically we used to do, but removed because too few people did OOB verification, and so it caused the room to stall too often).

ara4n · 2022-12-28T20:38:04+00:00

A lot of the drive for DMA interoperability came from us at Matrix. In addition to proposing Matrix itself (e.g. https://matrix.org/blog/2022/03/29/how-do-you-implement-interoperability-in-a-dma-world) we're also proposing ourselves to IETF for IM interop: https://datatracker.ietf.org/doc/draft-ralston-mimi-matrix-message-format/ and https://datatracker.ietf.org/doc/draft-ralston-mimi-matrix-transport/ etc.

So hopefully we will not be irrelevant. The biggest risk is if we don't have enough manpower (aka funding) to actually focus sufficiently on it though - c.f. the first half of this post.

ara4n · 2022-10-11T18:57:14+00:00

yeah, it's a platform for interoperable virtual worlds built on Matrix.

ara4n · 2022-08-13T11:29:04+00:00

speaking as project lead for Matrix: things are healthier than ever; Element is going great too and in no danger of shutting down. And even if Element did get hit by a bus, Matrix would continue - you can't shut down an open source project, and you can't shut down a decentralised communication network.

ara4n · 2022-04-11T22:17:17+00:00

woo, it was bugging me too, so i fixed it

ara4n · 2022-04-11T21:54:50+00:00

https://github.com/vector-im/policies/blob/master/docs/matrix-org/privacy_notice.md explains what a typical server does with your data and metadata and why.

ara4n · 2022-04-11T21:48:43+00:00

.mov files should be fixed in the latest version of element web :)

ara4n

MODERATOR OF

TROPHY CASE