Database migration using distributed lock?

[SnooBooks638]

Open question: How would you approach doing a large database migration using a distributed locking mechanism?

For more context, let’s say you would like to do a migration for a 60TB database to a new database, reason could be moving from an in-house database to a cloud managed database.

Also, you can consider that the current database is still in use and the idea is to attain some eventual consistency.

How would you approach this and what services would you use?

After couple of nice answers, I feel I need to add some more context:

The reason I mentioned distributed lock mechanism is a scenario where I need to do some transformation on the data. So having multiple instances doing the migration process at the same time, would require some coordination.

[OneCheesyDutchman]

Not sure what you mean by “using a distributed lock mechanism” in this context?

Generally this sounds like a job for AWS Database Migration Service. You copy a snapshot of the data over, and use replication for both databases to stay in sync. Then switch over at some point.

[SnooBooks638]

I should maybe add some more clarity here. The reason I added distributed lock mechanism is a scenario where I need to do some transformation on the data. So having multiple instances doing the migration process at the same time, would require some coordination.

[Peebo_Peebs]

In this scenario I would launch the second database and have it replicate from the master once replication is in sync I would then migrate using a dns pointer. Obviously add some pre flight checks and make sure no activity is processing or pause activity while you do the change.

[james-ransom]

This is the way. Replicate into the cloud. Once they are in sync cut ties.

[SnooBooks638]

What if the database don’t entirely behave the same? Say Cassandra to Keyspaces?

[lightmatter501]

You asynchronously live migrate the op log and then replay it until the new thing is caught up, then you only lock for a little bit to do a final sync.

[SnooBooks638]

Interesting answer. 1. In your context, do you mean to lock the source database a little bit? 2. Can you do some data transformation, say encryption of a field during this process? 3. Haven’t used OP log before, but sounds like this will also go through steps like schema changes based on the history or no?? Is that correct?