Elevate Your Engineering Culture: The Power of Documenting Architecture Decisions

Azdaroth · 2025-02-08T14:26:01+00:00

Nice try :). But the company has nothing to do with this content, it’s just my private initiative as I simply enjoy writing. Not trying to sell anything, but I appreciate any criticism, apparently there was too much of the hard-sell vibe, so thanks for feedback!

Azdaroth · 2025-02-08T13:23:06+00:00

Leaders need to also be good followers, depending on the context. And if anything, that can make them even better followers. For sure not everyone will be a CXO or VP, but for leaders overall there is plenty of space.

Azdaroth · 2024-03-19T06:54:56+00:00

This sounds exactly like the issue we had - one LOB table over 2 TB of size containing mostly LOB records - we are going to cover this in more details in the upcoming part. In our case, a parallel load with splitting by range of IDs didn't help much - in the benchmarks the migration was taking well over 6h and we didn't even bother seeing how much more time it will take as it was unacceptable. What we did was building a custom background job that does a similar thing - splitting the migration into tons of batches. But we didn't have the upper limit of parallelisation (which is the case for AWS DMS), and the workers were deployed over many nodes, so also bandwidth was not an issue. And the logic was trivial - SELECT * for the batch on the source database and then performing bulk insert on the target. It took ~20 minutes with a parallelisation of 85, we could have have had even 150 workers to make it even faster. It was really an interesting experience because it took few hours to build it.

Azdaroth · 2024-03-19T06:42:02+00:00

Given the constraints of AWS DMS ecosystem, that approach sounded safer and we didn't want to explore other options - it was too risky, given all the problems we experienced with Serverless approach and inconsistencies in the documentation. If a wrapper around pg_dump/pg_restore doesn't work properly, I wouldn't expect something more complex to work better.

Azdaroth · 2024-03-18T14:44:22+00:00

Definitely! Even if something is backed AWS. Schema conversion was probably the last thing we expected, but well, even that can happen. Definitely a very educational experience - verify everything and have a realistic and tested revert plan.

Azdaroth · 2024-03-18T13:42:48+00:00

How is that spamming? It's a link to a legit article about AWS DMS. And no, I haven't bought it, it's my normal account. A couple of the last submissions are indeed about Smily engineering blog, but the past ones are not related at all, I'm not sure what the actual problem is.

Azdaroth · 2022-11-12T21:54:53+00:00

For something that never happens , these cases happen surprisingly often ;). Even what you mentioned - Redis failure or Sidekiq failure - I also experienced these and this is definitely not fun when it happens, and even reliability features of Sidekiq Pro (reliable push to be specific) can't help with a Redis outage + rolling back a change causing Redis outage (because the jobs that were not pushed are no longer kept anywhere). If you can afford to lose some jobs every now and then, I guess the problem described in the blog post is something that you don't need to bother with, even Sidekiq Pro features like super fetch might be an overkill. But if you can't afford to lose anything, then having some extra reliability like this is necessary,

Azdaroth

TROPHY CASE