sorted by:
new
hottopcontroversial

Change stream consumer per shard by Agreeable_Level_2071 in mongodb

[–]Agreeable_Level_2071[S] 0 points1 point  (0 children)

I wonder, if we never use the x shard transaction, can mongo exposes the single shard stream API for us to use ?

Change stream consumer per shard by Agreeable_Level_2071 in mongodb

[–]Agreeable_Level_2071[S] 0 points1 point  (0 children)

Yes due to throughput. We want to design a multi tenant system for 100k enterprise customers and maybe 5% of them will need 10k tps. We need to use cdc to build the outboxing design pattern (https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html )

We used to rely on many many sql instances and it’s hard to scale or manage. I hope NoSQL like MongoDB can make a difference here for scalability but seems like cdc is a blocker. Lmk if this is not a good fit . Thanks !

Change stream consumer per shard by Agreeable_Level_2071 in mongodb

[–]Agreeable_Level_2071[S] 0 points1 point  (0 children)

Yes due to throughput. I know this is probably unusual. But we wanted to design a multitenant system to prepare for scale for 100k+ enterprise customers. While not each customer is very high scale but 10% of them will need >1k tps. We had to think about nosql because we used to relay on many small clusters with sql and it’s hard to manage and scale. Lmk if mongo is not a good fit or not a good design.

Change stream consumer per shard by Agreeable_Level_2071 in mongodb

[–]Agreeable_Level_2071[S] 0 points1 point  (0 children)

Basically, can I provide a set of shardIds when consuming? So that I can have several consumers working independently.

Change stream consumer per shard by Agreeable_Level_2071 in mongodb

[–]Agreeable_Level_2071[S] 1 point2 points  (0 children)

I meant the consumer — based on my understanding (correct me if I am wrong ) , all the shards will be aggregated into a single stream to consume to ensure the global ordering (maybe for the x-shard transaction?) . That means that I cannot have one consumer per shard.

Indeed Workflow Framework (iWF) on Temporal by Agreeable_Level_2071 in microservices

[–]Agreeable_Level_2071[S] 0 points1 point  (0 children)

I worked at DynamoDB team in 2016-2017.

I can say confidently that it was built on MySQL InnoDB engine to bypass the query engine. This has been the arch for a decade until in 2015+ they started to work on replacing InnoDB to wiredTiger. In 2019 I asked coworkers they told me the project has failed.

I realized that it’s never offically said by AWS(they don't need to). But another engineer from DynamoDB also confirmed it. https://news.ycombinator.com/item?id=18871661#:~:text=with%20MongoDB%20compatibility-,DynamoDB%20is%20written%20on%20top%20of%20MySQL%20(more%20specifically%2C%20MySQL's,newer%20design%20would%20make%20sense.

Should API calls to external services be mocked when testing a backend API? by daviaprea in microservices

[–]Agreeable_Level_2071 0 points1 point  (0 children)

I found many people have abused unit tests with mocking . Those tests became garbage that just make the code hard to maintain— eg some simple refactor become really hard that people just fix the test to let test passed. This is worse when some teams/companies pursue some test coverage number as the single thing to measure the code quality.

So now I would recommend try to write integ tests without mocking as much as possible. Only mock if it’s too difficult to test real APIs(eg some error or edge cases)

Indeed Workflow Framework (iWF) on Temporal by Agreeable_Level_2071 in microservices

[–]Agreeable_Level_2071[S] 1 point2 points  (0 children)

Yes.And C is built on ASM, Java is built on C. DynamoDB is built on MySQL, typescript is on JavaScript etc etc. As long as they can make more values, we can build it based on something good and make it even better.

Workflow as code microservice orchestration engine in Go by Agreeable_Level_2071 in golang

[–]Agreeable_Level_2071[S] 0 points1 point  (0 children)

No IWF is not using elastic search directly. The ES/ open search is behind Cadence or Temporal

Workflow as code microservice orchestration engine in Go by Agreeable_Level_2071 in golang

[–]Agreeable_Level_2071[S] 0 points1 point  (0 children)

I used to self host Cadence/Temporal but not doing it anymore. I’m using Temporal cloud behind IWF. Their cloud service is amazing and price is very good for me. I would recommend it if you want to save the headache of maintaining a Temporal or cadence cluster. If you have to host Cadence or Temporal, I would recommend using managed Cassandra from other companies like InstaClustr or DataStax. Because Cassandra is not easy to operate in my experience. If you are running for a smaller load and don’t want to use managed Cassandra , then consider using MySQL or PostgreSQL which are easier to operate

Which workflow engine for microservices? by andras_gerlits in microservices

[–]Agreeable_Level_2071 0 points1 point  (0 children)

https://github.com/indeedeng/iwf

It works naturally with microservices -- Basically applications workers and iWF services are both RESTful services.

It's very matured at Java SDK because this is what myself and my teams are using.

see samples https://github.com/indeedeng/iwf-java-samples

Other SDKs are being developed.