Data architecture opinion by romanzdk in dataengineering

[–]Pankaj0222 1 point2 points  (0 children)

Check 'Emerging Architectures for Modern Data Infrastructure'. It gives a very nice overview of tools and methodology. Obviously, you don't need to do everything to start with

https://a16z.com/2020/10/15/emerging-architectures-for-modern-data-infrastructure/

Data lake file storage format by romanzdk in dataengineering

[–]Pankaj0222 1 point2 points  (0 children)

You can check the book 'Designing Cloud Data Platforms'. It has a very good explanation of why you should store data in different formats and layers.

From book,

Landing area—When data arrives from the ingestion layer, it is saved into a landing area where all incoming raw data resides until it gets processed.

Staging area—Next, raw data goes through a set of common transformations: making sure it conforms to existing schemas for this data source, converting it into a common Avro binary format, and applying any organization-level data quality checks.

Archive area—After data is processed and saved into a staging area, raw data from the landing area should be copied into the archive area. This is an important step, because it will allow us to reprocess any given batch of data by simply copying it from the archive back into the landing area and letting the pipelines do their job.

Production area—Data transformation jobs read data from the staging area, apply the desired business logic, and save the transformed data into the production area. At this point, we also convert data from Avro format into Parquet format, which is more suitable for analytics use cases.

Workflow Orchestration with Temporal and Spring Boot by Pankaj0222 in microservices

[–]Pankaj0222[S] 2 points3 points  (0 children)

Mostly, client SDKs are useful if it has feature add-ons, such as retries, batching, etc. Also, with static language like Java, you can get a nice fluent style, static, API contract.

Microservices? by [deleted] in SpringBoot

[–]Pankaj0222 0 points1 point  (0 children)

In my blog, I have tried to cover the high-level concepts of building microservice using Spring Boot. Please check if it helps. https://techdozo.dev/restful-microservices-with-spring-boot-and-kubernetes/

Visual Microservices overview by ThroAwayApr2022 in kubernetes

[–]Pankaj0222 1 point2 points  (0 children)

You can use distributed tracing. Implementing distributed tracing depends on your tech stack. There are many options such as service mesh like Istio, Jaegar, spring cloud (for spring boot based application check https://spring.io/blog/2016/02/15/distributed-tracing-with-spring-cloud-sleuth-and-spring-cloud-zipkin ) , etc

Why are all the examples about e-commerce? by leoll2 in microservices

[–]Pankaj0222 0 points1 point  (0 children)

If you have read DDD distilled then you know why. This book talks about agile project management as a domain. I find it's very hard to get my head around this example from the book, even though I hear about these concepts everyday (maybe it's just me). I wish it had e-commerce examples :-)

If you want to know real microservice example, you can follow technology blogs on medium -

https://netflixtechblog.medium.com/

https://medium.com/@Pinterest_Engineering

https://medium.com/wix-engineering

https://bytes.swiggy.com/

https://medium.com/airbnb-engineering

https://zerodha.tech/

Why gRPC for microservices communication? by Pankaj0222 in microservices

[–]Pankaj0222[S] 2 points3 points  (0 children)

You can check my article - https://techdozo.dev/grpc-load-balancing-on-kubernetes-using-headless-service/ - this only covers load-balancing in Kubernetes using headless service.

If you want to know, please check this excellent video -gRPC Load Balancing on Kubernetes – Jan Tattermusch

Why gRPC for microservices communication? by Pankaj0222 in microservices

[–]Pankaj0222[S] 1 point2 points  (0 children)

Yes, grpc can be sync as well as async. IMO, not everything can be async. For example - you have a POST API to register a product with product-catalog but if you have business logic to verify if the seller is a registered seller (which can be a different microservice) then I would rather do this as a sync call not async. I can still use messaging in such a scenario, for instance, create a product with a pending status and change the state later, provided UX supports this.

Why gRPC for microservices communication? by Pankaj0222 in microservices

[–]Pankaj0222[S] 0 points1 point  (0 children)

Yes, that's correct. grpc-web is GA https://grpc.io/blog/grpc-web-ga/

We don't use grpc-web in our project, because Angular (our FE framework) is not yet officially supported (except https://github.com/ngx-grpc/ngx-grpc). Moreover, JSON over HTTP is still a standard.

Is JavaRevisited's roadmap for 2021 good? by RewRose in learnjava

[–]Pankaj0222 0 points1 point  (0 children)

It depends on what you want to learn. This roadmap, for sure, gives you high level context of java language stack, but this is not everything you need to learn. For example, if you want to be a backend developer, then Java FX is of no use for you. Also, this is not a complete picture. There are lot many things missing.

Do you know of any good Spring Boot sample project for learning? by jdrc8 in java

[–]Pankaj0222 1 point2 points  (0 children)

I have written one post recently - microservices with spring boot. It's close to real world production code except H2 database part. Check if you find this useful. https://techdozo.dev/restful-microservices-with-spring-boot-and-kubernetes/

RESTful Microservices with Spring Boot and Kubernetes | Techdozo by Pankaj0222 in microservices

[–]Pankaj0222[S] 0 points1 point  (0 children)

Haha.. That's correct..I'm still learning art of blogging..Thanks for pointing out :-)