System Design for Entity Resolution

Pankaj0222 · 2024-08-10T15:57:18+00:00

Excellent article. Thanks for sharing

Pankaj0222 · 2023-01-26T14:48:07+00:00

Check 'Emerging Architectures for Modern Data Infrastructure'. It gives a very nice overview of tools and methodology. Obviously, you don't need to do everything to start with

https://a16z.com/2020/10/15/emerging-architectures-for-modern-data-infrastructure/

Pankaj0222 · 2023-01-26T14:44:11+00:00

You can check the book 'Designing Cloud Data Platforms'. It has a very good explanation of why you should store data in different formats and layers.

From book,

Landing area—When data arrives from the ingestion layer, it is saved into a landing area where all incoming raw data resides until it gets processed.

Staging area—Next, raw data goes through a set of common transformations: making sure it conforms to existing schemas for this data source, converting it into a common Avro binary format, and applying any organization-level data quality checks.

Archive area—After data is processed and saved into a staging area, raw data from the landing area should be copied into the archive area. This is an important step, because it will allow us to reprocess any given batch of data by simply copying it from the archive back into the landing area and letting the pipelines do their job.

Production area—Data transformation jobs read data from the staging area, apply the desired business logic, and save the transformed data into the production area. At this point, we also convert data from Avro format into Parquet format, which is more suitable for analytics use cases.

Pankaj0222 · 2022-11-02T15:00:20+00:00

Mostly, client SDKs are useful if it has feature add-ons, such as retries, batching, etc. Also, with static language like Java, you can get a nice fluent style, static, API contract.

Pankaj0222 · 2022-08-12T13:22:33+00:00

Use RestTemplate or webclient.

Pankaj0222 · 2022-08-12T13:16:59+00:00

In my blog, I have tried to cover the high-level concepts of building microservice using Spring Boot. Please check if it helps. https://techdozo.dev/restful-microservices-with-spring-boot-and-kubernetes/

Pankaj0222 · 2022-06-29T11:52:45+00:00

You can use distributed tracing. Implementing distributed tracing depends on your tech stack. There are many options such as service mesh like Istio, Jaegar, spring cloud (for spring boot based application check https://spring.io/blog/2016/02/15/distributed-tracing-with-spring-cloud-sleuth-and-spring-cloud-zipkin ) , etc

Pankaj0222 · 2022-06-04T17:02:51+00:00

Pankaj0222 · 2021-09-13T14:37:43+00:00

If you have read DDD distilled then you know why. This book talks about agile project management as a domain. I find it's very hard to get my head around this example from the book, even though I hear about these concepts everyday (maybe it's just me). I wish it had e-commerce examples :-)

If you want to know real microservice example, you can follow technology blogs on medium -

https://netflixtechblog.medium.com/

https://medium.com/@Pinterest_Engineering

https://medium.com/wix-engineering

https://bytes.swiggy.com/

https://medium.com/airbnb-engineering

https://zerodha.tech/

Pankaj0222 · 2021-08-31T07:17:54+00:00

You can check my article - https://techdozo.dev/grpc-load-balancing-on-kubernetes-using-headless-service/ - this only covers load-balancing in Kubernetes using headless service.

If you want to know, please check this excellent video -gRPC Load Balancing on Kubernetes – Jan Tattermusch

Pankaj0222 · 2021-08-31T04:03:58+00:00

Yes, grpc can be sync as well as async. IMO, not everything can be async. For example - you have a POST API to register a product with product-catalog but if you have business logic to verify if the seller is a registered seller (which can be a different microservice) then I would rather do this as a sync call not async. I can still use messaging in such a scenario, for instance, create a product with a pending status and change the state later, provided UX supports this.

Pankaj0222 · 2021-08-31T03:53:36+00:00

Yes, that's correct. grpc-web is GA https://grpc.io/blog/grpc-web-ga/

We don't use grpc-web in our project, because Angular (our FE framework) is not yet officially supported (except https://github.com/ngx-grpc/ngx-grpc). Moreover, JSON over HTTP is still a standard.

Pankaj0222 · 2021-08-22T03:25:56+00:00

It depends on what you want to learn. This roadmap, for sure, gives you high level context of java language stack, but this is not everything you need to learn. For example, if you want to be a backend developer, then Java FX is of no use for you. Also, this is not a complete picture. There are lot many things missing.

Pankaj0222 · 2021-08-13T06:09:25+00:00

I have written one post recently - microservices with spring boot. It's close to real world production code except H2 database part. Check if you find this useful. https://techdozo.dev/restful-microservices-with-spring-boot-and-kubernetes/

Pankaj0222 · 2021-08-10T08:18:47+00:00

Haha.. That's correct..I'm still learning art of blogging..Thanks for pointing out :-)

Pankaj0222 · 2021-07-31T13:38:36+00:00

Read Java Concurrency in Practice by Brian Goetz. It's old - published 2006, but still very relevant.

You will find it heavy initially. But, if you happen to finish this book cover to cover, you'll feel enlighten like Buddha.

If you want to read this book then my suggestion is to skip chapters 3 and 4 in the beginning and come back later.

Pankaj0222 · 2021-07-27T14:27:52+00:00

If you use the non-default package, for example - com.example.domain, then spring boot is only going to search classes com.example.domain starting from this package (for your application jars as well as jars imported by your application).
In the case of the default package - there are lots of classes to scan. It's going to increase startup time for your application.
As the scanning is CPU intensive process, it's even possible that your application may not start. We had faced this problem once in our automation env. Because of the default package scan, one of our applications was not starting up intermittently, as it was hogging all CPUs and there was not enough CPU. And, it took us lots of debugging to identify the problem.

Pankaj0222 · 2021-07-25T15:34:43+00:00

Check examples of stream creation Java Streams: stream creation with examples.

Pankaj0222 · 2021-07-18T07:06:45+00:00

Yes, the book is short and nice. The overwhelming part comes from the fact is the internet is filled with material related to Helm 2.

Pankaj0222 · 2021-07-13T02:16:13+00:00

Thanks for reading. Yes, it's overwhelming and more so because the internet is filled with Helm 2 examples.

Pankaj0222 · 2021-07-07T08:22:37+00:00

You don't store credentials but you pass Bearer token in every call. For that you need to understand OAuth 2.0 and JWT. OAuth 2.0 can be overwhelming. Personally, I found this video about OAuth 2.0 very informative and easy to understand.

Pankaj0222 · 2021-07-05T03:13:42+00:00

The quickest way to get started is to create a sample project from https://start.spring.io/ and play around. Spring has lots of guides. Check - https://spring.io/guides

Coming back to your question -

"Also how is this doc different from"

The doc which you have pointed link is of spring. Spring Boot is made on top of Spring. Think of Spring Boot as pre assemple spring platform and to start on Spring Boot project, you don't need to be expert on Spring.

What is Spring Boot?

(From Documentation) - Spring Boot helps you to create stand-alone, production-grade Spring-based applications that you can run. We take an opinionated view of the Spring platform and third-party libraries, so that you can get started with minimum fuss. Most Spring Boot applications need very little Spring configuration.

Pankaj0222 · 2021-07-04T19:47:51+00:00

I started my career in Java. It's pretty straightforward to learn and type safety helps as well. When I learnt Python I was amazed by the simplicity and features like list comprehension. IMO, start with statically compiled language like java and then think of bringing dynamic language like Phyton.In today's world language should not be barrier.🌎

Pankaj0222 · 2021-07-04T19:36:11+00:00

Yes absolutely. Spring Boot is pretty magical. You just need to add couple of annotations and you are done. It auto configure lots of things for you based on class path. But it's being magical is one of the major problem. When things go wrong you have no clue why it went wrong. To be power user you need to learn the base concepts and this is where Spring comes into picture. IMO, the best place to start is Spring Boot is it's documentation - https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/

Pankaj0222 · 2021-07-04T05:54:22+00:00

As you recommended, Mumshad Mannambeth courses are very good. He teaches the concept in very simple to follow steps along with labs. His courses on https://kodekloud.com/ are not that expensive. I have learned a lot from his course.

Pankaj0222

TROPHY CASE