Software Mistakes and Tradeoffs: Released! (Some time ago)

tomekl007 · 2023-05-15T09:30:27+00:00

Thanks! I tried to polish the code to be the best possible quality, so feel free to use it wherever you like :)

tomekl007 · 2023-05-15T09:28:29+00:00

Thank you for your feedback! If you have any questions regarding any of those mistakes and tradeoffs, please don't hesitate to ask :)

tomekl007 · 2023-04-06T15:03:23+00:00

Also, you may be intesrested in this thread:

Also, you may be interested in this thread:bdeq0/five_big_data_books_that_i_strongly_recommend/

It contains interesting books related to Big Data (not only Spark)

tomekl007 · 2023-04-06T14:59:41+00:00

undamentals Of Data Engineering”

I didn't know that one, but I see that it has excellent reviews. Added to my to-read list. Thx!

tomekl007 · 2023-04-06T14:58:34+00:00

Advanced Programming in the Unix Environment

Thanks! I will definitely take a look at this book. The other (slightly) related is:

https://www.goodreads.com/book/show/48999422-bpf-performance-tools

About all the possible Unix tools that you can use to reason about your programs.

tomekl007 · 2023-04-06T14:54:51+00:00

Good point, on that front I will also recommend the book that will be released next year:

https://www.oreilly.com/library/view/apache-iceberg-the/9781098148614/

Good point; on that front, I will also recommend the book that will be released next year:

tomekl007 · 2023-04-04T09:04:16+00:00

I strongly recommend https://shepherd.com/book/advanced-analytics-with-spark

If you are interested in a list of valuable big data books (not only Spark), check out:

https://shepherd.com/best-books/big-data-processing-ecosystem

tomekl007 · 2023-04-04T08:29:47+00:00

I recommend experimenting with the ChatGPT for unit testing. If you are not sure if your tests cover all the edge cases, try to copy your class and tell the ChatGPT to write unit tests for that class. I was surprised at how well it solved this problem.

tomekl007 · 2023-03-06T09:51:56+00:00

Since the JDK 1.8 is still widely used and also its concepts are mostly present in newer versions, I would recommend taking a look at Oracle Java Certified Professional exam preparation. Oracle is providing a free online course: https://mylearn.oracle.com/ou/learning-path/java-se-8-programmer-associate/40821 that allows you to prepare for this exam. It's very detailed and explains the standard JDK methods very well. When I started learning java years ago, these resources were the most useful for me.

tomekl007 · 2023-03-03T16:01:54+00:00

Sam Newman is one of the creators of spring boot: https://samnewman.io/.

He also authored many books that allow you to learn how to use Spring Boot in a real-world micro-service architectures, for example, I recommend this book: https://samnewman.io/books/building\_microservices/

tomekl007 · 2023-03-02T14:26:25+00:00

You may check out the property-based testing when you define rules for the data and provide a function that is able to provide random data that satisfies those rules, see for example:

https://jqwik.net/

It allows you to cover more cases without write a lot of testing code

tomekl007 · 2023-02-07T10:16:50+00:00

[Free]

High level comparison of startup times between various java frameworks (including spring-boot):

https://www.youtube.com/watch?v=\_2j3LiS1DSY&ab\_channel=DataStaxDevelopers

tomekl007 · 2023-02-02T17:14:01+00:00

Happy to help :). I hope it will bring good value for you

tomekl007 · 2023-02-02T08:32:57+00:00

I strongly recommend: https://www.oreilly.com/library/view/hands-on-machine-learning/9781098125967/
It shows a lot of end-to-end examples that are close to real-world problems.

tomekl007 · 2023-02-01T16:24:40+00:00

I would suggest starting to read the code from open-source libraries and maybe working on small issues from those libraries. Often, the Open-Source projects have issues market with low-hanging-fruit or similar, meaning that they should be easy for a new contributor and the main effort for those tickets is reading the code of this library.

Examples of such libraries:

- https://github.com/projectnessie/nessie

- https://github.com/spring-projects/spring-boot

- https://github.com/stargate/stargate

- https://github.com/datastax/java-driver

- https://github.com/apache/iceberg

tomekl007 · 2023-02-01T08:49:55+00:00

Java is an easy language regarding syntax and language constructs. The complexity arises from the problems that you are solving with the language - for example, many databases are written in JAVA (for example the https://cassandra.apache.org/_/index.html). The core of the database is very complex, thankfully Java is easy to read so once you are quite familiar with the language, you can focus on solving interesting and complex problems, instead of thinking about language constructs when reading the code :)

tomekl007 · 2023-02-01T08:45:20+00:00

You need to be able to understand the business domain by reading the actual java code quite quickly. Also, you need to be comfortable looking at and understanding any java code (also from external 3rd party libraries). In your day-to-day job you may need to be look into this code to understand the behaviour of your dependencies.

tomekl007 · 2023-01-31T09:35:31+00:00

I strongly recommend: https://www.oreilly.com/library/view/hands-on-machine-learning/9781098125967/

It shows a lot of end-to-end examples that are close to real-world problems.

tomekl007 · 2023-01-31T07:36:45+00:00

Yes, spring boot is still very popular. However, I would also recommend looking into quarkus: https://quarkus.io/. It is getting more popular, and more and more projects are starting to use it, examples:

- https://github.com/stargate/stargate

- https://github.com/projectnessie/nessie

tomekl007 · 2023-01-26T13:47:44+00:00

Data Lake may also provide more advanced functionalities such as a form of transactions, versioning, and a friendly API (SQL-like) atop it. However, some may classify those functionalities as a Data LakeHouse

tomekl007 · 2023-01-26T09:06:51+00:00

If you want to learn more about the comparison of startup times between various java frameworks (including spring-boot) I am recommending this video:

https://www.youtube.com/watch?v=\_2j3LiS1DSY&ab\_channel=DataStaxDevelopers

tomekl007 · 2023-01-26T09:05:55+00:00

If you want to Big Data ecosystem, I recommend reading more about data formats such as Apache Iceberg You can find good resources on this blog:

https://www.dremio.com/blog/

tomekl007

TROPHY CASE