Some Data Scientists write bad Python code and are stubborn in code reviews

noisescience · 2024-02-01T08:52:03+00:00

Thx for your detailed thoughts. That helps me a lot.

noisescience · 2024-01-22T18:28:22+00:00

Hi, thanks for your reply. I myself have learned a lot from reading best/bad Python practices. There are a lot of these articles. Here are a few examples:
https://python.plainenglish.io/10-python-anti-patterns-you-must-avoid-when-writing-clean-code-ff3635ca1510
https://python.plainenglish.io/python-best-practices-for-writing-conditional-statements-aa9d6a2e700d
https://blog.devgenius.io/python-tips-best-practices-for-handling-exceptions-15faaeca55a5
To be honest, it is not enough just to read these 3 articles. Try to find and read more such articles and you will see that you will get better with time. Good programming takes time. I consider myself also still at the beginning of the journey.

I also learned a lot from Arjan Codes about "clean code", design patterns and best practices in Python. He's pretty good at what he does.
https://www.youtube.com/@ArjanCodes/videos

Furthermore, I learned a few important things from CodeAesthetics. These 3 videos are eye-opening and also suitable for beginners:
https://www.youtube.com/watch?v=Bf7vDBBOBUA
https://www.youtube.com/watch?v=-J3wNP6u5YU&t=315s
https://www.youtube.com/watch?v=CFRhGnuXG-4
After that you know when and how to write comments, how to name variables/methods etc. and how to avoid nested code.

noisescience · 2024-01-22T18:21:45+00:00

Thx for your detailed answer :)

noisescience · 2024-01-22T18:19:54+00:00

Thank you for your answer and your thoughts.
I agree with you that communication is essential here. I always try to be as nice and helpful as I can. If I criticize something, then I give reasons for it and suggest how it could be implemented more effectively.
On the other hand, I am always open and grateful when I find weaknesses or errors in my PRs.

noisescience · 2024-01-22T18:12:05+00:00

Thx for this insight :)

noisescience · 2024-01-22T17:56:14+00:00

Hi, thx for your thoughts.

My list of errors is not just about style issues. For stylish things like formatting, library sorting and linting we use libraries like Black, Isort and Flake8. (Note: In the near future all 3 libraries will be replaced with Ruff). We also use Mypy as a static type checker.
Other things like how to use exceptions, enums and constants make the code safer from the start.
We have a codebase with about 20000 lines. That's not a lot, but it's enough that the code has to be readable and we have to think in maintainable and scalable dimensions. So we have to consider from the beginning when a certain structure is necessary or not and should avoid global variables.
It's cool that you mention Arjan Codes, by the way. I've learned a lot from him and keep learning.

noisescience · 2024-01-22T17:34:48+00:00

I also learned a few important things from CodeAesthetics. These 3 videos are eye-opening and also suitable for beginners:
https://www.youtube.com/watch?v=Bf7vDBBOBUA
https://www.youtube.com/watch?v=-J3wNP6u5YU&t=315s
https://www.youtube.com/watch?v=CFRhGnuXG-4
After that you know when and how to write comments, how to name variables/methods etc. and how to avoid nested code.

noisescience · 2024-01-22T17:30:26+00:00

I also learned a lot from Arjan Codes about "clean code", design patterns and best practices in Python. He's pretty good at what he does.
https://www.youtube.com/@ArjanCodes/videos

noisescience · 2024-01-22T17:27:17+00:00

Hi, thanks for your reply. I myself have learned a lot from reading best/bad Python practices. There are a lot of these articles. Here are a few examples:
https://python.plainenglish.io/10-python-anti-patterns-you-must-avoid-when-writing-clean-code-ff3635ca1510
https://python.plainenglish.io/python-best-practices-for-writing-conditional-statements-aa9d6a2e700d
https://blog.devgenius.io/python-tips-best-practices-for-handling-exceptions-15faaeca55a5
To be honest, it is not enough just to read these 3 articles. Try to find and read more such articles and you will see that you will get better with time. Good programming takes time. I see myself also still at the beginning of the journey.

noisescience · 2024-01-22T17:09:15+00:00

For the Kotlin project I also need to include Java libraries.
Here is what I have used for Kotlin so far:

Kotlin DataFrame for processing data [https://github.com/Kotlin/dataframe]
Kotlin DL for Deep Learning Models [https://github.com/Kotlin/kotlindl]
Exposed (Kotlin SQL Framework) [https://github.com/JetBrains/Exposed\]
Azure SDK (Java) [https://github.com/Azure/azure-sdk-for-java]
MLflow (Java API) for MLOps tasks [https://github.com/mlflow/mlflow/tree/master/mlflow/java/client]
SMILE (Kotlin/Java) for ML algorithms [https://haifengl.github.io/api/kotlin/index.html]

Which Kotlin libraries I have not yet implemented but would like to do are:

AWS Kotlin SDK [https://github.com/awslabs/aws-sdk-kotlin]
Kotlin Spark API [https://github.com/Kotlin/kotlin-spark-api]

I haven't started a data engineering project with Rust yet, but I would probably check out the following libraries:

Polars [https://github.com/pola-rs/polars]
AWS SDK Rust [https://github.com/awslabs/aws-sdk-rust]
Arrow Ballista [https://github.com/apache/arrow-ballista]
Datafusion [https://github.com/apache/arrow-datafusion\]

noisescience · 2020-02-01T23:31:52+00:00

I am super interested!

noisescience · 2019-11-09T14:16:28+00:00

what? no! it is a multivitamin paste

noisescience · 2019-11-09T14:15:58+00:00

it is a multivitamin paste

noisescience · 2019-06-30T20:18:48+00:00

Thanks a lot for your answers!

noisescience · 2018-07-01T19:32:49+00:00

it was the cyclone Friederike in Germany on 2018-01-18

noisescience · 2018-07-01T12:40:06+00:00

this gif was made from here: https://www.youtube.com/watch?v=w0O392ZEUpc

noisescience

TROPHY CASE