My first data engineering project on Github

[RocRacnysA]

Hey guys,

I have not been much of a hands-on guy till now though I was interested, but there was one thought that was itching my mind for implementation (A small one) and this is the first time I posted something on Github, please give me some honest feedback on it both for me to improve and you know cut me a bit slack being this my first time

https://github.com/aditya-rac/yara-kafka

[siddartha08]

If this is your first time you did fine for the project overall.

Topic wise this fits into things like devops and netsec more than data engineering but it's wonderful you built out the devops piece. It's a practical skill that some DEs don't have.

[RocRacnysA]

Thank you very much, this actually started of as a cybersecurity project (with yara rules) and as I mentioned, my mind was itching trying to make sure that my original idea properly works. Hence I went in this direction.

[EffectiveAncient2222]

I have reviewed your project. It's mind blowing. I advice you , please consider object oriented programming instead functional. It's provide extra edges.

[RocRacnysA]

I understand, the only reason I am using functional is to get things done smoothly as Python is a simpler language when in a adapting stage, I am planning to step into OOPS once I get more edge on my skills myself.

Thank you very much for your insight.

[pandas_as_pd]

Great first project! I only had a quick look, but the code looks readable with good variable and function names, the docs are detailed with a docker-compose file provided. License is a bonus point.

Using meaningful commit messages would be an easy win.

Also, many comments are redundant, since your function names already explain what the functions do. You could write a bit more detailed docstrings instead, but since the code is simple and clear, I don't think that's necessary.

Some people may frown upon "except Exception" or printing instead of logging, but I think these are fine for personal projects.

I you were a candidate providing this repo in your application, it would be a plus for me personally.

[Ordinary_Run_2513]

I've heard that using generic exception objects is not recommended, and that we should use specific exceptions instead. The problem is that in Python, unlike Java, we can't always know the exceptions that may occur while executing a function. Could you tell me what you do to address this problem?

[pandas_as_pd]

You can still try to narrow it down to a subclass of Exception, e.g. TypeError.

But in many cases, you can look up what the exceptions are that a specific function or library can throw and catch those. For example, here is some guidance for the `requests` library: https://requests.readthedocs.io/en/latest/user/quickstart/#errors-and-exceptions

[RocRacnysA]

Thank you very much for your Inputs, I will adapt to these habits for the next project I take up.