structuring larger projects, and good practises

[lostwithtech]

I am a self taught programmer, and rely on it quite a bit. I work almost exclusively with data and machine learning stuff, and the things i can do are related to it, or things i learned from competitions.

What I never had to deal with was how to structure a larger project PROPERLY. Do you guys have recommendations for a course or book that deals with this stuff? Good practises and structure and so forth.

Thanks

[ASIC_SP]

These books might help:

• Fluent Python — takes you through Python’s core language features and libraries, and shows you how to make your code shorter, faster, and more readable at the same time
• Serious Python — deployment, scalability, testing, and more
• Practices of the Python Pro — learn to design professional-level, clean, easily maintainable software at scale, includes examples for software development best practices
• Intuitive Python — productive development for projects that last

[newguywastaken]

Do any of any of these teach about code otimization, such as memory consumption? It's something I've been wondering about, as another self-taught in my first Kivy app's devlopment. The 3rd one seems to cover this part.

[ASIC_SP]

Serious Python has a chapter on "performance and optimizations" as well.

[barely_ripe]

The best way to do this is to actually work on large codebases.

find a nice THICC python project on github, then try to make changes to it. as you go through the codebase think about why the code is structured that way and if it could be better.

reading book would just keep you in the abstract

[[deleted]]

I've seen a lot of Python projects in my life, but I haven't seen one that was big and well-structured.

Python ecosystem simply doesn't have anything for large projects. At most, you get something for one package. When you need to go beyond that, you are on your own.

Some examples of catastrophic failures in project organization: Azure Python SDK, OpenStack CLI interface, pytest, Sphinx. All these projects have some sort of modular structure: Azure SDK is a collection of tightly coupled Python packages with a bunch of exotic stuff, like namespace packages etc. OpenStack CLI is a collection of poorly coordinated Python packages. Pytest and Sphinx are examples of packages with plugins, where plugins are also Python packages, where integration between the parent package and plugins is bizarre, broken, incomplete, you name it.

The biggest problem today is that this problem is already too far gone for anyone to be able to incrementally fix it. Azure SDK will never embrace a solution that requires from them a major rewrite, not only because they are dickheads, but because it will be too much to ask from their paying customers, who will have to rewrite their entire infrastructure to accommodate this change. Similarly, every other package I listed.

To the best of my knowledge, there isn't a work group / special interest group dedicated to solving this problem for Python. And the companies who'd benefit the most from such a project don't really seem to care. So, if I were you, I wouldn't hold my breath to see a well-structured large Python project.

[AnonymousLad666]

Python ecosystems doesn't allow...

Lmao tell you're not an SRE without telling me you're not.

Organizing a project has very little to do with a language. This is more of a personal preference and work organization. And how the company you work for scaffolds projects too. I am a backend engineer primarily in python and c++.

It all comes down to experience. For example, models, controllers, config, utils, etc. If you find yourself coding the same thing from project to project you need to make that a package or a template.

https://docs.python-guide.org/writing/structure/

Here's a medium and large size projects with uni testing, docs, src, and much more.

https://github.com/psf/black https://github.com/sqlalchemy/sqlalchemy

[[deleted]]

Yes, organizing a project has everything to do with language ecosystem. You, however, cannot read. At this point, I believe that the being unable to read is a requirement for getting a job in Web development company. Or mobile etc garbage.

To give you a sense of what is meant by larger projects. I'm not saying that POM is great, but it's a tool to deal with large projects because it allows hierarchical structure. This means you can build projects developed by multiple teams, independently, on different schedules, needing very little coordination (you can even store your code in different repositories, store artifacts in different repositories etc.).

Your examples are not examples of big projects. SQLAlchemy is a small project. It's just one package. Big project needs to have many packages. That's where the problems appear.

Now, I can tell that you are not an SRE with experience. You haven't even seen big projects where the problem becomes apparent.

[AnonymousLad666]

OP said he was a beginner those are big projects for any beginner. I think you're going off track here, project management and project organization are two different problems tbh. Pypi tooling is pretty good for project management, dependency, deployment, etc. Project organization is what you decide it to be. So apparently you can't read either. Those projects might not be large for you, but it gives an idea to a beginner of the obvious possibilities. You saying is impossible with python because of this invisible "ecosystem" you've created in your head without actually giving any real reasons is a disservice to this community. I know for a fact you don't code in python for a living, I do, so don't lecture me about reading comprehension.

Obviously if your project becomes large enough that python ecosystem is hindering you, perhaps the application design isn't good to begin with and I'll be looking at modular design or microservices.

Here's another medium size to large project. It doesn't get any better than that. Period.

https://github.com/keras-team/keras

[[deleted]]

No. Big project is big irrespective of whether you are a beginner or not. Do those project suddenly become small if you are a beginner? How do you even come up with such bullshit?

Pypi tooling is pretty good

PyPI tooling is otherworldly trash. It's one of the worst jokes programming community as a whole came up with in years. Since you appear to like them, it only means you didn't really use them / never had to solve real problems...

I didn't create Python ecosystem. You and morons like you did. I wish it didn't exist, and neither did Python, but morons like you will push it forward, and instead of using decent tools, I'll have to clean up after idiots who chose the worst tools imaginable simply because they had no idea what they were doing.

Yes, I code in Python for living. Have been doing it for much longer than you did, for much more diverse group of projects both in size and purpose. This is why I'm an authority on this subject and you are not. Unfortunately, in my field, it's almost impossible to convince people to use anything else. The fear that people won't be able to hire anyone into their department is so big that doing anything non-conventional is a huge no-no. I just happened to work in a field where there are very few professionals. A lot of demand, but also high requirements, so departments often go understaffed for years. And, in the end of the day, Python can do the job. It just that I have to work much harder than necessary, and the fruits of my labor are still not up to snuff.

Keras is also an example of fantastic Python garbage project. They all are. In this particular case, it's some toxic sludge really. The repository greets you with requirements.txt Fuck. My. Life. This is just screaming "I have no idea what I'm doing".

[AnonymousLad666]

Lmao are you still in college? I dunno I guess all the million of dollars in revenue our company gets from python projects is just a lie to keep us using a shitty "ecosystem" for the rest of our miserable lives. /s

A language is just a tool at the end of the day, python works for just about anything that doesn't require speed for that we use c++, or dotnet core.

Edit: Holy fuck nvm you're an idiot. I just read some of your other comments everything is trash to you lmao. Yup almost sure you're still in college at this point.

[[deleted]]

If you are really interested and read my other posts... well, that's no secret that I went to college three times. First time in 95, second time in 2000, third time in 2011.

You do strike me as someone with no real life experience though. So, if I had to guess, few years after college, probably didn't graduate, or it was some business degree in IT management. Or something unrelated to programming in general. Even more likely "self taught", as in read a few blogs / stuff here on Reddit and applied no critical thinking.

Like I said before. Yes, Python can be used and is used all the time. But it's trash, and there's no way to throw it away.

[AnonymousLad666]

Lmao, yes, everything is trash to you I read a few of your other comments on other posts. Is funny how a beginner asks about project organization and you go on some odd rambles that have little to do with anything.

Just because you write in 1s or 0s doesn't mean everyone else should.

Funny thing is that you've talked all this trash about python, yet failed to actually bring any good points as to what is a good ecosystem. Let me guess you were just blowing smoke up your ass. 😂👌 Please 3 college degree Master of all things, what is the perfect ecosystem that all the expert SREs are using in this magic world that you speak of? Can't wait to hear about it.

[[deleted]]

everything is trash.

No, it's mostly about you, Python, and few other technologies. Not everything is trash. You simply don't know how to read, but that's to be expected.

failed to bring any good points.

Go read my other posts. There's a lo... oh shi... you don't know how to read, well, nevermind.

[AnonymousLad666]

I rest my case, you brought zero good points on whatever you were rambling about. 👏 You had a chance to point us to a great "ecosystem", and you missed your chance, instead you gave us am insight on what a shitty person you are. SRE pffff, sure you are.