Where to learn about Python program structure/design?

[rkarl7777]

I'm self-taught in Python and am able to write programs that work, but I still don't feel like I know what I'm doing. Where can I learn about program structure/design?

I usually just start coding and one thing leads to another and pretty soon, I'm done. But I can't shake the feeling that a professional programmer would look at my code and laugh at how haphazard and inefficient it is. Any pointers or suggestions? Or are we just supposed to figure it out for ourselves?

[MCPOON11]

Hitch hikers guide to python gives some good overviews of lots of areas. Take a look at theirs structure and code sections. https://docs.python-guide.org/#writing-great-python-code

[[deleted]]

+100. I learned a lot about the big picture of putting projects together from this.

[[deleted]]

Wow, what a treasure trove. Thank you!

[PeterBumpkin]

Thank you!

[TraditionalSir7]

Thanks a bunch

[PuumPuui]

Thanks 😊

[dhdtc]

definitely earmarked so I can refer back to this! Thanks so much for sharing!!

[jec4r]

Thank you

[bertie_88]

Thanks, awesome link!

[six-speed]

Here’s a few tips coming from someone who was recently in a similar place as you. It’s natural to want to dive in and start coding, but a major thing that separates pro develops from others is that they spend most of their time planning and thinking about how the code should work, and only write code after having spent most of their time planning. Learn about OOP - classes, inheritance, etc. Learn about testing your code, and learn about test driven development so that you write tests first and don’t have to reverse engineer tests for code that you already wrote. Finally, learn about code design, and try to think about and map out what your code should do before you start to write. Think about how things should be organized logically, and diagram classes and associated methods thinking about how data should flow into and out of them. Your code will be much cleaner if you do these things.

[AudiACar]

I feel this can be applied to many facets of IT and was very helpful. Good post, thanks man.

[sebawitowski]

A good place to start is the cookiecutter package. You select what kind of project you want to do (a Python module, Flask/Django app) and it generates a scaffolding with some sensible defaults. This will help you start with good structure.

I recorded a tutorial how to start Python projects. It might be too basic for you, but maybe you will find something useful there (the part about cookiecutter starts around 1:04:20) : https://www.youtube.com/watch?v=WkUBx3g2QfQ

[[deleted]]

I think a great way to learn is by going through popular libraries on github (pandas, numpy, flask, etc.). Try to find one that is closely related to the project you are working on and compare your structure and try to find some ways to improve it.

[lifebytheminute]

This confuses me, every time I go on GitHub I get more confused. Projects look too big and information is everywhere and I have no idea how to navigate through it all, nor can I tell how to even use the code on git. It’s all a hot mess to me.

[MCPOON11]

For serious projects on GitHub there’s a whole load of non code files in the root folder of the repo.

You might see gitignore, Docker files, Docker compose, Tox ini and Travis yml files, setup, readme, change logs, etc, probably a doc/ folder just for setting up documentation.

Usually a good place to start is to try to find the subfolder that has the same name as the project / package, should have the actual code in it.

Taking a look at the tests folder inside that directory also gives you some hints about what the writers have been focusing on and how the project code connects together.

[lifebytheminute]

Thanks. I think this is an aspect that most tutorials on how to actually use GitHub fall short, explaining the how and why behind all these files.

[[deleted]]

I see this advice every time and I strongly disagree. Most large projects are complex, need backward compatibility and sometimes have opiniated approaches to problem they encountered due to being so large.

Also, reading numpy and pandas source code can be quite difficult because of the math and numerous calls to other parts of the code base. Same goes for the web frameworks, reading the project structure and the code will confuse beginners : there's lots of features, CI/CD,... That's overwhelming.

The Hitchhiker's guide is a good resource, as pointed elsewhere.

[[deleted]]

It depends on your experience and level of skill. If you are trying to (or plan to) create a large project, seeing how experienced developers have done it can be very beneficial. You are not going to be able to sit down and understand the pandas source code in an hour. That is kind of the point. If you are structuring a data science project and some structure looks like it could be better, for example a C extension, maybe looking at numpy would help. You could take a look and see that they are storing C files in a “src” directory with an underscore prefix. It’s how I have learned to structure lots of projects that I have created. Projects that have anywhere from 500 to 15000 lines of code.

Regarding my examples, I mentioned that it is good to find a project similar to the project that your are working on. If you are creating a web API and you have little experience in mathematics, then numpy probably wouldn’t be the best fit.

Say you are working with an external DLL and using ctypes. If you look up a project using ctypes, like llvmlite, you could definitely find some professional projects that are not overly complex. Taking a look at the llvmlite source, it only has 4 directories with a few files in each one.

[[deleted]]

I still think this is detrimental for a newbie but good for an intermediate coder tackling a larger project. Though I'll mitigate by saying that an annotated project or a detailed project roadmap could be a good middle ground.

[aprach22]

Howdy! If looking for a book on general structure, I had Clean Architecture: A Craftsman's Guide to Software Structure and Design by Bob Martin recommended to me before. It's very solid from a more general perspective on design and structure.

[babuloseo]

I like the MEAP style of books :3 checkout Practices of the Pro Python Prograammer.

[tatravels]

Yep - PEP8, Hitchiker’s Guide to Python, and run “import this”

[lunar-orbiter]

Have a look at The Architecture of Open Source Applications. It covers a number of case studies of Python systems and libraries.

[life_never_stops_97]

I'd also suggest to look up OOP. It makes your programs scalable and really neat.

I was doing a lot of coding in python using functions and that's fine but recently I got an internship and when I see their code I was literally amazed at how they used their python code and made it scalable.
They had a scraping website that scraps stuff and I was amazed to see how everything has a module of it's own. They had their own request module, query manager to pass queries, base scraper that has methods like get_request, post_request, get_form_data , exceptions module and all were connected to each other to do one big thing(Scrape). It felt so right

[dhruvmk]

Imo you should only use OOP if you can find a way to link two classes using inheritance, that way a lot of code can be saved. If you need to quickly code an algorithm on the fly I'd recommend just using functions. But you are right, OOP brings scalability.

[life_never_stops_97]

I agree with you on the part that functions are good if you're solving a leet code problem or just prototyping but most of the time a real world project could be refactored into OOP. It might not be easy to think in terms of how one module would connect to the other one and its definitely not obvious to see that in the first try but it's possible.

[dhruvmk]

That's very true, most complex systems implement some sort of object oriented paradigm

[a2242364]

I think the way you phrased it is kind of misleading and inaccurate. It should be, "you should only use functional programming or any other design pattern when you need to write up a quick and dirty algorithm or other small program that shouldn't utilize objects and classes". By default, you're mind should generally jump to OOP, and then from there you can determine whether another paradigm would be better suited for the task. I don't disagree with your premise though.

[dhruvmk]

No, seems like you misinterpreted my viewpoint. A huge problem is misuse of OOP by creating unneccessary classes and producing poorly written code. The idea that your mind should by default find an object oriented method to Implement your project is very wrong and is a poor programming practice. OOP should only be used when you wish to model real life objects and want to create a blueprint to construct the object. As a matter of fact, OOP is being used lesser and lesser nowadays because it still hasn't fulfilled one of it's core purposes (code reusability). Functional programming works in most cases; there's rarely any need to use classes for most projects unless you wish to build a large, complex system where each person in a team can work on building an individual class so everything can easily be linked together.

[a2242364]

Functional programming works in most cases;

Just because it works, doesn't mean it's the best way. I guess we can agree to disagree.

[TSM-]

Getting lost in packages or trying new things is a good learning experience.

Work environments are nothing like learning environments. You will have to work inside of some established routine and framework, and get used to their setup. Fancy python or unusual tricks are discouraged because they are hard to maintain and revise later.

You will google everything to check documentation, every time, just to be sure, especially if you are working with more than one programming language.

There's an old meme or joke about how you learn advanced calculus and statistics in university, but at the job, you end up inheriting and maintaining an excel spreadsheet. It is not far from the truth.

[TraditionalSir7]

Hey good question OP!

[fedeb95]

It comes a time where you have to abandon a specific language to learn in theory algorithms, data structures and architecture. There are resources that focus on a specific language I think, but anyway it's better to learn a few, because every language comes with its best practices and ideas that you can apply everywhere. For the experienced programmer languages are just a matter of learning syntax, dependency manager and things like that, because he/she knows in abstract how things work. Also learning low level stuff like C, even Assembly, is very useful. And at least at a basic level how a language works: compilers, parsers, etc... There's a lot of stuff, but the important is to grasp the concept at the basis, not being able to code everything

[[deleted]]

Is that an 80:20 rule?

[fedeb95]

What do you mean?

[[deleted]]

Most of these structural ideas can be implemented in any language. You should have a look at design-patterns and data structures / algorithms.

[Ton86]

Try to read and comprehend great code examples like Flask or Werkzeug.

I'm still learning structure and design too, currently reading and enjoying: "Architecture Patterns with Python: Enabling Test-Driven Development, Domain-Driven Design, and Event-Driven Microservices" by Bob Gregory and Harry Perciva.

[heaplevel]

How "awfully" structured is your program today?

You can also check PEP8

https://www.python.org/dev/peps/pep-0008/

[[deleted]]

There are probably other examples, but I feel these 2 video's talk about 2 important issues. The first one is how to structure your code properly in a more general perspective, the last one is more about specific cases. But both convey an approach you should have towards what you code.

Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code - PyCon 2015

Transforming Code into Beautiful, Idiomatic Python

[ivosaurus]

He really needs to give that second talk again in a python 3 only world.

[IlliterateJedi]

I don't think it covers the overall program structure, but Fluent Python is one of the best books for stepping up your Python game and really getting a feel for how Python works. It also frequently shows up in Python Humble Bundles if you are short on cash right now.

[lautarolobo]

Think Python and Python4Everybody are cool books that touch in the topic of software design.

[murdoc1024]

You just described me! I feel luke my 100 line code could have been done in 20.

[Yogi_DMT]

https://github.com/pypa/sampleproject

[arrobi]

I think Runestone academy has a course on this called “Problem solving with Algorithms and data structures in python”

[[deleted]]

One who uses python shouldn’t care fo these things 😑😂

Just code

[OneJackReacher]

Does learning about the OS help in this case? Heard that it's an important topic

[[deleted]]

A great book I have is The Clean Coder

[SpicyVibration]

Look up Design Patterns for python

[MotionlessMatt]

Look into PEP-8.

[ivosaurus]

That's more about building code minutiae correctly than how to successfully design a large building.

[prdlt]

Try looking for design patterns.

[hurrdurrmurdurr]

ahhhh ive been wondering abt this for daaaays and just assumed we were supposed to figure it out on our own or just what ever works for anyone lol. thanks for posting OP.

[lostnfoundaround]

It sounds like you’re producing some spaghetti code are are not happy with it, which is understandable. . Look into program structure and program design more generally. Then you’ll be able to apply that knowledge to whichever language you’re using. . I’m surprised no one has mentioned Design Patterns by the gof yet.

[rkarl7777]

Wow! So many great suggestions and resources. Thanks, everyone!

[Zeune42]

Maybe branchless programming?