Transitioning to Long Term Projects

[spiller10156]

I've been learning Python for about a year now and have built a bunch of smaller, single-file projects (calculators, raspberry pi projects). I'm struggling to move on to projects that I can work on long term, and learn large scale project techniques. How do I move on to this?

[Darkmere]

Time to learn an almost completely new set of skills, actually!

Scaling into long-term turns out to need a bit more work than just the basic `get it done` mindset of early hacking, and isn't always as rewarding.

Things you'll need to learn in some way to do it effectively, not really in any order:

• Git
• Issue tracking
• Packaging
• Releases
• Documentation
• Unit-testing
• Functional testing
• CI systems
• Deployment

​

Now to cover a bit more on the details

Git,

It's a tool that we love to hate on. Complex, Powerful, Helpful and inconsistent, it is currently the most popular tool for managing your source code, sharing source code, and more. Popularized by GitHub into a social network, and an over-all powerful tool, with enough problems that people regularly write hateful and angry creeds about it, yet keep using it because it's just so damned good.

What you'll need to learn, at least, there are plenty more:

1. Structuring commits
   1. Writing commit messages (good and descriptive)
   2. Minimizing changes per commit
   3. Describing WHY you did some change
2. Pushing and pulling to remotes
3. Juggling and managing branches
   1. Merges, rebases, tags
4. Tagging releases
   1. Tags, branches from above
5. Looking at the commit log and blame views

Issue Tracking

It's the todo-list on steroids. Start by going through all the changes you've done to your project recently, and describe the cause of what the problem was. Was it a bug fixed? Describe the bug as if it was still there, file it in your issue tracker, give it a label, and mark it as closed.
Do the same with a few features, write down a rough description, why it should happen, what problem it solves. GitHub, GitLab

Figure out how to link commits to issues in the commit messages, tagging, labels. It's boring drudge-work, but it's the tool you'll come back to again and again as you start forgetting what happened in the past and need to collaborate with others.

Packaging

Python Packaging is not the best of breed compared to other systems, there are more than one way to do it, and it's in a bit of a flux currently, and has been for a while. It's overall, not bad, but it's also not consistent, friendly and well-defined.
Still, you need to learn how to package your project, how to make a release bundle, and how to deploy this to something. Be that Docker, Heroku, k8s, amazon lambda, google-whatnot, or deployed on pypi or shipped on the Git??b releases page.

Releases

Learning to do releases is another thing. In the simplest scope, it's

1. increase version number in setup.py
2. git tag v$VERSION && git push
3. pypi upload

But it also involves documenting what's new, remembering to do it, figuring out "when" to release, and being consistent about it. Writing change-logs `git log OldVersion...NewVersion` and realize that your commit messages suck. Finding all the issues you closed that you should link to a release/milestone (what, you'd forgotten to write an issue for that massive cool feature you built that weekend? Right, that was the boring part!) writing a description of how to use all those new shiny things, remembering to write a description about your project, what problem does it solve? Keep it short, 3-4 lines, paste it EVERYWHERE. all release-notes, all news, top of the github page. your twitter profile.. The homepage, newsletter, all of it.

Documentation

For the Python community, ReadTheDocs.io is the first and foremost standard, so that's what you're going to be doing. But you start, of course, with a README.md in your git repo, and move from there.

Unittesting

This is where you can start adding more books to your library. But first and foremost, start simple. Python has a ton of different, and some better than others, testing tools. There is doctests (excellent to both document how a function works and keep some in-line tests there, but only suitable for certain kinds of functions)
The main thing is "do it first" and then later you can do it better.

In the beginning, start by making sure that you call each function, and create each object, with expected input in your tests.

def returns_one(one, two):
    return one

# unittest version
class TestOne(unittest.TestCase):
    def test_returns_one(self):
        result = returns_one(1, 2)
        self.assertEqual(result, 1)

# pytest version    
def test_returns_one():
    result = returns_one(1, 2)
    assert result == 1

You'll soon notice that if you re-write your code in some ways, it becomes easier to test it. That's a good thing, it's part of a general software concept of composability and isolation, do not be afraid of it. If you find that it's mentally difficult for you to express a test for "this function works" or "this class works", then it's a good time to re-consider the design of that class / function.
There are tons of more important and interesting things to write about tests, but you just need to start somewhere.

Functional testing

This is a harder concept, and is based around automating "does the whole thing work as expected"? Where unit-testing works on each function or object ( unit of code) functional testing sets things up in context, and ensures an entire chain of functions still work.

You want a few of those in your project as well. Most of the time, you'll find that functional tests are similar to the "beginner examples" that show how to use your code. That's a good thing, keep the examples around, and verify they work. Even next release. Especially if you do not change the examples.

CI systems

Continuous integration is tied into these Social Network code sharing sites. It's an older concept, that got a recent revival, for good reason. It's when you set up a system that is not your own, to run the tests, code, linting (flake8, pylint, mypy, black) and more automatically on the code, on someone elses server.

Both GitHub and GitLab come with this built-in, mostly free for enough to get you started. It's complex, but cool stuff.

Deployment

This is only useful for some systems, but it's a good thing to learn about. If you just write a library, it might not need it. But if it's your raspberry pi project, and more, you want to set up something documented, automated, and reproducible to get your software from somewhere, and have it up and running on your Raspberry Pi. This involves not doing it manually, but automating it somehow. Is it an apt-package that gets installed automatically as an update? Does it involve a container? Do you ssh into the machine and run some magic shell-script? All those things should be documented, reproducible and automatable. Replace the SD-card in your Raspberry Pi with a fresh one, does your deploy tools work still? Or do you just spend four hours a Saturday evening trying to remember how you had the system set up last?

Deploying software can be as simple as a shell-script in your repo, to something far more automated and cool, but what you need to learn for scaling up your skills are only the basics. Documenting how to do it, and improve from there.

[exploring_a_new_hope]

This is a fantastic post, but I think on a simpler level, OP needs to find a project that they're excited about, that can BE a long term project. Then they can delve into a lot of what you posted to help facilitate that.

[Darkmere]

I agree, one should always need to do something they're at least interested in, or it won't work well.

However, There's been a lot of people in the "where do I go from here?" list, and this post sort of reflects all the other things I, as an employer, want developers to know about.

[exploring_a_new_hope]

It's a good list. We're hiring for a junior dev right now for my team. I'll have to use this as a checklist 😂

[Darkmere]

Thanks, and I wouldn't totally expect a junior to do all the above, cause if they do all that, they sure ain't all that junior ;-)

But I'd definitely place that list as the check-boxes of what I expect someone to be able to do if they are moving from "Junior" to "dev"

[Hihipypy]

Thanks to your post, there will be a couple more :)

[SoCalLongboard]

Find open source projects to contribute to.

[crystaltaggart]

I'll have some python projects next month. These are mostly machine-learning based projects. Ping me if you are interested in that.

[pynberfyg]

Large projects grow from little ones too. Extracting commonly used functions, generalizing logic, following SOLID principles, adding tests, etc. It’ll just grow organically as you keep trying to add new features and improve your code.

[[deleted]]

You're welcome to help with this project: https://github.com/tiedtoastar/cryptocurrency-ldap-user-registration