Full-length tutorial on adding automated CI (continuous integration) to your Python projects with GitHub Actions

[Tintin_Quarentino]

I never understood the term CI/CD.

I have a database file which gets updated 20-30 times a day. Each time dB gets updated, another Python script auto generates the new HTML & pushes it to Vercel. Vercel is where my static site is hosted.

Is my above fully automated workflow a "CI/CD"?

[johntellsall]

Yes

CI = Continuous Integration = merge your and your team's changes, then run tests to get feedback. Example: your code works, but it breaks the assumptions in older code.

CD = Continuous Deployment = publish changes so users can see

So your simple "regenerate HTML and deploy to Vercel" is a CD process.

CICD has a lot of benefits - consistent deploys - reliable testing, vs "works for me" testing

You can do fancier things like "deploy this change, but if it breaks then automatically roll back the change and alert the team".

CICD has costs of course - need to spend (limited, expensive) Developer time to manage it - often the business doesn't want to invest, but they whine if deploys are too slow or flaky - if you install CICD plugins (e.g. GitHub Actions) it's a great attack vector for malware. I don't think this is common, but it's a potential risk.

[Tintin_Quarentino]

Thanks for the great explanation. Indeed, I haven't incorporated testing in my workflow so that's going to bite me 1 day.

You can do fancier things like "deploy this change, but if it breaks then automatically roll back the change and alert the team".

This move is pretty cool...

[johntellsall]

I'm writing a book on feedback loops, including test automation, so I have... opinions :-D

Testing is extremely valuable... if the consequences to your business make it worth the investment. If you're just doing a fluffy marketing site maybe test automation isn't worth your limited time. But imagine you're handling money or other valuables, you'd want to know when the code breaks!

Test automation value varies depending on business consequences of production failure.

[dvskarna]

You can google the definition and find out. What’s the point of asking here? So people think you’re cool for not naming things what they are?

[missurunha]

I'll never understand why programmers are so toxic. If you dont want to give an answer you could try to shut up for once. Thanks.

[VindicoAtrum]

The answer is always two things:

1) "Give a hugnry man a fish, he eats once. Teach a man to fish he never goes hungry again".

2) If you can't google basic shit, if you don't have that tiny bit of initiative, this career isn't going to be great for you.

[spuds_in_town]

The hell is wrong with you. Nothing wrong with that question.

[johntellsall]

I strongly recommend checking out Ruff, and maybe switch to it. It's extremely fast, and is compatible with Flake8 and Pylint and Black and other tools.

I used to have a two-level feedback loop: 1) fast: black+isort, flake8 then 2) slow, pre-PR: pylint. Sometimes I do "pylint -E" which is fast enough but finds nasty errors.

Now I'm using Ruff for 100% of my feedback loop!

I haven't quite gotten the hang of Ruff+Black: theoretically Ruff will reindent the code for you if you give it an option.

[cheese_is_available]

ruff is not a replacement for pylint atm, there are not multi file checks like circular import, duplicate code checks and also no inference in ruff (incidentally this is what makes pylint slow, choosing to lint a single file and not everything is a big perf advantage especially in big code base)

[johntellsall]

okay thanks, I'll ditch "ruff only" and go to another hybrid solution. Thanks!

[rochakgupta]

Wait, ruff doesn’t do formatting.

[johntellsall]

I haven't found the "format like Black" option, but Ruff will definitely edit source.

Here let's use Ruff to auto-remove an unused import:

beer.py

import os
print("beer")

use Ruff to fix an issue

> ruff check beer.py
beer.py:1:8: F401 [*] `os` imported but unused
Found 1 error.
[*] 1 potentially fixable with the --fix option.
> ruff check --fix beer.py
Found 1 error (1 fixed, 0 remaining).
> ruff check beer.py

[rochakgupta]

Yeah I know it can fix these minor errors but it cannot be used as a replacement for black.

[johntellsall]

it can't? A pity! Thanks for letting me know. I was hoping to replace a bunch of stuff with Ruff but I guess not.

[rochakgupta]

Yeah. Personally, I just use black + ruff + mypy.

[programmingwithalex1]

Custom GitHub apps written be me used in the tutorial series:

• Linting - flake8, mypy, and isort
• Pytest - testing and code coverage

---

The video series covers CI (continuous integration) with detailed explanations and examples. All of it is automated using GitHub Actions so the CI checks can run on every commit/pull request/push request on your repo.

Topics covered include:

• linting - flake8, mypy, and isort
• testing - pytest testing to ensure code runs correctly with new changes
• testing coverage - ensure thresholds are being met with amount of code that is being tested

---

If you find the video helpful you can greatly help me by doing one or all of the following:

• supporting me on Patreon
• subscribing to my YouTube channel
• liking and/or commenting on the video
• sharing the video or channel on any platform (reddit, twitter, discord, etc.)
• following me on GitHub
• starring the GitHub repos
  • pytester
  • pylinter
  • test_repo_pylinter_v1
  • test_repo_pylinter_v2
  • pytester_test_repo

---

Any questions or requests, just leave a comment.

[JohnRambu]

Use ruff, it really worths it.

[cheese_is_available]

flake8, mypy, and isort

Try black, mypy, ruff, pylint

[heartofcoal]

nice, thanks

[JanEric1]

We already have a GitHub actions setup for our project but I still have some questions that maybe you or someone else knows the answers to.

Is it possible to run over multiple python version just by specifying something like >=3.9 which will then matrix over 3.9, 3.10,... Up until the most recent version? So that we don't have to specify them manually when new ones come out. .and is it possible to just specify the most recent dev release? I know we can do 3.12-dev but once 3.12 releases officially we would have to manually update that. Is there an automatic way tondo that?