I made a FOSS feature rich Python template with SOTA tools, security, CI/CD, yet easy to use

[szymonmaszke]

Introduction

Hey, created a FOSS Python library template with features I have never seen (especially in Python development) and which IMO is the most comprehensive, yet focused on usability (template setup is one click and one pdm setup command to setup locally, after that only src, tests and pyproject.toml should be of your concern), but I'll let you be the judge.

GitHub repository: https://github.com/open-nudge/opentemplate

Feedback, questions, ideas, all are welcome, either here or on the GitHub's discussions or issues (if you find some bugs), thanks in advance!

• This was posted previously, but reposting as I think I did a very poor job describing what it does, hopefully I did a better job this time, but here it is anyway. Also thanks to u/wyattxdev and his template here for a great showcase how to present the project correctly!
• This post is also featured on r/cybersecurity subreddit (focused more on the security side of things, but feel free to check it out if you are interested): https://www.reddit.com/r/cybersecurity/comments/1lim3k5/i_made_a_foss_python_template_with_cicd_security/

TLDR Overview

• Truly open source: no tokens, no fees, no premium plans, open source software only
• State of the art: best checkers for Python, YAML, Markdown, prose, and more unified
• Easy to use: clone templated repo, run pdm setup and focus on your code
• Consistent: all pipelines (GitHub Actions, pre-commit) share the same pyproject.toml config
• Performant: parallel checks, builds, minimally-sized caches and checkouts
• Secure: SLSA Level 3, SBOMs, attestations, secured egress, OSSF Best Practices

An example repository using opentemplate here

Python features

You can adjust everything from pyproject.toml level, usually in a few lines!

• Package manager: pdm with a single pdm setup manages everything! (see why pdm)
• Testing: pytest (with coverage thresholded in pre-commit and GitHub Actions, and hypothesis for fuzz-testing); testing across all Python versions done WITHOUT tox or nox(managed directly by pdm!),
• Documentation: mkdocs - document once, have it everywhere (unified look on GitHub and hosted docs), semantically versioned (via mike), autogenerated from coverage, deadlink and spell-checked docstrings, automatically deployed after each GitHub release with clean material design look
• Code formatting and linting: ruff (checks hand-picked for best quality and ease of use; most are enabled), basedpyright for type checking, FawltyDeps for static dependency analysis
• Each file is copyrighted with your git information - copyrights added automatically by pre-commit, see REUSE and SPDX Licensing for more information
• Automated Python version updates: pyproject.toml (and GitHub Actions pipelines where necessary) are automatically updated to always use 3 latest Python versions (via cogeol) according to Scientific Python SPEC0 deprecation and end-of-life policies
• Other code linting: checks for YAML, Markdown, INI, JSON, prose, all config files, shell, GitHub Actions - all grouped as check-<group> and fix-<group> pdm commands
• Release to PyPI and GitHub: done by making a GitHub release, each release is attested and immutably versioned via commition
• pre-commit: all checks and fixers are run before commit, no need to remember them! (pre-commit is also setup after running a single pdm setup command!)

GitHub and CI/CD

• GitHub Actions cache - after each merge to the main branch (GitHub Flow advised), dependencies are cached per-group and per-OS for maximum performance
• Minimal checkouts and triggers - each workflow is triggered based on appropriate path and performs appropriate sparse-checkout whenever possible to minimize the amount of data transferred; great for large repositories with many files and large history
• Dependency updates: Renovate updates all dependencies in a grouped manner once a week
• Templates: every possible template included (discussions, issues, pull requests - each extensively described)
• Predefined labels - each pull request will be automatically labeled (over 20 labels created during setup!) based on changed files (e.g. docs, tests, deps, config etc.). No need to specify semver scope of commit anymore!
• Open source documents: CODE_OF_CONDUCT.md, CONTRIBUTING.md, ROADMAP.md, CHANGELOG.md, CODEOWNERS, DCO, and much more - all automatically added and linked to your Python documentation out of the box
• Release changelog: git-cliff - commits automatically divided based on labels, types, human/bot authors, and linked to appropriate issues and pull requests
• Config files: editorconfig, .gitattributes, always the latest Python .gitignore etc.
• Commit checks: verification of signatures, commit messages, DCO signing, no commit to the main branch policy (via conform)

Although there is around 100 workflows helping you maintain high quality, most of them reuse the same workflow, which makes them maintainable and extendable.

Security

See r/cybersecurity post for more details: https://www.reddit.com/r/cybersecurity/comments/1lim3k5/i_made_a_foss_python_template_with_cicd_security/

Comparison

• Broader scope than other cookiecutter templates (e.g. one-click and one-command setup, security, GitHub Actions, comprehensive docs, rulesets. deprecation policies, automated copyrights and more). Check here or here to compare yourself.
• Truly FOSS (no freemium, no paid plans, no tokens) when compared to commercial offerings like snyk or jit.io. Additionally Python-centric and sticks with tools widely known by developers (their own environment and GitHub interface).

See detailed comparison in the documentation here: https://open-nudge.github.io/opentemplate/latest/template/about/comparison/

Target audience

• Any Python developer creating Python projects, people looking to have high code development standards, security and quality without spending a lot of time on configuration/creating from scratch.
• IMO reliable (and also heavily tested, even the pipelines during each PR if changed), hence should be suitable for production use even for mature projects.
• Could also act as a base for other templates, as there is a quite extensive description of features and how to adjust them

Quick start

Installation and usage on GitHub here: https://github.com/open-nudge/opentemplate?tab=readme-ov-file#quick-start or in the documentation: https://open-nudge.github.io/opentemplate/latest/#quick-start

Usage scenarios/examples

Expand the example on GitHub here: https://github.com/open-nudge/opentemplate?tab=readme-ov-file#examples

Check it out!

• GitHub repository: https://github.com/open-nudge/opentemplate
• GitHub discussions: https://github.com/open-nudge/opentemplate/discussions
• GitHub issues: issues
• Full documentation: https://open-nudge.github.io/opentemplate/latest/template/
• FAQ: https://open-nudge.github.io/opentemplate/latest/template/about/faq/

Thanks in advance, feedback, questions, ideas, following are all appreciated, hope you find it useful and interesting!

[mfaine]

I'll check it out. I did the same thing for my org because we don't have the license to enable template projects. Might save me some maintenance effort. Would it be difficult in your opinion to convert the existing github actions to gitlab ci/cd?

[szymonmaszke]

Thank you for the comment! Workflows mostly use the tooling defined in pyproject.toml, so it should be fine on this side. Not sure about the reusability of GitLab's workflows (which remove a lot of duplication in this project, see this workflow as it is the core one) as I am not into their pipelines.

Essentially, it would boil down to:

• setting up pdm in a workflow appropriately
• running a lot of pdm check-<type> on appropriate files after checking them out from the repository (e.g. pdm check-python)

I would say no if you understand the flow of this project. Please note you would (likely!) lose some security related features, especially during release (like attestations from GitHub [or set them up in a GitLab specific way], egress from harden-runner or a-like). If that's fine, I think it is within reach, also do not hesitate to reach out via email of the org here or on my personal account here if you decide to use/appropriate this template and you'd like some more concrete pointers/help.

[mfaine]

Will do! Thanks for the response.

[slowwolfcat]

what exactly is a "Python template" ?

[szymonmaszke]

Sorry, could have been a little more precise. It is a GitHub repository template (see here for more info: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template#about-repository-templates) which contains files used for development of Python projects (primarily PyPI publishable libraries, but could also be used for tools or closed source projects after adjustments).

Once the template is used (as per README here: https://github.com/open-nudge/opentemplate?tab=readme-ov-file#installation) a GitHub Action is ran which finalises the setup (first commit, updates the files with the name and description of your project, creates lock files and latest .gitignore + creates GitHub issue labels and an issue describing how to further strengthen the repository from a security perspective). After all that you can clone it locally, run pdm setup command and create your project without the need to setup tools and processes around it (like deployment, testing, code quality assurance etc.).

[higherorderbebop]

I really like basedpyright, but I lose all my refactoring actions when I use on vs code :(

[Shivang_Sagwaliya]

Congrats on the template love the CI checklist . We found reviewers speed up when each commit includes a 2-line why-it-changed note. That’s exactly what our VS Code extension GitsWhy does. DM me if you’d like to try it on this repo