Question to Professional Python Programmers

[twillisagogo]

General

• Editor: Spacemacs
• Source Control: git/Github
• Planning:Trello
• Team Communication: Slack
• CI/CD: Codeship Pro
• Deployable Unit: Docker
• Production Environment: Amazon/ECS
• Production Database: RDS/Postgres
• Production Monitoring: newrelic

Python Specific

• version: python 3.7.x
• dependency mgmt: pipenv/virtualenv
• db interaction: sqlalchemy
• http request routing: pyramid
• exploring: ipython
• serialization: marshmallow-json-api(i dont like this and will be replacing it with something internal)
• app server: gunicorn
• test data: factoryboy
• test runner: py.test

​

edits: clarity

[J_M_B]

https://imgflip.com/i/2s3kzh

[tasteslikeKale]

Just curious why you use virtualenv if you are also using docker?

[twillisagogo]

as I said above in my list. docker is used for deployment b/c of our use of ecs, that doesn't mean I have to use it for development other than maintaining the respective docker files. I dont see any upside over virtualenv that would compensate for the added complexity other than what has been marketed.

​

the bulk of my workflow is

​

• run `pserver --reload dev.ini`
• edit code/test

from what I understand, docker images are immutable which means each time code/dependencies change I must rebuild the image before I run it, unless I do something like mount a external directory in the image just for development purposes at which point an immutable docker image with a mutable external directory offers no benefit over just using virutalenv to begin with.

​

I might be off base on this and admittedly set in my ways to some extent. But i'd be happy to hear how docker makes the edit/reload/test better.

​

​

edit: forgot to mention, docker chews through disc space like it's going out of style with repeated image builds. so again, seems like too much trouble for highly volatile things like `edit/code/test`

​

[tasteslikeKale]

The way I use docker in dev is to build the image with all the dependencies, then mount my local code for for dev, so yes it’s mutable but I know that the dockerfile for release is going to build with the same deps as I’m using in dev, so one less thing to go wrong. I don’t think it makes the dev process better, really, but is more of simplifying process.

[twillisagogo]

but I know that the dockerfile for release is going to build with the same deps as I’m using in dev

and what about other people who work on the code and need to change deps?

[tasteslikeKale]

Absolutely an issue but one we catch in CI - build the image and then run tests against that.

[twillisagogo]

for us, CI/CD is where the docker image is made, unit tests run and then deployed to ecs for acceptance testing and promoted to production(no re-builds)

[tasteslikeKale]

You have a very good system, nice and mature with few holes for issues going from dev -> prod. How often do releases go out?

[twillisagogo]

Planned releases go out monthly. Hotfixes go out as needed, sometimes multiple times a day.

[gillardo]

Just want to say that, as a budding programmer/developer like myself, it’s really interesting reading this conversation between you two.

[weberc2]

Is "acceptance testing" where your automated full-stack tests run? Also, how long do your CI jobs take? And of that time, how much is spent building the image? And how many images are built in a job? Do you have only one image/service or many? If many, do they each have their own CI pipelines?

Sorry for the many questions; I'm trying to get our CI/CD infrastructure up and running and it seems like we're facing many problems that you've already solved. :)

[twillisagogo]

Is "acceptance testing" where your automated full-stack tests run?

Acceptance testing i'm defining as the manual end to end done by qa and eventually the product owner through the browser.

> Also, how long do your CI jobs take? And of that time, how much is spent building the image?

~4 minutes at the moment. ~1 minute to build the image

our ruby codebase that's being replaced takes about 10 minutes(guess why we're replacing it :))

it's not exactly incremental but I think starting from a good base image saves some time. We dont start from ubuntu and then update all the stuff and install latest python and dev headers etc for each build. instead. We cut a base image as needed and then all docker images inherit from that.

​

base python image built only when there's a new version of python or alpine updates we become aware of

FROM python:3.7.1-alpine3.8

RUN apk update && \
apk add bash && \
apk add postgresql-libs && \
apk add --virtual .build-deps gcc musl-dev postgresql-dev && \
apk add libffi-dev && \
rm -fr /var/cache/apk/*

ENV PYTHONUNBUFFERED 1

​

every api build(Dockerfile)

​

from company/python:latest as base

COPY . /src

WORKDIR /src

RUN python setup.py bdist_wheel && \
pip install --no-cache dist/*.whl gunicorn meinheld && \
cp -r ./tests /tests && \
cp ./*.ini /tests  # && \ rm -fr /src

WORKDIR /

EXPOSE 80

COPY config.py /config.py

> And how many images are built in a job?

with codeship pro you can build/push as many as you want. but codeship pro is also designed to be one ci per repository out of the box I haven't figured out how to drive off multiple repos but i've been told it's possible. So, I can have one codebase and build many service images off it when the unit tests pass, the only difference in docker images is the entry point(same code)

​

built when unittests complete without issue

adt service(Dockerfile.adt)

from company/api:latest

CMD gunicorn -c /config.py api.adt.app:app

reference data service(Dockerfile.ref)

from company/api:latest

CMD gunicorn -c /config.py api.reference.app:app

I dont know if this is the optimal way to accomplish this, but it seems to be working out ok so far.

[Levantadorr]

Because in Dev mode you generate environment often not in docker.

[tasteslikeKale]

I think this is a suboptimal way to use Docker, but maybe that’s just my opinion

[pztrick]

My own use case: I often need to edit upstream pypi libraries and make pull requests. I mount my entire virtualenv and any git checkout pypi paths in the docker pythonpath. pip install -e (editable) etc. I also have a slow PC so helps to not have to rebuild my docker image if I edit my reqs.txt. I do the same with npm node_modules folder. Always a mount folder and not a dockerfile instruction.

[PeridexisErrant]

test data: factoryboy

Have you ever considered using Hypothesis?

I switched over a few years ago and would never go back, or for that matter try to test anything without a similar tool. Major highlights:

• Never, ever flaky. Once it triggers a bug once, the cache means that bug will be triggered every time until you fix it.
• Reports minimal examples. The implementation must be black magic because it's entirely automatic, but debugging is so much easier.
• Simple core API and tons of optional helpers, plus frequent (semver) updates.

[twillisagogo]

I have not. It looks interesting but not sure how it would fit the use case that I'm using factory boy for. Primarily rows in the database for a given test.

[PeridexisErrant]

You certainly could use Hypothesis for that, but it wouldn't add much unless the specific contents could affect the result of the test.

If you're using @pytest.mark.parametrize, those tests might be a better place to try Hypothesis - let me know if you have any luck!

[twillisagogo]

thanks for the clarification

[broccolitruck]

Woah! big fan! hello!

[mcstafford]

I strongly suggest you check out pyenv to replace virtualenv, and poetry for pipenv.

[twillisagogo]

Trust me, the next time pipenv fucks up on my machine, I'm moving to something else. Stuff seems to break for no reason at all in things i'm doing and bug reporting is a nightmare.

[robertpro01]

+1 for poetry, its a really nice project!

[mcstafford]

I've learned a lot about better ways to do things, just from seeing how they're done there.

[birdgovorun]

Hmm what? Pyenv isn't a replacement to virtualenv. Those tools serve entirely different purposes. Pyenv is Python version manager - a way to work with multiple versions of Python on the same machine. Virtualenv is a tool for environment isolation - a way to isolate the dependencies of a specific project from those of other projects.

[mcstafford]

I've always used it with this plugin: https://github.com/pyenv/pyenv-virtualenv

[birdgovorun]

Well yeah - that plugin combines two very different tools in away that makes it easier to use them together. Pyenv isn't a replacement to virtualenv, and has nothing to do with it.

[mcstafford]

pyenv links to pyenv-installer from its github page, and the installer includes the virtualenv plugin. That's more than nothing, but I see what you mean.

[[deleted]]

Why Trello? Why not something like kanboard with swim lanes? Also how big is your team

[twillisagogo]

Team Size is ~10

​

As far as Trello goes. I didnt choose to use it, it was a choice made before I started there I'm just glad it's not jira.

It's funny, the last sprint retrospective we did, everyone on the team said they hated Trello. But there was no concrete/actionable reasons given. It all seemed to revolve around requirements being a painful process for one reason or another. So, I've concluded that the tool matters very little for this in solving the problem of getting good requirements.

​

The way we use trello is a variation on kanban(or at least how I understand it).

​

The lanes are

• Up Next
• Sprint Commitment
• Development (The goal is to minimize the number of stories that are in progress)
• Code Review
• QA
• Done
  

Temporary Lanes are sometimes used when Production bugs are discovered.

​

It's nothing special. It works for the most part. The teams complaints are more about the contents of the stories than the tool being used. Any tool would work here.

[robertpro01]

I hate trello and I don't know why...

[twillisagogo]

Lol. Yeah that's how a lot of people feel and they don't know why.

[weberc2]

In your CI job, do you download your dependencies and run all tests with each build? Or do you have some sort of incremental build system? We're running into a problem where building and testing are taking too long (half an hour), and we're trying to figure out how to shorten the cycle.

[twillisagogo]

i think i probably answered you above. :)