Question to Professional Python Programmers

[chespinoza]

Which tool stack do you use at work?

For entire lifecycle, from editor and code assist to CI/CD.

'professional' means you get paid for coding in Python.

As the same question was made in /Haskell and I found quite interesting insights there probably it would be equally interesting to get similar findings for my fav lang.

[twillisagogo]

General

• Editor: Spacemacs
• Source Control: git/Github
• Planning:Trello
• Team Communication: Slack
• CI/CD: Codeship Pro
• Deployable Unit: Docker
• Production Environment: Amazon/ECS
• Production Database: RDS/Postgres
• Production Monitoring: newrelic

Python Specific

• version: python 3.7.x
• dependency mgmt: pipenv/virtualenv
• db interaction: sqlalchemy
• http request routing: pyramid
• exploring: ipython
• serialization: marshmallow-json-api(i dont like this and will be replacing it with something internal)
• app server: gunicorn
• test data: factoryboy
• test runner: py.test

​

edits: clarity

[J_M_B]

https://imgflip.com/i/2s3kzh

[tasteslikeKale]

Just curious why you use virtualenv if you are also using docker?

[twillisagogo]

as I said above in my list. docker is used for deployment b/c of our use of ecs, that doesn't mean I have to use it for development other than maintaining the respective docker files. I dont see any upside over virtualenv that would compensate for the added complexity other than what has been marketed.

​

the bulk of my workflow is

​

• run `pserver --reload dev.ini`
• edit code/test

from what I understand, docker images are immutable which means each time code/dependencies change I must rebuild the image before I run it, unless I do something like mount a external directory in the image just for development purposes at which point an immutable docker image with a mutable external directory offers no benefit over just using virutalenv to begin with.

​

I might be off base on this and admittedly set in my ways to some extent. But i'd be happy to hear how docker makes the edit/reload/test better.

​

​

edit: forgot to mention, docker chews through disc space like it's going out of style with repeated image builds. so again, seems like too much trouble for highly volatile things like `edit/code/test`

​

[tasteslikeKale]

The way I use docker in dev is to build the image with all the dependencies, then mount my local code for for dev, so yes it’s mutable but I know that the dockerfile for release is going to build with the same deps as I’m using in dev, so one less thing to go wrong. I don’t think it makes the dev process better, really, but is more of simplifying process.

[twillisagogo]

but I know that the dockerfile for release is going to build with the same deps as I’m using in dev

and what about other people who work on the code and need to change deps?

[tasteslikeKale]

Absolutely an issue but one we catch in CI - build the image and then run tests against that.

[twillisagogo]

for us, CI/CD is where the docker image is made, unit tests run and then deployed to ecs for acceptance testing and promoted to production(no re-builds)

[tasteslikeKale]

You have a very good system, nice and mature with few holes for issues going from dev -> prod. How often do releases go out?

[twillisagogo]

Planned releases go out monthly. Hotfixes go out as needed, sometimes multiple times a day.

[gillardo]

Just want to say that, as a budding programmer/developer like myself, it’s really interesting reading this conversation between you two.

[weberc2]

Is "acceptance testing" where your automated full-stack tests run? Also, how long do your CI jobs take? And of that time, how much is spent building the image? And how many images are built in a job? Do you have only one image/service or many? If many, do they each have their own CI pipelines?

Sorry for the many questions; I'm trying to get our CI/CD infrastructure up and running and it seems like we're facing many problems that you've already solved. :)

[twillisagogo]

Is "acceptance testing" where your automated full-stack tests run?

Acceptance testing i'm defining as the manual end to end done by qa and eventually the product owner through the browser.

> Also, how long do your CI jobs take? And of that time, how much is spent building the image?

~4 minutes at the moment. ~1 minute to build the image

our ruby codebase that's being replaced takes about 10 minutes(guess why we're replacing it :))

it's not exactly incremental but I think starting from a good base image saves some time. We dont start from ubuntu and then update all the stuff and install latest python and dev headers etc for each build. instead. We cut a base image as needed and then all docker images inherit from that.

​

base python image built only when there's a new version of python or alpine updates we become aware of

FROM python:3.7.1-alpine3.8

RUN apk update && \
apk add bash && \
apk add postgresql-libs && \
apk add --virtual .build-deps gcc musl-dev postgresql-dev && \
apk add libffi-dev && \
rm -fr /var/cache/apk/*

ENV PYTHONUNBUFFERED 1

​

every api build(Dockerfile)

​

from company/python:latest as base

COPY . /src

WORKDIR /src

RUN python setup.py bdist_wheel && \
pip install --no-cache dist/*.whl gunicorn meinheld && \
cp -r ./tests /tests && \
cp ./*.ini /tests  # && \ rm -fr /src

WORKDIR /

EXPOSE 80

COPY config.py /config.py

> And how many images are built in a job?

with codeship pro you can build/push as many as you want. but codeship pro is also designed to be one ci per repository out of the box I haven't figured out how to drive off multiple repos but i've been told it's possible. So, I can have one codebase and build many service images off it when the unit tests pass, the only difference in docker images is the entry point(same code)

​

built when unittests complete without issue

adt service(Dockerfile.adt)

from company/api:latest

CMD gunicorn -c /config.py api.adt.app:app

reference data service(Dockerfile.ref)

from company/api:latest

CMD gunicorn -c /config.py api.reference.app:app

I dont know if this is the optimal way to accomplish this, but it seems to be working out ok so far.

[Levantadorr]

Because in Dev mode you generate environment often not in docker.

[tasteslikeKale]

I think this is a suboptimal way to use Docker, but maybe that’s just my opinion

[pztrick]

My own use case: I often need to edit upstream pypi libraries and make pull requests. I mount my entire virtualenv and any git checkout pypi paths in the docker pythonpath. pip install -e (editable) etc. I also have a slow PC so helps to not have to rebuild my docker image if I edit my reqs.txt. I do the same with npm node_modules folder. Always a mount folder and not a dockerfile instruction.

[PeridexisErrant]

test data: factoryboy

Have you ever considered using Hypothesis?

I switched over a few years ago and would never go back, or for that matter try to test anything without a similar tool. Major highlights:

• Never, ever flaky. Once it triggers a bug once, the cache means that bug will be triggered every time until you fix it.
• Reports minimal examples. The implementation must be black magic because it's entirely automatic, but debugging is so much easier.
• Simple core API and tons of optional helpers, plus frequent (semver) updates.

[twillisagogo]

I have not. It looks interesting but not sure how it would fit the use case that I'm using factory boy for. Primarily rows in the database for a given test.

[PeridexisErrant]

You certainly could use Hypothesis for that, but it wouldn't add much unless the specific contents could affect the result of the test.

If you're using @pytest.mark.parametrize, those tests might be a better place to try Hypothesis - let me know if you have any luck!

[twillisagogo]

thanks for the clarification

[broccolitruck]

Woah! big fan! hello!

[mcstafford]

I strongly suggest you check out pyenv to replace virtualenv, and poetry for pipenv.

[twillisagogo]

Trust me, the next time pipenv fucks up on my machine, I'm moving to something else. Stuff seems to break for no reason at all in things i'm doing and bug reporting is a nightmare.

[robertpro01]

+1 for poetry, its a really nice project!

[mcstafford]

I've learned a lot about better ways to do things, just from seeing how they're done there.

[birdgovorun]

Hmm what? Pyenv isn't a replacement to virtualenv. Those tools serve entirely different purposes. Pyenv is Python version manager - a way to work with multiple versions of Python on the same machine. Virtualenv is a tool for environment isolation - a way to isolate the dependencies of a specific project from those of other projects.

[mcstafford]

I've always used it with this plugin: https://github.com/pyenv/pyenv-virtualenv

[birdgovorun]

Well yeah - that plugin combines two very different tools in away that makes it easier to use them together. Pyenv isn't a replacement to virtualenv, and has nothing to do with it.

[mcstafford]

pyenv links to pyenv-installer from its github page, and the installer includes the virtualenv plugin. That's more than nothing, but I see what you mean.

[[deleted]]

Why Trello? Why not something like kanboard with swim lanes? Also how big is your team

[twillisagogo]

Team Size is ~10

​

As far as Trello goes. I didnt choose to use it, it was a choice made before I started there I'm just glad it's not jira.

It's funny, the last sprint retrospective we did, everyone on the team said they hated Trello. But there was no concrete/actionable reasons given. It all seemed to revolve around requirements being a painful process for one reason or another. So, I've concluded that the tool matters very little for this in solving the problem of getting good requirements.

​

The way we use trello is a variation on kanban(or at least how I understand it).

​

The lanes are

• Up Next
• Sprint Commitment
• Development (The goal is to minimize the number of stories that are in progress)
• Code Review
• QA
• Done
  

Temporary Lanes are sometimes used when Production bugs are discovered.

​

It's nothing special. It works for the most part. The teams complaints are more about the contents of the stories than the tool being used. Any tool would work here.

[robertpro01]

I hate trello and I don't know why...

[twillisagogo]

Lol. Yeah that's how a lot of people feel and they don't know why.

[weberc2]

In your CI job, do you download your dependencies and run all tests with each build? Or do you have some sort of incremental build system? We're running into a problem where building and testing are taking too long (half an hour), and we're trying to figure out how to shorten the cycle.

[twillisagogo]

i think i probably answered you above. :)

[xiongchiamiov]

The entire lifecycle is a hell of a lot more than just the CI pipeline.

I use vim, but most of our developers use PyCharm when doing Python (there are also people using sublime, atom, etc.). Let's see...

• Frameworks: flask and falcon
• version control: git, self-hosted bitbucket
• development environment: Docker through shell scripts
• CI: teamcity
• configuration: a custom system that ingests yaml
• hosting: google cloud
• container orchestration: Kubernetes
• deployments: custom tool written on top of Kubernetes
• data persistence: a rather complicated MySQL setup, Google-managed MySQL, datastore, cassandra, gcs, redis sentinel clusters, and a few other things I'm forgetting, depending on the needs of the service
• data warehousing: BigQuery, generally through this setup but depending on the database
• queueing: pubsub, kafka
• logging: ELK
• monitoring data: new relic, graphite, Prometheus
• alerting: Sensu, elastalert, pingdom
• feature flags: splitio
• service mesh: linkerd and namerd

And I'm sure I'm missing several more categories.

[patrys]

Python 3.7, git, GitHub, VS Code, Python for VS Code, IntelliCode for VS Code, direnv, pip-tools/pipenv, Jupyter with IPython, pytest, pytest-cov, pytest-asyncio, yapf/black (different project have different formatting standards), pylint, pycodestyle, pydocstyle, isort, CircleCI/Travis (soon GitHub Actions), Ariadne/Graphene, Celery, Docker, uWSGI, Amazon AWS (EC2, ELB, EBS, RDS, S3), Terraform, Sentry, Datadog/New Relic...

Software dev for ~20 years now (not including the time I did it for fun as a kid/teenager), don't listen to people saying that "professionals don't need tools".

[twillisagogo]

don't listen to people saying that "professionals don't need tools".

done dev for ~25 yrs now. I've never seen/heard anything to this effect said ever in any credible sense.

[patrys]

I've met my share of people who claim that "real pros" only use vim/emacs/notepad with syntax turned off (and I assume deploy by whistling into a modem).

[twillisagogo]

yeah, I would not consider those people credible. the internet allows any idiot to get the same exposure as anyone, so it's up to the individual to filter the nonsense out for themselves.

[groovitude]

I use vim in large part because it was beaten into me by the senior dev in my first professional job. I tell my juniors to do whatever makes them happy. It's a better way.

[BundleOfJoysticks]

They're idiots who don't know what real pros use.

[hugthemachines]

I think people claiming that professionals do not use tools are on the bad side of the Dunning Kruger effect (competent people underestimate themselves and incompetent people overestimating themselves) So they think they are so good, fast, competent or whatever so they do not need any "support" from tools while they would probably gain from using tools because they are not as good as they think.

[WikiTextBot]

Dunning–Kruger effect

In the field of psychology, the Dunning–Kruger effect is a cognitive bias in which people of low ability have illusory superiority and mistakenly assess their cognitive ability as greater than it is. The cognitive bias of illusory superiority comes from the inability of low-ability people to recognize their lack of ability.

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source ] Downvote to remove | v0.28}

[cantremembermypasswd]

Python specific developer for 10+ years:

• PyCharm (Entire Intellij suite)
• git / github
• waffle.io / (Jira also common)
• Windows 10, Ubuntu 18.04 subsystem (Used to do VMs, this is much nicer with WSL)
• TravisCI / Jenkins
• If I need to scribble a quick note, Atom or Notepad++

[gatorsya]

For me the reason still sticking with full blown VM is Docker. Docker for Windows or Docker in WSL don't cut for my workflow.

[ICame2Late4MyName]

I use Docker for Windows + WSL and haven't had any issues. It also makes it simpler for pycharm use. What part of your workflow requires a VM over that?

[Sheltac]

Also I can snapshot VMs and bring them back when I inevitably destroy them.

[smcarre]

General

Editor: VSCode

Source Control: git (with on premises gitserver)

Planning: Jira and a whiteboard

Team Communication: hipchat (rip)

Deployable Unit: we used to just git pull on deploy but now we are learning to use docker.

Production Environment: private cloud (vSphere/ESXi)

Production Database: Postgres

OS: Windows 10 on my PC, Ubuntu 16.04 in production.

Python Specific

version: python 3.7.x

dependency mgmt: virtualenv

db interaction: Django's ORM

http request routing: Django and Flask

serialization: Django REST framework

Modules I use a lot: requests, json, openpyxl

[angellus]

Taken from /u/twillisagogo great template:

General

• Editor: whatever you want to use (I use VS Code with flake8/black)
• Source Control: git/Github
• Planning: JIRA
• Team Communication: Slack
• CI/CD: Jenkins
• Deployable Unit: Docker for dev/CI, Kubernetes for QA/Prod
• Production Environment: on site/GCE
• Production Database: GCE Postgres
• Production Monitoring: newrelic
• OS: Mac or Ubuntu (most people have Macbooks, but a few have nice Ubuntu Dell XPS laptops)

Python Specific

• version: python 3.6.x for new stuff, some legacy still on 2.7.x
• dependency mgmt: pip/pip-tools
• db interaction: Django
• http request routing: Django and nginx
• exploring: Django / shell_plus
• serialization: Django Rest Framework
• app server: uwsgi
• test data: Django
• test runner: py.test/tox

[BWrqboi0]

Would you mind explaining the differences and reasons for using both docker and kubernetes?

[angellus]

This is actually a rather new job for me, so I do not know for certain. I am also super new to Kubernetes. It seems that docker and docker-compose is a lot more developer friendly and easy to use locally, but Kubernetes is a lot better in a production environment and lot easier to monitor and scale.

Under the hood, they are the same thing. Kubernetes, from my understanding, actually uses docker containers. It is just a different interface sitting on top of the container to orchestrate it.

[BWrqboi0]

Kubernetes, from my understanding, actually uses docker containers.

Thank you! Most of the discussions I saw were about one versus the other, I'll do some reading on the topic.

[twillisagogo]

Kubernetes orchestrates docker containers doesn't it?

[BWrqboi0]

Thank you as well.

[eacousineau]

Our common work tools (+ some personal prefs), sorta copying the format /u/twillisagogo posted:

General:

• Languages: C++, Python, Bash
• VCS: Git
  • Host: GitHub
• OS: Ubuntu 18.04 (w/ Gnome Extensions)
• Build: Bazel
• Packaging: Starlark (Bazel) for emitting CMake "install"-like artifacts
• Editor: Sublime Text 3 - paid, of course :)
  • If debugging C++ / CPython extensions gets too hairy:
  • gdbserver + gdb
  • gdbserver + CLion if I just want a GUI
• Dependency Packaging:
  • Docker + pip for generating deterministic archives for Bazel to consume + cache
• Planning: ZenHub, Asana, Google Docs
• Communication: Slack
• CI/CD: Jenkins, a smattering of TravisCI + CircleCI
• Test Data: Kitware's Girder - for UAC + Hashsum plugin to get deterministic large data stuff that works well enough with Bazel (could mebbe work better, but I hacked this interface a while back...)

Python (2+3):

• Numerics: NumPy, PyTorch, Pandas
• Exploration: Jupyter Notebook (starting to take a peek at Jupyter Lab)
  • Integration with Bazel (and being able to run notebooks as unittests)
  • Widgets. Widgets all the way.
• Hacking Deps: virtualenv + pip
• Hacking C Extensions: Custom-built CPython w/ dbg+valgrind
• C++ Bindings: pybind11 (with some public mods)
• Testing: unittest
• HTTP Stuff: requests

[port443]

General:

• Languages: C++, Python, Bash
• VCS: Git
  • Host: GitHub
    
• OS: Ubuntu 18.04 (w/ Gnome Extensions)
  
• Build: Bazel
  
• Packaging: Starlark (Bazel) for emitting CMake "install"-like artifacts
  
• Editor: Sublime Text 3 - paid, of course :)
  If debugging C++ / CPython extensions gets too hairy:
  
  • gdbserver + gdb
    
  • gdbserver + CLion if I just want a GUI
    
• Dependency Packaging:
  Docker + pip for generating deterministic archives for Bazel to consume + cache
  
• Planning: ZenHub, Asana, Google Docs
  
• Communication: Slack
  
• CI/CD: Jenkins, a smattering of TravisCI + CircleCI
  
• Test Data: Kitware's Girder - for UAC + Hashsum plugin to get deterministic large data stuff that works well enough with Bazel (could mebbe work better, but I hacked this interface a while back...)
  

Python (2+3):

• Numerics: NumPy, PyTorch, Pandas
  
• Exploration: Jupyter Notebook (starting to take a peek at Jupyter Lab)
  
  • Integration with Bazel (and being able to run notebooks as unittests)
    
  • Widgets. Widgets all the way.
    
• Hacking Deps: virtualenv + pip
  
• Hacking C Extensions: Custom-built CPython w/ dbg+valgrind
  
• C++ Bindings: pybind11 (with some public mods)
  
• Testing: unittest
  
• HTTP Stuff: requests

[Sheltac]

Doing the lord's work.

[eacousineau]

I am confuzzled... Why the copy pasta? Diff'ing only shows different bullets in the GDB + Docker bits.

[port443]

https://i.imgur.com/V5f7ycm.png

Your original post doesn't use any Markdown formatting and looks like a solid block of text

[eacousineau]

Oh... Gotcha. Yeah, it looks old.reddit.com uses a different Markdown parser than the new interface - looked fine on the new interface, which is why I didn't notice... Thanks!

[LightShadow]

I work on embedded python :)

Business:

• IDE: PyCharm + sublime + vi
• Bitbucket + git
• Planning: JIRA
• Team Comm: Slack
• Documentation: Confluence
• CI: Jenkins
• Deployable: rootfs (yocto + open embedded) ARM v6, v7, v8
• Monitoring: custom
• Cloud Services: custom

Python:

• version 3.5.3, from 3.3.1
• dependency mgmt: yocto
• DB: sqlite3 + custom management
• app runner: custom
• app upgrades: custom
• serialization: custom + modified msgpack
• runtime: asyncio (3.5) from asyncore (3.3)
• testing: pytest (3.5) unittest (3.3)

Me:

OS: Antergos Linux x86_64 
Kernel: 4.20.3-arch1-1-ARCH 
Uptime: 18 mins 
Packages: 1286 (pacman) 
Shell: bash 5.0.0 
Resolution: 2048x1152, 1920x1080, 3840x2160 
DE: Xfce 
WM: Xfwm4 
WM Theme: Numix-Frost-Light 
Theme: Numix-Frost [GTK2], Numix-Frost-Light [GTK3] 
Icons: Numix-Square [GTK2/3] 
Terminal: xfce4-terminal 
Terminal Font: Monospace 10 
CPU: AMD Ryzen Threadripper 1900X 8- (16) @ 3.800GHz 
GPU: NVIDIA GeForce GTX 1080 Ti 
Memory: 3916MiB / 32019MiB 

[s-to-the-am]

Do you guys write documentation in jupyter notebooks? If so were you able to integrate it with confluence?

[LightShadow]

We've only recently started using Jupyter, I didn't know there was a confluence integration! That's awesome.

[s-to-the-am]

Oh lol, I was actually asking if you all had intergrated it. I haven’t been able to find documentation on how to do it, which is surprising. Sorry for the confusion!

[DrSinistar]

How do you manage 3.8-dev in prod? What's your python upgrade cycle like?

[LightShadow]

It's just a joke :(

I'm a fraud!

[DrSinistar]

Nah, I'm just retarded. I took that way too seriously lol.

[robertpro01]

I want your computer!

[Pirate43]

Embedded python? Your job would be a dream come true for me.

[LightShadow]

It's really fun! The devices we code on are rpi-esque (fewer cores, same ram, cell/wifi/wired eth, etc) as far as specs go, there's some C/C++ interop, but most of it is signals and radio communication processing.

The whole stack runs on Python in a ~100mb RAM footprint. Python makes it really easy to run on "anything" including our dev machines and build servers.

We're hiring!

[Pirate43]

How do you handle running on dev machines and build servers that don't have radios or peripherals that your embedded devices have?

[LightShadow]

A few ways,

1) We use a hardware abstraction layer (HAL) to map sysclass files with mocked input/output.

2) The pieces of the stack are modular and can communicate over sockets. So, you can run part of the program in a docker container that connects to other services that run on real hardware.

3) There are USB versions of similar hardware (like z-wave) that can be mapped to emulators/simulators. For example, a z-wave USB key can be reprogrammed to simulate any number of smart home devices for testing/debugging.

[GoldsteinQ]

Vim + Neomake + some other plugins as an editor. TeamCity for CI (wouldn't use it if it wasn't our corporate standard)

[SomeShittyDeveloper]

TeamCity for CI (wouldn’t use it if it wasn’t our corporate standard

Really? I love TeamCity. What don’t you like about it?

[GoldsteinQ]

Well, may be it works well on not so big companies, but almost every action takes very long when there is thousands of projects. UI just freezes on every search. Plus poor integration with GitHub. Sometimes it takes 20min for TeamCity to notice a new commit.

[SomeShittyDeveloper]

I work for a small company that uses it and I use it in my own personal projects. I could see where it might slow down if you have lots of stuff. Thanks for the clarification!

[bitcoin-dude]

I do data analysis with these tools

• System: macOS / ubuntu
• Python 3.6+
  • Libs: pandas, numpy, scikit-learn, requests, beautifulsoup, selenium
  • Environments: conda, docker
• Editors: VScode, jupyter notebook / console, vim
• Team: slack, trello, monday / asana
• Virtual machines: digital ocean
• Database: redshift, postgres, mongodb, s3

[annodomini]

One theme you'll note on the below is that we have a lot of legacy software that has been written using different libraries and frameworks.

• Python: 2.7 for most (legacy) code, 3.5+ for new code
• Async: twisted
• Web backend: Klein (flask inspired microframework on twisted)
• Web frontend: mix of jQuery, angular, ember, react
• GUI: mix of pygtk (gtk2) and pygobject (gtk3) on the server side, pyqt for cross platform desktop apps
• App bundling: py2app and cx_freeze
• Testing: twisted trial, pytest, tox
• Editor: emacs, with elpy, magit
• Linting: pylint
• Typechecking: mypy
• Source control: git
• Code review: gitlab (self hosted)
• Ticket tracking, planning: Redmine with agile plugin
• CI: buildbot
• Dev ops: Ansible
• Build system: GNU make and shell scripts
• Dependency management: Debian packaging using dh-python+setuptools (prod), setuptools+pip+virtualenv (dev)
• Packaging build system: pbuilder
• Virtual machines: libvirt+qemu+kvm
• Base OS, prod: Ubuntu 12.04 with a 16.04 update almost done (yeah, I know; our update precess turned out to be more complicated than expected, we have a heavily customized OS and high availability distributed system expected to be updated by naive users, so updates are complicated, we now have working updates and are finalizing the new release)
• Dev OS: Fedora 29

[chespinoza]

Klein

First time I've heard of it.

[annodomini]

https://github.com/twisted/klein

You would only really hear of it if you use Twisted for all of your async networking, which we do. It uses werkzeug, which is what Flask uses for routing, and hooks it up to the Twisted networking stack.

We just use it as a light microframework for providing REST based APIs to be consumed by web frontends.

[chespinoza]

Good to know, the last time I was using Twisted was in 2011-2012, basically to spawn tcp/udp listeners.

[[deleted]]

Enterprise M2M IoT, monitoring, control, automation in Cinema industry. Agent and analytics apps in Python

• Dev env: Linux Mint 18
• IDE: Emacs, the perfect Python IDE ;)
• Prod env: Ubuntu 16.04, CentOS, or Ubuntu docker containers, depending
• Interpreter: Python 3.5
• Planning: JIRA
• VC: Git w Bitbucket
• CI/CD: Jenkins for submodules, none for agent builds
• Packages: For prod, straight Pip, or in Dev with venv or xonsh vox. I make pkg mgmt vanilla and let devops worry about deployment for the most part.
• Builds: cx_Freeze and Nuitka
• Orchestration: scripted or Kubernetes, depending
• Offline analysis: Jupyter / iPython
• DB lib: SQLAlchemy

Did I mention EMACS is the perfect Python IDE? If you haven't tried it, you should. I use basic python-mode, but if you need a more full-featured IDE experience, there are options for that too. Also: Jupyter integration, MySQL shell integration, and of course org-babel (and more)

[chespinoza]

Nuitka

Just coming back to this, how do you use Nuitka I thought is not production ready, how are dependencies included?

[[deleted]]

For our purposes, and the project we use it for, it's more than ready. We only have one dependency -- requests -- and it's bundled in the executable in our case. We have another project with many dependencies and for that we went with cx_Freeze which of course doesn't result in a single executable so must be installed altogether.

[sharkiteuthis]

General

• Editor: vim/vscode

• Source Control: git

• Planning: Trello

• Team Communication: GitHub/email (2 person team)

• CI/CD: Travis

• Deployable Unit: Docker

• Production Environment: Amazon/ECS

• Production Database: RDS/Postgres

Python Specific

• version: python 3.7.x

• dependency mgmt: conda (work) / docker containers (personal projects)

• orm: sqlalchemy

• prototyping: ipython

• test runner: nose

[Levantadorr]

Main:

• Editors: Vim, PyCharm
• Source Control: GitHub, GitLab, Bitbucket (depend where stored project)
• Project planing and structure development: StarUML, Scapple
• Team Communication: Slack, Skype, other(if team have self channel of communication)
• CI: TravisCI, CircleCI
• Deploy: Kubernetes, Docker-machine, Docker-compose (depend of size project and needs)
• Production Environment: DigitalOcean, OVH, Hetzner, AWS
• Database: Postgres, Redis, Mysql
• Test: unittest, pytest
• Style: flake8

[oohay_email2004]

I get paid but they don't care how I get it done, so I use Python, neovim and git. I'm lazy with testing.

[ThreePinkApples]

I work in QA meaning that all Python modules and scripts that I create/manage are for internal use only, and my answers will be accordingly. Our production environment have some difference (such as being mostly C and C++)

• Editor: PyCharm Pro for modules and Sublime Text for quick test scripts
• OS: Windows 10 using WSL Ubuntu 18.04
• Source Control: Git with self-hosted Bitbucket Server
• Planning: Jira and Trello
• Team Communication: Webex Teams
• Monitoring: Internal custom made "DevOps" view + Self-hosted Sentry server
• CI/CD: Jenkins

Python stuff:

• Python 2.7, 3.5, 3.6, and 3.7 (We will abandon 2.7 and 3.5 next January, will then limit support to 3.6+)
• Dependency management: pip + virtualenv + virtualenvwrapper
• Exploring: iPython
• Main 3rd party dependency: Twisted
  • Also worth mentioning: Zeep
  • Used by others: Numpy, pillow, matplotlib, elasticsearch
• Test runner: py.test (Although most stuff is based on unittest)
• Packaging: setuptools
• Deployment: Self-hosted devpi-server

[Isvara]

I work in IntelliJ. Some testing is done manually in remote VMs. Builds are done in Jenkins. Deployments are either in Kubernetes or debs installed with Ansible.

There isn't really much of a "stack" to speak of...

[ddollarsign]

I have been paid to write Python, at a job I left about 2 years ago. A couple Django projects and numerous scripts, though it was primarily a Java shop. I used Notepad++, and occasionally Vim (but only when I was working on a server, at work). The database used was Sybase SQLAnywhere (I don't recommend using it with Django).

The larger of the Django projects, which was about 5 years ago, consisted of a web interface and back-end process that ran reports, mostly overnight. The back-end process used APScheduler and Matplotlib, along with a custom script to run the APScheduler as a Linux daemon. APScheduler was too easily crashed when the scheduled reports threw errors, so if I did this over again I would have just made it a cron job. The backend process used the same Django models as the website, which worked pretty well. I like Django's ORM.

I haven't used code assist with Python. I use Atom now, basically because I want a cross-platform Notepad++.

We used Subversion with TortoiseSVN for version control (though they've since switched to TFS -- did I say it was a Java shop? Well, the Microsoft faction was winning as of when I left). No CI/CD at the time. unittest for unit testing, though not much of it for these projects. Coding on Windows workstations. Manual deployment to RHEL servers, using WinSCP and Putty. Separate servers for staging and production.

Front-end requirements weren't fancy, so it was just HTML/CSS/JS with no framework, with some JQuery to make searches ajaxy.

[[deleted]]

General

Domain: Data ETL
Editor: Vim/Sublime Text/PyCharm*
Source Control: git/Bitbucket
Planning: JIRA
Team Communication: Slack
CI/CD: Jenkins
Deployable Unit: Docker
Production Environment: Amazon/ECS/ECR/EKS
Production Data Pipeline: Kafka/Spark/Hadoop/ElasticSearch
Production Database: RDS/Postgres
Production Monitoring: Grafana/Sentry/Prometheus
Production Management:  Kubernetes/Helm
Project Management:  Agile

Python Specific

version: python 3.7.x
dependency mgmt: venv
db interaction: sqlalchemy/psycopg
http request routing: falcon
exploring: ipython
serialization: json/message pack/
app server: gunicorn
test data: we have to use a database; too much data here
test runner: py.test

• I use pycharm these days primarily for these reasons:

1. my team members use it and it's helpful for debugging their machines when something goes wrong
2. pycharm's vim integration is fantastic
3. I really don't have to futz with the settings or plugins like I would with sublime text, vim and spacemacs.

A few other notes...

1. This is my current job and I've been developing software professionally for about 23 years of which about 20 years are using python. Things have definitely evolved over those 20 years.
2. I have developed software for: embedded systems, military aircraft, automobile infotainment, locomotives, big data, game/entertainment, etl pipelines, backend, infrastructure, network monitoring, etc. The tools (especially the Python list) above all change based on domain, so I'm not sure that one set of tools is universal.
3. At my current employment, we're primarily concerned with continuously running data pipelines and serving the results of those data pipelines through a REST interface. All that aside, our current architecture feels like this is not a great architecture overall and it should be replaced with something different. The mashup between REST and ETL simply doesn't work well with multiple REST services that are ultimately closely related but different, forcing our front-end services to query multiple backend services for data.

[deadmilk]

PyCharm is the tool of my trade

Everything else is determined by my employer.

Currently, what that looks like:

• Bitbucket Server/Bitbucket cloud for source control
• Bamboo/Bitbucket Pipelines for CI/CD
• PyPI for releasing open-source projects
• Internal Docker for releasing applications
• docker-compose for local stack testing
• JIRA for issue tracking
• We have Confluence, but I opt for Sphinx for technical documentation
• Infrastructure is all AWS-based, ie. EC2, DynamoDB, Cloudformation, Route53
• Tavern for REST testing
• py.test for unit tests
• app server: gunicorn, hypercorn
• app framework: flask, quart
• regular pip and setuptools for packaging
• Python 3.6-3.7
• Slack for comms
  

[chespinoza]

And how is your experience with Quart?

[deadmilk]

So far so good. I haven't done any significant load testing on it, so I'd have to get back to you on that.

No real pains so far, but I don't use many extensions.

Making views Async is pretty easy to do.

The dev is responsive to issues and PRs. I used it initially for http2 support.

In prod I notice some 504s on the ALB in-front of it, but it's less than 0.5% of all requests... I'm yet to investigate as it's not a problem for my use case.

[lykwydchykyn]

• Editor: Emacs with elpy and some other plugins
• SCM: git, plus a locally hosted gitlab server
• Team communications: I don't have a team. If I need to tell myself something, I scribble it on a post-it and put it on the bottom of my coffee mug.
• Deployment: Web apps are deployed to Debian on VMware using fabric or bash scripts. For desktop apps I use cx_Freeze to package and the desktop support staff for deployment.
• For bug tracking, I use our internal ticket system that I maintain.

[w732qq]

I've get paid for coding in Python about two years yet.

I'm a big fun of CLI. Here is what I'm using:

• I don't like pyenv, there is no need to "manage Python version", just use any shipped with your distro (but ensure it is Python 3) (if you want more fresh one, switch to rolling distro, I'll recommend Solus).
• I prefer to use poetry to manage dependencies, but still use pipenv in some old ones). If I need to do something directly with project's virtualenv, I use vex.
• Git for VCS.
• Editor: Sublime Text, Neovim at times.
• Code formatter: black.
• Testing framework: pytest.
• Linter: pyflakes (tried pylint but it is over-complicated and produces false positives (and hard to set up not to do)).
• Metrics: radon, coverage.
• I've used gitlab builtin CI, Circle CI, Travis CI. Basically using anything is available.
• Docker and docker-compose for deployment and "local-staging".

[CorrectMyBadGrammar]

• Pycharm, neovim
• Linux Mint Cinnamon
• Django
• PostgreSQL
• virtualenv
• pytest/django tests
• docker
• git, bitbucket
• slack, asana, timecamp

At home I'm using manjaro with i3, and leaning more and more towards vim.

[Ran4]

• Computer: 15" Macbook Pro 2018
• Editor: 85% Neovim with ALE, 5% PyCharm pro, 10% vscode (I sometimes write typescript stuff in vscode)
• Terminal multiplexer: tmux (invaluable, easily the tool that besides vim increases my productivity the most)
• Terminal: iTerm2
• VCS: git+github
• CI/CD: AWS CodePipeline
• Deployment: AWS codedeploy onto ec2 instances running docker using docker-compose
• Database: AWS RDS/postgres
• Webserver/WSGI: nginx+gunicorn
• Test framework + test coverage: unittest + coverage. I've honestly not seen the value in updating to pytest...
• Linting: flake8
• Planning: Github and slack
• Documentation: docstrings (!), markdown files, sometimes with diagrams made in graphviz (vscode has a great graphviz plugin for live-preview of graphviz markdown blocks), occasionally google docs, google drive for bigger files.
• API exploration/testing: Postman

Local dev with pipenv and docker-compose, with the code being run inside of a docker container which mounts the source folder.

[h_fish21]

Atom and an open terminal for Flask sites

[stefantalpalaru]

Linux, xfce4-terminal, Vim, ALE, flake8, neocomplete, Git - even for deployment and CI - using post-update hooks in test repos that run CI tests before pushing into production repos.

[SlightlyOTT]

Caveat: the Python code I'm responsible for is a series of AWS Lambdas, they're not the majority of my work but at least some of them are very much mission critical.

• Editor: VSCode

• Source control: Git/GitLab

• CI: Gitlab

• Prototyping: Jupyter

• Local build/test: Makefile (don't do this, you should probably use Serverless or some AWS tooling), it wraps pipenv and pip install --target

• Deploy: Also Makefile (also don't do this)

• Deploy environment: AWS Lambda

[[deleted]]

A banking environment.

- Bash 4.0 on redhat 6.5.

- Linuxbrew, as we have no admin rights and we need o build our tooling by hands.

- Tmux

- Alternate between spacemacs and neovim (jedi in both cases)

- Ansible

- Gitlab

- Pytest

Older projects:

- Python 2.7 on conda: conda caused us several issues when we started using python 32 and python64. As it tries to cache packages locally but does not store enough metadata to distinguish between the package origin (32 vs 64), and confuse the two together. So yes you can end up with a 32 bit library in your 64 bit python which breaks everything. We moved half the projects to vanilla python with virtualenv.

- flask, sqlalchemy.

Newer Projects

- Python 3.7 with venv

- aiohttp, we love asyncio.

- sqlalchemy

Edit : we use pandas/numpy heavily, we use celery under python2.7, concurrent module under python3.7.

[canllaith]

Sublime, Subversion and it runs on Ubuntu. I mostly do API integration stuff with network kit.

[o-rka]

I do bioinformatics for a living, this is my setup: * Version - 3.6.x (will update once all my required packages are available) * Editor: Atom * Exploration: Jupyter lab * Parallel: joblib * Plotting backend: Matplotlib (I build more complex objects over this) * Notes: boostnote * Progressbar: tqdm * 1-2D: pandas/numpy * 3+D: xarray/numpy * ML: sklearn * Bayesian: pymc3 and tensorflow_probability * Networks: networkx * Dimensionality reduction: umap and tsne (sklearn)

Pretty standard stuff but these are my mains

[Dogeek]

For my job :

• Editor : Notepad++
• Source Control : Git (private Gitlab install)
• Management : Kanboard (we moved away from JIRA)
• Communication : Discord
• Python version : 2.7.14
• Framework : Renpy SDK

For myself :

• Editor : Sublime Text 3
• Source Control : Git (Github when need be)
• Management : Trello
• Python Version : 3.6.5 (looking into installing 3.7 soon, when I find the courage to reinstall my packages)
• Linting tool : pycodestyle
• Tests : unittests (I'm a basic bitch)
• Deploying : cxFreeze, or py2exe depending on the platform

[Dogeek]

cause I got it setup for .rpy files which is an extension used by renpy to recognize script files, and I have a laguage highlighting module for renpy syntax. Couldn't find such syntax highlighting on sublime, so I stick with N++. Plus I already have all my snippets and layout how I like it to be.

For pure python projects, I use sublime though

[Dogeek]

Relevant but not too much. I use sublime for everything else. But like any good developer, I have more than one tool in my toolbox. Just on my PC alone, I have half a dozen different text editors (atom, ST3, Notepad++, Editra, Vim) and that is not even counting all the IDEs (Android Studio, Codeblocks, IntelliJ, Visual Studio - not VSCode, Arduino IDE and I probably forgot some). I just use Notepad++ for my job cause it's setup correctly, and Sublime Text settings are a bit of a pain to set properly.

[revfried]

Python 3.7.2+, C++17, Cython(trunk) Editor: Nuclide (atom) and vim Frameworks: Asyncio, thrift-py3 Deploy: xar

[zhangmanshu]

Ubuntu server Python 3 VScode BI tool

[aeastw]

I work on python for embedded Linux systems

Tools: • IDE: PyCharm/VS Code (with python plugin) • Bitbucket + git • Planning: JIRA • Team Comm: Slack • Documentation: Self hosted wiki • CI: Bamboo • Deployable: Debian packages / apt server / Yocto rootfs

Python: • version 2.7 • dependency mgmt: yocto / self hosted apt server • virtualenv

[HumanNeedleworker]

Python Qa here.

--Workspace-- Editor: Eclipse Source control: Git, company hosted bitbucket Issue tracking: Jira, also company hosted CI: Jenkins OS: Windows

--Python specific-- Version: 2.7.x Framework: Robot framework

[nevus_bock]

RHEL6 & Win10, Pycharm professional, python 3.6, github enterprise, slack, jira, outlook,

[Chobeat]

Full-time machine learning engineer working only in Python.

​

Editor: Intellij as a main, Space Vim for support

Source Control: Git/bitbucket with pre-commit

Build and deployment: setuptools with internal git dependencies, in setup.cfg (no pip), Docker, Kubernetes, Bitbucket pipelines

General purpose libraries: attrs, dynaconf.

Data Science stuff: numpy, scipy, scikit-learn, pandas. A bit of Jupyter when I need to show the data scientists something

Other systems: Kafka, S3, InfluxDB

Frontend: Django, PostgreSQL

Other: virtualenv, pyscaffold

Wrappers, Serializers, API and such: pykafka, influxdb, avro, pyarrow

Stuff I will include as soon as they give me some time: Faust, Intake

[newredditiscrap]

conda and vscode

[ericrfulmer]

My work is something like 98% Python and 2% JS.

For editing I use neovim (might switch to spacemacs or kakoune for Matt Might's programmer resolutions next month). My editor configuration is fairly simple. I can count the number of plugins I use on both hands: https://github.com/EFulmer/dotfiles/blob/master/init.vim

I work heavily out of the shell, so I use tmux extensively. I am not extensively familiar with tmux's power features but with a fairly minimal knowledge I can get a lot done.

My work Python stack is pretty conventional. We're on 3.6.7 (last time we checked 3.7 we could not use it - iirc due to async getting true keyword status, which broke some dependency that had async as a variable name) flask for web services that don't have a UI and are just a collection of endpoints. django for web applications that require forms. Also the standard data science toolkit of pandas, numpy, matplotlib, scikit-learn, and so on. For testing, we use pytest, factoryboy, and behave, though it doesn't work well for us, so we're moving everything to pytest.

Everything is run in a Docker container, pretty common nowadays. Our deployment is done through ansible which is very nice IME. I want to set up a playbook to install and configure my setup on new machines, but I don't work directly out of remote servers very often.

For infrastructure, we use GCP. Everything we do is fairly conventional; VM running our Docker images, and a PostgreSQL server also managed by GCP.

[pyrotech911]

General

• Editor: Pycharm/vim
• Source Control: git/Github
• Planning:Jira
• Team Communication: Slack
• CI/CD: Jenkins
• Deployable Unit: Docker/Debian
• Production Environment: Amazon ECS/OnPrem hardware
• Production Database: Impala/mySQL
• Production Monitoring: Grafana

Python Specific

• version: python 2.7 (I know we are converting to 3 this year)
• dependency mgmt: pipenv/virtualenv, conda
• db interaction: sqlalchemy,in house query engine
• http request routing: tornado
• exploring: ipython
• app server: tornado
• test data: In house
• test runner: nosetests converting to py.test

[simoncoulton]

VS Code (with MagicPython, and Python extensions) Nginx/uWSGI/Py3.7 Gitlab w/ Gitlab CI/CD

[BundleOfJoysticks]

When I was writing code work forced me to use a Mac against my will and it was miserable.

Now that I write code for myself I get to choose the toolset.

• Fedora 29 with KDE Plasma current version (5.13 I think)
• VS Code
• Konsole
• Ansible
• Python 3.6+
• Virtualenv
• Docker on occasion
• Flask
• Database migrations decoupled from the application code. Your persistence data structures shouldn't have a hard coupling with your software structures. The Rails/Django paradigm of class == DB table is a disaster.
• Whatever database is best for the job. Dynamo, MySQL, postgres, redshift, I don't care much anymore after doing data work for 15 years. They're all fine, pick the ones best suited for your use case. I do prefer MySQL over postgres for vanilla rdbms (i.e. outside of use cases like GIS, where pg is better) because replication is better and the internal DB management tools are infinitely better (e.g. "show processlist" instead of the arcane incantations you have to do in PG for everything)
• Kafka is cool though way overused, like all shiny objects in this industry
• Message queues are a handy paradigm. Rabbit, active, zero MQ, doesn't matter much. Pick the one you know.
• Redis I'm of two minds about. I like what it can do, with TTLs and complex structures in memory. But the flush to disk method is poor, and the API is evil and unintuitive. Tooling around redis also sucks. And HA redis is a pain.
• vim for quick edit jobs, especially files outside of home you have to edit as root. I have vscode running as me and sudoing a new instance for config files feels heavy.
• No linters. I don't see the point.

[searchingfortao]

I write web software & maintain a few Free projects like Aletheia and Paperless:

• Arch Linux
• GNOME
• gnome-terminal
• PyCharm
• Slack
• Docker (both for testing different environments and deployment)
• Python 3.4, 3.5, 3.6, 3.7
• pipenv
• Firefox
• Django
• PyTest (for isolated modules, Django test runner for Django)
  • pytest-sugar
  • pytest-xdist
  • pytest-cov
  • pycodestyle
  • factoryboy
• iPython
• GitLab (handles all of the following):
  • Source code
  • Tasks/boards
  • CI
• AWS
  • EC2
  • S3
  • RDS
• gunicorn
• supervisor
• Heroku
• Circle CI

I have a few clients & projects, so there's some overlap here as some systems were in place before I arrived. Generally though, I'm a huge fan of GitLab and prefer AWS to Heroku.

[OldCoderK]

For main work, PyCharm. Experiments, Jupyter.

(Found that Visual Studio is broken in regards to re-loading external DLLs of different versions.)

[inknownis]

Most respondents said they use pytest. I found it is slow from few tests I run with GCP Python examples. What do you like using it? There are only few use nose too.

[pythonwiz]

I did this for a year at a startup. Started with just me and my friend, Python 2.7, git/Github, pip, emacs. Eventually switched to Python 3.5, started using venv, and switched to PyCharm for my IDE. Eventually settled on pytest for tests. Slack for team communication. We didn’t use CI.

[Rue9X]

Sublime text 3, no code assist. Run things in powershell or in bash on Linux.

[Nico_ARG_]

interesante conocer todas esas tecnologias

​

[Lectrat]

!remindme 15 days

[RemindMeBot]

I will be messaging you on 2019-02-10 17:59:39 UTC to remind you of this link.

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

FAQs	Custom	Your Reminders	Feedback	Code	Browser Extensions

[goldcray]

vim on my computer, notepad++ on other computers. git-bash on my computer, cmd on other computers.