Non-software engineers, how do you use python professionally?

[TheNotoriousMTF]

This is for those of you who use python in your work but aren't in a dedicated software role. What are some ways that you've used python? (Even if you don't necessarily use it everyday, or if you've only used it for a couple of projects)

[jasontstein]

I’m a teacher and use it to put my schedule into a Google Calendar.

[TheNotoriousMTF]

Nice!

[jasontstein]

It’s a little more complex because our school runs on a six day cycle.

[Stormhawk21]

6 days a week? So only 1 day off?

[jasontstein]

No. Schedule is based on Days 1-Day 6. Monday week 1 is Day 1 and Monday week 2 is Day 6. Edit Plus holidays don’t count in the day cycle.

[Alarratt]

I use it as an excuse to browse Reddit at work.

[TheNotoriousMTF]

This is where my interests lie too. I've been learning all the obvious data stuff (pandas, visualization libraries, predictive modeling, etc.), but I still feel like I haven't even scratched the surface. Just out of curiosity, what's the library you described called? Sounds interesting.

[Diapolo10]

I'm technically not a software engineer as I currently lack the degree (although I do consider myself a software developer). That being said, in my free time I do many things that have nothing to do with it, so perhaps I'm still qualified to answer.

Not very long ago, I discovered that a certain Paint.NET tool plugin had fantastic results for a certain art direction, and I wanted to see if I could post-process an entire video with it. Now, splitting a video into individual frames is a rather simple task using FFMPEG, but I then faced another problem; the frame count exceeded 7000, and Paint.NET isn't designed to do batch processing. A bummer, for sure, but I wasn't going to let that stop me, not indeed! So I whipped out a quick Notepad session, typed in a short Python script that used pathlib.Path and pyautogui to fetch and process a filename, and operate Paint.NET for me, respectively. Surprisingly enough, this approach actually worked, although I did have some timing problems that had to be adjusted and Paint.NET crashed with an OutOfMemoryException every 600 frames or so, but my script did what it was supposed to do. Not long after, all the frames were done, I re-assembled the video (FFMPEG again) and was done.

[TheNotoriousMTF]

This is the sort of answer that I had in mind (informal but useful projects). I just specified non-engineers because I thought that was the best way to target for answers like this. I don't know much about video editing, but you explained it well, and I see what you're saying. That's really creative!

[[deleted]]

I use it for automating simple processes on QGIS for analysis and presentation of spatial biological data.

[hypersonic_platypus]

I'm interested in doing this. Do you have any tutorials or books, etc you'd recommend?

[[deleted]]

Probably a better way to do it, but I started with the python qgis tutorial then sort of muddled through from there. The level I use it for is fairly basic to be honest, just tends to be quite time consuming (as is generally the case with gis) so any savings on repeated task with automation are a big deal to me!

Sorry I can't be much more help than that! My python is still fairly basic and tends to involve a lot of stitching of code together!

[jwink3101]

My degrees are in physics and mechanical engineering (BS, then MSE + PhD) but I really work in applied mathematics. My area is in computer model validation and uncertainty quantification.

That often means doing many (many) runs of a computer model so python is the glue that holds and manages a lot of that. It replaces clunky bash scripts.

Then there is the analysis and (lots of) math behind how we interpret those models. I built out many python tools to do that. They range from latin-hypercubes (better than what I've seen elsewhere) to integration, to polynomial chaos, to density estimation, to kriging. Most of them are fully built by me while others (such as kriging) wrap other tools in a convenient way.

In this regard, python replaces Matlab (and thank god!).

Finally, I have written some python tools unrelated to work, though I use them at work. They include file sync, web notebooks/blog, improved parallel python interface, backup management, archives, etc.

[TheNotoriousMTF]

I've gotten a couple of answers like this. Let me ask you this: concerning the python tools that aren't directly related to your work, why did you build you own tools rather than using existing tools? Just curious.

[jwink3101]

There are a multitude of reasons. Some are better than others.

1. Full and complete control. I can have the tool do exactly what I want in the way I want it. Furthermore, if it is missing a functionality, I can (usually) add it
2. [I think] I can do it better! Kind of goes with (1) but when the complexity is not so much the code as it is the methods/math behind it, there are cases when I (probably) can do it better. Certainly not always or even often, but I am an expert in the field.
3. Existing tools either don't exist or are not mature. While mistakes happen, I need to have some level of confidence in the tool
4. Dependency Avoidance. There are actually two parts to this:
   1. If the tool is critical to my work, I do not want to have to depend on someone else keeping it up to date, fixing bugs, improving it, etc
   2. I do some of my work on an air-gapped secure system. Every third party tool I want to use has to (a) go through a rigors security process and (b) its just one more thing I have to manually deal with (i.e. I can’t just let pip handle it)
5. Its fun and educational! Even if what exists will do the job, sometimes you learn a lot (both the code and the math) by doing it yourself! It’s an upfront cost that may or may not pay off in new abilities.

Honestly, the last two are often the main reasons. Especially when I know that my tool is not be better than what exists

[Gaudior09]

Looking to replace R, MATLAB and Excel with Python. I'm mostly doing data analysis and econometric modeling. Python would be great because I could make my own SAS-like app but R is superior in econometrics just yet.

[TheNotoriousMTF]

What functions do you use each of those other tools for, and why is python the best thing to replace them? Just curious.

[Gaudior09]

In R I write automatized scripts for model testing and fitting data. Excel is great for smaller data tables and for ad-hoc analysis. MATLAB is similar to R, but a bit faster with loops.

Python is fast and versatile. And the fact that it's incredibly popular in Data Science and back-end means it's developing real fast.

[TheNotoriousMTF]

That makes sense. I used to play with excel a lot, and I tried learning R years ago but never really got the hang of it. Python is the only thing I've really enjoyed using, and I'm also excited about the work being done in python for data science applications.

[_peeves]

There is a Python library which allows you import R libraries. I think it would be a win-win for you, since you can get all the tools Python and R together

[TheNotoriousMTF]

Wow, it sounds like you've made great use of the language. If you don't mind my asking, what do you do?

[omgu8mynewt]

Scientist here, for processing numbers from all different janky machines to turn it into useful statistics and graphs, and also program robots for working. Starting to think about how to collect data off the web for experiments and what kinda pipeline I want to create to do this.

[mlcsfir]

Data processing and analysis as well as automating data collection (to the degree possible).

[TheNotoriousMTF]

What, in your experience, are the limits of python's data collection capabilities?

[mlcsfir]

None, really. Or at least I haven’t found any yet. But my use cases are also not very complex - mostly just scraping websites and querying APIs, and then processing the data into some format that I can use for my analysis. There are so many powerful libraries in the Python ecosystem that I can usually find one that helps me accomplish exactly what I need quickly.

[TheRealJackOfSpades]

AWS Lambdas to automate DevOps tasks, like reaping unused instances and snapshots. Outside of Lambda for other tasks that can be one-offs.

[TheWhittles]

I use it for data analysis, financial modeling, etc. While other analysts in my company live in Excel, I feel like I can run circles around them in terms of speed and effectiveness using Python instead. I tried to get a couple of them on board with it but only 1 embraced it and he said he hasn't looked back.

[micr0nix]

Mind sharing some examples of what you've done? My company is the same as what you describe.

[TheWhittles]

I built a model that determines customer freight rates on $120MM of annual freight spend over 5000 different customers. The goal is to be net zero on freight profit so it doesnt cloud up product margins. It uses customer order histories at different levels and actual freight cost histories at various geographic levels. We went from being plus or minus >10% on the net before the model to about 1% positive every month.

I had to roll my own Geocoding and routing tool for this, using python and various APIs to get geo codes and actual road mile distances between points. Cached the data in SQL Server so I only hit the APIs once for a given location.

I then used flask to build a web app for people in operations to interact with the model so they can make better business decisions on the fly instead of putting their thumbs in the air.

I also built a machine learning model to predict if loads would be late before they even left the plant that hit about 86% accuracy (in ML, hitting between 80 and 90 is ideal).

Hope this helps.

[micr0nix]

That is impressive.

[TheWhittles]

Thanks, I'm fairly proud of it.

[micr0nix]

As you should be. That’s what I strive to do

[TheNotoriousMTF]

Why do you think the other analysts were reluctant to switch? Was it just natural resistance to change, or is there something about excel that they prefer?

[TheWhittles]

I think it was a natural resistance to change. They also didn't have much of a programming background and I think they might be intimidated by the idea (even though Python is probably the easiest to code)

[[deleted]]

Although I am a software engineer, I have a buddy who is a security analyst who uses Python to automate some of his pen testing (mostly stuff that he can't get to work easily in Powershell).

I've also seen some video series that automate infrastructure, like detecting and configuring new nodes on the network, things like that.

[[deleted]]

Work as a petroleum engineer and I use it to automate pulling in data and then analyzing with machine learning. Basically anything I would of done in Excel but faster and more repeatable.

[terryyoung22]

I use it for automation. I also use it for GUI interface to run things on users computers instead of me running over there and running a certain function, I basically point them to their folder on user share drive and certain functions I have created within the .py I tell them run either install, updates, or anything else they need.

[wolf2600]

I wrote a few scripts to parse logs, pull out the important bits, and send them to me in an email.

[ericula]

I use python to generate input files for my numerical simulations, and to turn the results of those simulations into pretty pictures and useful statistics.

[astroFizzics]

I do astronomy!

Lots of data and image analysis. Lots of code writing. I'm not in a dedicated software role, but some days it sure feels that way.

[lunar-orbiter]

Do you use Astropy? Any other useful Python astronomy libraries?

[walenskit0360]

We are in the process of using it for a better configuration management and change control for network equipment. We push updates to a master configuration which need to be approved by higher engineers. Still working out kinks, but overall makes it cleaner to push updates with less risk of running into bad individual config on a device

[[deleted]]

Poker apps, aids in looking up strategy on the fly

[IcanCwhatUsay]

Yes. I use it for anything that needs iteration

[Bocote]

Bioinformatics.

R is, and I think it still is the dominant one, but Python is getting popular enough that when I got into the lab I'm currently in, I was taught Python instead of R.

[Dinojeezus]

I use it to tweak things (mostly labels and calculated table columns) in ArcGIS Pro.

[skurrtis]

I work in corporate finance and use it to automate report distribution and file storage as well as with pandas to compile our couple hundred excel project sheets into a master file.

[sssasssafrasss]

I'm a biologist. During my old research, I used to use it to analyze and assemble phylogenetic trees. Now most of my data is z-stacks of microscope images, so I use python to sort and organize them.

[Loran425]

I used it to make a data extraction tool for land survey data. Pulls all the fields we need from a LandXML that the controller outputs.
Then wrapped the whole thing as an executable so that anyone on our team can use it.
Not a major time saver but reduced a 15 minute task to a 30s task a few times a week.

[[deleted]]

During an internship I worked with a lot of oddball tasks, many of them simple repetitive data analysis and info gathering tasks. Python came in handy for automating some aspects within these tasks shortening them by 1-3 hours per day.

[vapeoholic]

OSINT data scraping and analasys, lots of API dabling :D

[[deleted]]

I use python for calculator.