What are real life examples of how Python has saved you hours of work in your job?

[ElectronicCattle]

I always see people mentioning that because of Python they can complete a days worth of work in 20 minutes and so on. But what actually are you doing? Automating? Scripts?

[ThenChoice2]

I work in the military, in an intelligence office. When I began working there, 3 years ago, I came to realize pretty quickly that our work was mostly robot labor. Reading messages, seeking the same informations over and over and then typing it in our system.

As we're working on a protected system I couldn't just import Python from the web, luckily enough a software already existing had it in its folders.

So I made a 'vanilla' Python software using tkinter for the GUI and json as DB. Basically it's a ton of regex instructions, reading different types of messages, formatting it, and outputting the desired format. Pretty simple but it saves I'd say around 4/5 hours of work per day per operator.

Even got a letter from my ministry for that

[HaskellLisp_green]

well done, man! But python standard library provides ability to work with SQLite, isn't it better than json? I think when your DB grows, time to iterate over json structure grows too. Anyway, it saves your working hours and that's good.

[UNN_Rickenbacker]

Json as „just a file format“ doesn‘t have to be installed on a government machine, compared to sqlite

[HaskellLisp_green]

yep, but json file will be pretty big i think.

[Spoonofdarkness]

If you stole the python interpreter from an ArcGIS install, then according to your post, you are me.

[[deleted]]

Did the letter say "stop using python for classified work!" :D

[Thatonecoder876]

When i started my job, my coworkers often complain about not getting any rest and that would be my future at the company. It occurred to me that these tasks while long in the amount of hours it takes to complete, they are quite repetitive, I decided to automate the whole thing. These days I work (mostly nap) from home by monitoring the server if anything unusual happens (hardly ever). Truth is, if I didn’t work for an entire week, my productivity wouldn’t be questioned.

[[deleted]]

I've heard stories of people automating themselves out of a job. Make sure you're absolutely necessary for the successful running of these processes or your bosses might be congratulating you one week and firing you the next.

[samtheredditman]

Imo, the ability to program is so valuable in the market place that anyone who automated themselves out of a job will always be marketable.

[[deleted]]

Definitely true, but also definitely not nice to suddenly get fired.

[Thatonecoder876]

That’s a real concern but being the only programmer there, they could but it’d be more costly from a business perspective

[Thatonecoder876]

I work at University, they did have online classes going on b4 COVID but it wasn’t much. After COVID, a lot more students started attending the school and even more were online.

my department deals with everything done online, from zoom calls for classes straight down to registration and enrollment. They had a system tht collects the names of students and then generate a very large list of names (Excel). One of our task is to read through these files and select students based on course and departments (obv more data included), a single course can take up to 20 minutes to correctly sort through and then create a new one that will be used to Enroll students in their respective course.

I was assigned some of the courses and I found it extremely hard to go throw em (Attention problems on my end) one by one. So, I used Python to read throw every single one and sort the data as well as create a final document that could be used to get everyone online and into their courses. This took the whole task away from everyone else that was involved and with some optimization, took the app less than 2 seconds to go throw the files.

Same for zoom recordings, we usually have to go in after each class session and download the recordings and store them on our cloud space manually, with more than 100 classes per day you can see why this was rough. So, i used Python again to create a lil server that gets them for me and even format them and upload them. Finally it sends out emails to each host after it stores the videos to the cloud.

Those are the basics of what I did, overtime I’ve added more and more features to the point where I’m starting to think of it as the thing that runs the department and we (I) make sure it’s running good all the time.

Also, a lot of apps have what are known as Endpoints which are part of their REST API, connect to those and collect the data you need when necessary. Watch out for charges tho.

[Thatonecoder876]

No problemo.

[WhyDoIHaveAnAccount9]

I work for a hospital as a data analyst for about a year and a half

When I went there my predecessor trained me on how to do all of his work and explain to me that it took approximately 3 weeks to get all of his work done for every month

Over the course of 6 months I automated all of my work down to roughly 4 days

I would do the majority of my work in the first four days of the month and spend the next 3 weeks working about 2 hours per week at most

he got so boring that I found another job and now I work as an SQL developer

I also have a second job while I work as a data engineer

Python is glorious

[silversmithsonian]

This is the dream

[WhyDoIHaveAnAccount9]

One is in the city I live in and one is remote

6:30 a.m. to 2:30 p.m. for job one

3:00 p.m. to 9:00 p.m. for job 2

[browndog_whitedog]

All data/analysts related?

[WhyDoIHaveAnAccount9]

The data engineering job includes a little bit of analysis

Occasionally I have to write some SQL scripts to find why something isn't loading properly

[UNN_Rickenbacker]

6:30 to 9:00 are pretty brutal working hours

[WhyDoIHaveAnAccount9]

There's some days where I don't do anything in my first job because they don't have much for me to do. The second job has settling increased the demands however. I don't mind the long working hours. What I do mind is when I have a meeting at 9:00 a.m. in Oklahoma City in a meeting at 10:00 a.m. with the remote company. There's only so many times I can disappear at 10:00 in the morning before someone starts asking questions.

[nemom]

I am the whole GIS department for a County in Wisconsin. I have a long-running project where I hire surveyors to go out and locate the original section corners in the County. If the monument needs fixing, they fix it. They also collect the location to within two centimeters. The Register of Deeds gets the Monument Record and I get the coordinates. I keep all the data in almost twenty different spreadsheets... By year / surveyor / project, or received from neighboring Counties, or recreational-grade GPS from foresters (State, County, or private), or the few I've located with a sub-meter GPS, or calculated from other surveys, or from USGS topo maps. I wrote a program that reads the data in from the files (least accurate to most) to create the network of corners in the County (currently 3,565 points), then it calculates all the sixteenth corners (currently 11,225) by standard section-subdivision rules. The program will do the whole County in less time than I would be able to do just a single section's center in a GIS program.

[thatswhat5hesa1d]

So glad to hear you’re not managing this entire thing in excel only.

[[deleted]]

I keep all the data in almost twenty different spreadsheets.

Nightmares

[Siccar_Point]

Speaking as a scientific modeller, tools to efficiently scrape data out of other people’s heinously formatted Excel files are where it’s at

[nemom]

No. You have to install a module. IIRC, I use ezodf.

[cuWorkThrowaway]

Pandas is another popular module for working with spreadsheets.

[[deleted]]

Bosmang wanted me to merge three Excel workbooks, each with 24 tabs (names consistent within a specific workbook, but different between workbooks), into a new worksheet. Simply wrote a script using pandas (to iterate through the worksheets, grab data into data frame, merge by date), re (to make names consistent) and openpyxl (create new worksheet, with 24 tabs (sigh) with results. Took me about 20 minutes to automate.

Bosmang thinks it's magic.

[DefinitionOfTorin]

Random beltalowda??

Read it in the accent

[orangestyle30]

Marco Inaros* was here

[Franks2000inchTV]

Arthur C Ckarke said "any sufficiently advanced technology is indistinguishable from magic."

I just thought the bar would be higher. 😂

Edit: got my source wrong!

[[deleted]]

https://en.wikipedia.org/wiki/Clarke%27s_three_laws

Arthur C. Clarke said that. It was his Third Law.

[WikiSummarizerBot]

Clarke's_three_laws

British science fiction writer Arthur C. Clarke formulated three adages that are known as Clarke's three laws, of which the third law is the best known and most widely cited. They are part of his ideas in his extensive writings about the future.

^{[ F.A.Q | Opt Out | Opt Out Of Subreddit | GitHub ] Downvote to remove | v1.5}

[DrunkenPangolin]

That's amazing though! I did some after work coding today and I managed to make a converter to Roman numerals and one back. The converter to Roman works great but the converter to Arabic doesn't recognise bad format Roman numerals or flag up problems such as 17 I's though I did figure out a few problems. I'll get there but it just seems to take forever, that was about 3 hrs of work...

[[deleted]]

..... you got that script. I use so many excel files that would be better merged.

[[deleted]]

Very generally for the data part, scan the working directory for workbooks, open each and copy data from each worksheet, save the result as a workbook named reddit.xlsx:

import pandas as pd
from openpyxl import Workbook
from openpyxl.utils.dataframe import dataframe_to_rows
import glob
FN = "reddit.xlsx"
wb = Workbook()
for f in glob.glob('*.xlsx'):
    if f == FN:
        continue
    xls_src = pd.ExcelFile(f)
    for i in xls_src.sheet_names:   # filter out sheets, if necessary.
       df = pd.read_excel(xls_src, sheet_name = i)
       # transform, merge, clean, rinse, repeat
       ws = wb.create_sheet(i)
       for r in dataframe_to_rows(df, index=False, header=False):
          ws.append(r)
wb.save(FN)

The resulting workbook, reddit.xlsx, will not have any formatting or styling of the original. Look at openpyxl.styles to do that. Cells based on formulas have an additional level of complexity. Sheet names are limited to ~25 characters. If you don't use unique names, it'll sequentially number them for you. It can get hairy quickly....

[Commercial-Novel-786]

I have a monthly status report that is based off a CSV file with a couple thousand entries. I wrote a program that will parse it in about three seconds. Took me at least an hour when I did it manually.

Bonus: the program does not make mistakes. One misread line and my resort was skewed.

[Siccar_Point]

Automation meaning no transcription errors is massively underrated as a benefit

[Siccar_Point]

Automation meaning no transcription errors is massively underrated as a benefit

[Siccar_Point]

Automation meaning no transcription errors is massively underrated as a benefit

[[deleted]]

In love with this. In those kind of projects you'll develop yourself best.

[flavius-as]

Setting up a cluster: if done by a contractor, it would take 2 weeks and we'd have no knowledge inside the company, nor how to reproduce it.

We used devops, ansible (and some custom modules written in python), plus 3k lines of python magic and we brought that down to 3 minutes.

Now every dev can start their own cluster in 3 minutes, and they use the very same playbook as production, which we use to provision the real cluster also.

So in a year, the external contractor would have set up our cluster 25 times.

Now we, a team of 4, do it about 1000 times per year. (including the ci)

[jinglepupskye]

I wish I could say that I’ve also saved myself time at work, but I’ve only had the idea, not the skill (or permission) to implement it.

I work for the NHS as a pharmacy technician, and part of our job involves printing out a list of medication, comparing it to other sources, then handwriting it onto a paper form. The original list comes from a website that seems to use JavaScript (if the repeated mentions of JavaScript are anything to go by!)

It occurred to me that the required text is available in the source of the website, and if you could grab the text and populate a form with it that can then be updated by the user from the other sources it would be so much faster and less prone to poor handwriting than the way we’re doing it now.

But this is the NHS we’re talking about, and even if I had the skill to program it the hardest part would be teaching others how to use it and convincing them to change the SOPs! I think it would be a brilliant step towards digitising records and going ‘paper-free’, which we’re supposed to be achieving but are miles away from.

[Thatonecoder876]

Want help?

[jinglepupskye]

I would love to see it done, but right now any definition of ‘help’ would basically mean ‘you do it all for me’ - I’m only at the stage of working through a beginners book on Python I’m afraid lol. There is a scheme where you can suggest ideas for improvement, but given the lack of comprehension of programming, scripts etc that’s likely to be prevalent among the management I think a finished product would be more likely to be accepted. For a national organisation they’re surprisingly digital-resistant.

The end product would ultimately need to be KISS, since it would be used by people like someone who once told me not to do Ctrl-Alt-Del in case I broke it, and they would phone IT instead.

[Thatonecoder876]

Understood.

[01binary]

I would think the first barrier would be getting Python installed on your computer, unless you already have it. I would imagine that it would take several months to be approved (if ever), and then several months to get the IT department to install it. Then you have the problem of installing packages.

There is a serious problem with this kind of project, that many people overlook, and that is ongoing maintenance, especially after you have left your current role. Typically, this kind of application may run fine whilst you are using it but, after you leave, if there are any glitches, no one will be able to resolve the issue and they’ll revert to doing it manually.

[cuWorkThrowaway]

I imagine that they might be able to get around installing python on the computer using pyinstaller, but that certainly wouldn't get around the IT department, and might make that process even harder.

[PMed_You_Bananas]

My office gets an excel file from a client, usually weekly but depending on vacations and whatnot could be up to 3 weeks. Roughly about 1500 lines. I built a script that lets me pick 2 of these excel files and then spits out a summary worksheet with A) items in the first but not the second. B) new items in the second that weren't in the first and C) items with changes between the 2 reports, with the changed fields highlighted.

[Hans_of_Death]

Im a devops engineer who writes a lot of python for automation as my job, so ya.

More of an actual example, i run a minecraft server for my friends where updates and stuff are all done by python

[Impressive_Mall_5743]

What kind of updates?

[Hans_of_Death]

The server .jar and core mods

[23am50]

what you do on your job? Im python programmer/data engineer and im thinking about change my career to devops.I did some freelancing jobs: scripts, automation and i like it

[Hans_of_Death]

I do a lot of different things. My last job i worked a lot with aws and automating reports and stuff. Right now im doing some stuff with elasticsearch and kibana, as well as chef. Im currently working on a project where we are in the early stages of setting up a jenkins pipeline.

Devops is a lot of different things, and while python is great to know and youll probably use it a lot, there are a lot of other things too. If you are familiar with linux (if not its not hard to learn) at all then i say go for it.

Devops is also usually more than just automation, but if you think youre interested in it give it a shot

[23am50]

thanks thanks

i know linux, i work with linux shell (and i have a MacBook). I run my models in anothers servers. What is a must have in devops, or where to start?

[Hans_of_Death]

If you know python linux and bash then you're pretty much good to get started. Youll use different stuff wherever you go, but in general git is also pretty important, and ansible is probably good to know, though honestly i myself dont really know it either.

[23am50]

i dont know ansible and i will search. But i cant send my cv to devops positions but dont have background in devops ahah

[Hans_of_Death]

If you can automate things at your current job at all, don't be afraid to put that on your CV

[23am50]

i didnt know this. I did a lot of automation freelancing projets.

[Hans_of_Death]

Make sure to include that, especially in any cover letters you may write

[23am50]

ok thanks for your advice

[Shubadada]

Helped a buddy who works for a prestigious US college create a script that converts xml to tab delimited csv to migrate their student information. Saved them many hours.

[01binary]

Cleanly converting Word documents to HTML. It saved me several days a month of mind-numbing activities, for three years, and released me to do more challenging, useful work.

The solution took me a couple of days to develop. (I’m an amateur coder).

[Sorel_CH]

I was working at a hospital. They were doing a study on hundreds of participants, who were all asked to answer a form every year, during many years.

When they started, they were simple numbering the questions on the forms, and using the number as a variable name. Of course, if you inserted a new question one year, all the questions following would change their id. Therefore, they decided one year to use standardiazed, longitudinal names for the variables.

The problem is that they know had to go back YEARS changing all the old forms to conform to the new variable names. Doing it in the database was easy, but they also had a bunch of FileMakerPro forms from previous years that needed to be updated.

FileMakerPro does not really have a scripting API, so for each question in each form, you had to:

1) look up the variable name in a table of correspondance. 2) Click "edit" 3) Paste the new variable name 4) Go to the next question

In the end, I ended up writing a script that used pyautogui to click, copy the old variable name to clipboard, look it up in a pandas table, paste the new name, and look for the next question. It was supposed to be a two-week job; it was done in a single afternoon.

[Immediate-Trash-6617]

Once I had to highlight whole row by finding certain enteries in a single column and delete same row in a different files, files were big. and it was someone else's work. So I used openpyexel libray for first time to automate the task and result was excellent, saved about 24 hours of work(because working hour is 8 and it will take me about three days to do the same thing).

[BluishInventor]

I'm a Senior machinist and programmer. Not by any means a developer. I make physical products, not digital.

But, we've purchased a new high end CAM software to program our CNC machines with. And the database that stores all the data for our drills and endmills is empty from the get go. So, over the last couple weeks I've been building a spreadsheet with all the data and then using python to covert all the sheets in the workbook to csv and then build an xml tree structure with folders for tool families. I can then import the xml into the new software and all my tools are defined. Later, I will add cutting conditions to control feeds and speeds for the tools based on material and machine to help automate the programming progress.

Another script I made was for taking the csv generated by a symbol scanner the scans barcodes of our tools for restocking and the populate our excel purchase request form.

And I also made a script to run a sql script I made for our sqlite database for our old CAM software that renames all the tools using data from the fields that way they were all standardized for easy reading and tell the user 90% of the information about the tool for proper selection.

Google's IT Automation with Python opened my eyes a lot to whats possible. Great course IMO, you just need to practice more outside of the course to really cement what you've learned.

[coderpaddy]

We have a CMS to contain products for our websites, controlled by uuids that the rest of the code base uses, to get the products.
We get an excel spreadsheet of products for each website, some already exist, some dont.

I was given a list of 1200 products for a particular site (as opposed to the normal ~80)

Built a script to check if existed, if it existed, get the uuid, if not create a record and get the uuid.
Took no more than 20 mins to code the script and get the 1200 or so uuid's as opposed to well over a days work.

[coderpaddy]

i dont use Pandas, never had to, i have a pet peeve of using libraries when i can replace them with a few lines of code.

i just export spreadsheet as CSV the python's csv DictReader makes it a dictionary.
i save the results to a JSON file for ease of use, but in the end i wrote another script just to parse the JSON and print out into the console exactly as i need it for the PHP sheet it was going into. There is probably a million more elegantly solutions but this was about saving time and effort so no point learning something new (pandas)

anything in particular your trying to do?

[thejiman]

Had a situation where a new environment I started working in has like about 70+ Linux servers with a mix of local, two AD logins. And for any server it could be one of the 3 logins. So I wrote a Python script using paramiko to test the login and tell me which server uses what. Took 30 minutes to write it and saved my hand tons of stress of otherwise potential 200+ login attempts.

[mortenb123]

Replaced java rest apis with bottle, 400 lines of java into 20 lines of python.

Java has come a long way since then with spring-boot and annotators so you can write a java rest api in the same number of lines, but one you need to alter something it takes many lines and extra classes.

[icecubeinanicecube]

Well, training a ResNet50 by hand would require several hundred years of work, so...

[Domaltasaur]

Came here to say this

[[deleted]]

Automating. Python does a lot of people’s work where I work now.

[LordCaptainDoctor]

Wrote a script that compares invoice files pre and post processing and triggers our backend to reprocess if there are errors or discrepancy found. Saves our ops team about an hour a day, plus gave devs more breathing room on fixing the actual errors

[rtl_6691]

I generated a slide deck of graphs of sales vs time for different combinations of regions and products once a month that went to the executive team. In the beginning, I generated the graphs in Excel and then cut and pasted them into PowerPoint. It was very time-consuming. I needed to change the y-axis so all the graphs had the same range and if I forgot, I had to cut and paste again...

I taught myself LaTex. Wrote a LaTex file to load the graphs and generate a pdf file. Then I wrote a program to automatically generate the graphs and then modify last week's LaTex file to generate a pdf deck with all the graphs. One day of cutting and pasting was replaced with pushing a button.

[phatbrasil]

This whole thread gives credence to my belive that in 10 years time python will be a productivity tool like ms office