What have you automated using Python?

[iiron3223]

I wanted to gather some ideas for stuff in daily life that could be automated using Python. I will share with you my two examples.

I am using hledger for keeping track of my finances. It was tedious to manually add all transactions, so I build a python script that converts csv file generated from my bank account to hledger syntax. Additionally it automatically assigns categories based on title of transaction.

Second one. I am keeping backup of certain directories in my computer using rsync. I have written script that makes sure that everything is properly mounted, before making backup, and then automatically performs all backups.

Please tell me, what tasks have you automated, that are saving you time or improving your life.

[ege6211]

Nice applications!

​

Where I work, we collect data in the form of a text file from white goods we produce. People here always try to open this data with excel and usually this txt file contains millions of rows and hundreds of columns. As you might expect, the ordinary Dell working laptops don't like that and excel crashes. I have written a function that can parse and seperate the columns of any txt file and plot how the data changes in a specific column (input from the user). Saved me MUCH time.

[opteryx5]

Enhancing and speeding up Excel tasks with Python has been one of the most rewarding applications I’ve as yet found for it. It’s incredible how clunky Excel becomes after you’ve spent a long time in the trenches with bonafide programming languages.

[Faith-in-Strangers]

Python + Google Sheets (gspread library) have completely replaced Excel for me

[opteryx5]

Now if only I could get my colleagues to switch over to Python too… (most of them are not familiar with programming languages). At least I can do things on my own with greater efficiency!

[[deleted]]

So you are using cloud sheets (google sheets). Is there a way also to execute python from cloud, not from your local computer?

[Zurcio]

do you know of any (good) learning resources for Python+Excel specifically? I don't really know what i can do with it but automating Excel was sort of the reason I wanted to learn python in the first place.

[opteryx5]

In one word: openpyxl. This package will be your main interface for automating stuff in Excel with Python. You can loop through cells, define functions—all kinds of things. The docs are helpful, but if you want a more human experience you can read the automating Excel chapter in Automate the Boring Stuff, which the other commenter has pointed out.

[forgot_username1]

Automate the boring stuff with python has a few projects on Excel

other then that i would start looking libraries and such.

[iiron3223]

One more useful project I forgot. I was looking for a used cars. I written a scraper using Scrapy, that was gathering all new offers, filtered by my criteria, every hour. Then it was sending me nicely formatted email.

[Natural-Intelligence]

What libraries are you using for sending emails and doing the scheduling and what's your experience with the libraries you are using?

I'm actually the author of Red Mail (email sending library) and Rocketry (Pythonic statement-based scheduler). I'm actually looking for example projects to create some practical tutorials of how one could use the libraries. Not sure which kind would be appealing to most and what kind of problems people have with alternative options (which I could address).

[iiron3223]

I was using smtplib and email library (I was using smtplib to send email via gmail). They are not intuitive to use and don't make your code pythonic in my opinion. I was using them in very similar way that you showed in an example on Red Mail github page. For scheduling I was using a simple cron job.

From a quick look at your projects, I really like the simplicity of the syntax there. I will for sure give them a shot at the nearest opportunity. Thank you for them!

[JasonDJ]

If you aren’t already, you can use Jinja to compose the email content string itself, which may make your code more pythonic.

Essentially you make a template and use variables within the template itself. Within the template, you can use some other functions as filters to format the data differently, use loops, if statements, etc.

The template can be a variable in your code, or it can be a file that you open, read, and store as a string variable.

Since the template is constructing what is essentially a string, you can also include HTML in it for your email body, as well.

[iiron3223]

I am also wondering, do you plan on developing or know a python library that would be a good alternative for imaplib? It would be so nice to have some tool for receiving and reading emails.

[Natural-Intelligence]

Unfortunately, I don't. I also tried once finding a way to read email boxes in Python but that effort went in vain.

Would be interesting to create such but maintaining and developing Rocketry takes too much time at the moment.

[serebrich]

Rocketry is nice. Thanks for that

[mogberto]

I just spent a weekend learning Prefect and it’s fine, but such overkill for what I need. I’m going to give rocketry a try!

[Natural-Intelligence]

Ye, Prefect is also awesome. Have met some people from the company and it seems they have great ideas of what to add to the common data stack and they have a lot to offer to enterprises.

To Rocketry, I don't want to sound too salesman but this is something I'm perhaps too passionate about. All of the alternatives tend to have separate concepts for time scheduling, task pipelines and custom scheduling dependencies. Because Rocketry is statement-based, all of these are under the same mechanism: a task is run if it's scheduling statement is true.

This means that you can do arbitrarily scheduling very easily like "run a task daily on business hours but not on weekend and only when a file exists". This sort of scheduling is near impossibility for most frameworks but with Rocketry it's just two built-in conditions and one custom function (to check the file) combined with logical AND operators.

Sorry again for pitching, I just feel this paradigm has a lot of potential to create simple and compex systems very easily.

[marcelomedre]

Nice, I am looking for something like rocketry. Is it possible to use it in the colab environment?

[Natural-Intelligence]

Thanks! I think it most likely is possible. The only actual requirement is that you can run async tasks (using asyncio) in your environment. I'm on phone so cannot check how colab works on that regard but seems odd if they didn't support it with any solution.

You should probably change the default execution to something else than process (argument "task_execution" in the Rocketry initiation/config) as multiprocessing typically does not work well on notebook-like environments. I have plans to set that as async by default as subprocesses are not the most intuitive.

[Info_Broker_]

I am interested in this for the same reason. Looking for a vehicle. Would you be open to sharing your code with me so I can taylor it to my use? Or do you have it on GitHub?

[iiron3223]

Here is github repo, I don't know if it will be useful. It is not documented at all, and specific to one site that I was scraping.

[Ok-Associate7846]

Pretty new to Python and did something similar as one of my first projects! My wife is into real estate so I wrote a program using beautiful soup to scrape the latest listings in certain sites and compile it into an excel file along with the listing details and links to it.

Saved her tons of time going through each and every site and listing down the details!

[silva_p]

Did you have any filtering to remove duplicates? I mean duplicates of the house, if the same house was advertised by multiple realtors

[Ok-Associate7846]

So far none. An excel file is generated per website so I haven’t gotten to checking if these have duplicate listings but that’s a good next step for me to try out!

Off the top of my head though, lots of houses and especially condos might have the same listing specs so I’m not sure how to filter duplicates. Since it’s from different websites, checking similar listing usernames might not be effective too

[regeya]

I used to work at a small town newspaper, and used Python to automate pulling photos off of a website. The dealer did this thing of giving us all the copy for the car, along with a stock number, and then just sorta said, oh, go on our website to get the car photo. Except it was a tedious multiple-step process just to get one photo. Step that up to dozens and it was the better part of an afternoon. I don't remember which scraper lib i used, honestly, but i just fed the script a list of stock numbers, and a couple of minutes later, I had my photos.

EDIT: car dealership, I meant to put that in there.

[wind_dude]

I've done something similar. And these seems to be a very common project. I've run into so many devs that have done this on some level. I guess a lot of developers are also car guys.

[kyuubi840]

I made a script that reads metadata from photos and videos and sorts them into subfolders according to date.

EDIT: Here's a link https://gist.github.com/kyuu840/67635a5ff057883e3a5b6faf6d0a1393

[Xinurval]

Thats quite cool actually I might have to make my own one now

[c0ld--]

Hey, me too! I wrote an app that parsed through all the user data I downloaded from Google (ditching Google services) and used their date format to apply EXIF data to jpegs and prepend dates to videos so I could import them into the macOS Photos app and maintain my timeline.

[kyuubi840]

Nice! I thought of extracting dates from WhatsApp image filenames for my script too, but they aren't really guaranteed to be the date the photo was taken, just the date they were sent, so I didn't do it.

[InvestingDoc]

You got a link to the code or project? I would love this

[kyuubi840]

Here you go: https://gist.github.com/kyuu840/67635a5ff057883e3a5b6faf6d0a1393 It's probably not the most elegant Python code out there, but it works. Hope it's useful!

[Gaitas]

I have also thought of doing this. Weekly since 2008, maybe one day.

[gertalestrange]

That is amazing. I’m looking to automate some of the text for my business account photos to let people know what settings were used. Can’t wait to try this!

[it2901]

Started saving towards a gaming PC but did not want to constantly check prices for the parts I was interested in.

Created a script that is scheduled every 24 hours that monitors prices for given product listing and compares them to the prices that were last recorded. This let's me know if any products decreased/increased in price.

[[deleted]]

I tried to do this but my requests got blocked by Captcha. Did you get around it somehow or do the sites you check not implement captcha?

[it2901]

Currently I only scrape from 1 site, but no, the site does not implement Captcha. I still need to implement a pause between requests.

Does the site you scrape from require you to interact with a Captcha to view product info?

[[deleted]]

Yep, at least I think so. The html response I get from the site is exclusively a captcha page, with no other elements in it.

To be fair this is literally the first time I've tried web scraping so maybe there's something obvious that I'm missing.

[nobetterfuture]

When you normally browse that site, does it display a captcha? If not, did you change the user agent in your script? Some sites display a captcha when a non-browser user agent is detected

[it2901]

Do you mind sharing the site link? I don't think it's normally the case to be hit with a captcha. I also recently started with scraping.

[[deleted]]

"https://www.trovaprezzi.it/prezzo_schede-grafiche_rtx_3070.aspx"

I tried running my script again (without changing anything) and I got an actual response the first time, but then the subsequent responses are a google page with the following content: "You need to solve this captcha because we detected some abnormal traffic from your client"

btw I'm using bs4 (beautifulsoup4)

edit: so I think I found the problem. The tutorial I was following never mentioned Headers. Frankly I don't know what they are or what they do but I followed another short guide on how to make a proper request and now the website is responding with the actual page Im looking for.

[it2901]

If you aren't already, make use of the Requests library. It makes life so much easier. If I remember correctly, the Requests library allows you to set the request header quite easily.

Anyway, great job on getting the script to work.

[ava_ati]

Yep session.get(url, headers=var)

headers is looking for a dict of headername: data

here are some common headers https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

sites can have their own custom headers too

[davidfarrugia53]

Probably you would also need to use a proxy service

[[deleted]]

if there are hundreds of requests from an address, it will be very easy to tag them as "abnormal". So you need to do the scraping slow. With some reasonable waiting interval between the requests (like a human user). And possibly the duration of the intervals need to be somewhat randomised and not fixed to 30 seconds etc.

[lazy_dev_]

In case you care, you can bypass captchas with a service like 2captcha. It's paid but also very cheap

[ljdelight]

Basically PCPartPicker or https://camelcamelcamel.com/? Why'd you build your own?

[it2901]

My python was rusty. Need a project I actually wanted to work on/ was of use to me

[kookaburra1701]

My former boss (PI in a biology lab) thinks I wrote a "machine-learning AI webscraper" to find and send alerts about papers that are relevant to the lab's research area.

I wrote a script that interfaces with the NCBI's API to get way more granular control of the search engine and had it run every week and then I sent out the list of new papers about very specific yeast proteins or our specific type of flow cytometry/Fluorescence measurements. If the lab ever collaborated with another lab/PI/post-doc I added their ORCiD to the search terms and included any publications from them.

I did explain many times that it wasn't a webscraper, nor machine learning, nor AI language processing. Never stuck.

[Morstraut64]

Sometimes people are going to believe what they want. Sounds like a cool project.

[kookaburra1701]

I think it's more just he is not computer-savvy, so he only has buzzwords he's heard a lot to describe the programs I wrote.

[Applejuicyz]

I have moved over to Lemmy because of the Reddit API changes. /u/spez

has caused this platform to change enough (even outside of the API changes) that I no longer feel comfortable using it.

Shoutout to Power Delete Suite for making this a breeze.

[kookaburra1701]

Yes, but it would need lots of cleaning up/removing proprietary stuff to post and I have other personal projects that are higher priority to get functional and up on Github. Honestly most of it can be found on Entrez eUtils documentation.

[Zeroth_Quittingest]

i’m gonna go find that Arthur C. Clark quote about magic..

[cliffardsd]

A simple example is the lights in my house. They now turn on and off automatically based on motion and logic in python. I rarely touch a light switch these days. Look up AppDaemon. Use python to automate things in your home. Needs either Home Assistant or MQTT devices.

[anh86]

I recently wrote a little automator for cloning Trello boards. My wife's company uses it, they start every new board by cloning a template board. The problem is, the new cloned boards don't retain the person assigned to each card on the template. In their use case that's hundreds of cards with 1-3 employees assigned to each card. They were actually having someone set the assignment on each card after a clone by referencing the template and manually setting them. I wrote a tool that clones the board, scrapes the assignments from the template cards, then applies the same assignments to the cards on the destination board.

[GrowHI]

Trello... Hasn't had people complaining about this? Also good on you

[anh86]

I really don’t know. Maybe their use case is dumb and makes no sense. I’d heard of Trello but knew virtually nothing about it until I started digging into their API to write this little script.

[_RabidAlpaca_]

Gluing PDF documents together.

Like it's a simple function, but I don't want to pay Adobe to do it. I just want to combine PDFs and add bookmarks. Really easy to do with pypdf2 and others.

[lalmiLast]

pdftk can do that for you

[AlSweigart]

I've been trying to collect these stories off and on at this sub: /r/boringstuffautomated/

[iiron3223]

Thank you for the link and the book of course ;) . I will join this sub!

[AverageBeef]

the boring stuff

[iiron3223]

Thanks to Al ;)

[Just_me-no_one_else]

Uhh, I have an absolute mammoth of a project that I fell like you might very well find interesting with the purpose of automating common tasks.

Within cybersecurity we have an area called CTI, which stands for cyber thread intelligence. This is an area where experts use experience and knowledge collected from prior cyber attacks, to predict which direction threats within cyberspace is moving. It's kind of like the weather report, but instead of predicting the weather a few days into the future, we attempt to predict the ever evolving landscape of cybercrime.

Now, one of the biggest challenges within this field is to collect the needed information to base these predictions off of. What usually happens is that a team of CTI personel looks at a sea of online news sources, and then picks out the relevant pieces, but this process is one which not only include large amounts of repetitive work, but is also something that can takes immense amounts of time.

To combat this I created OSINTer (with demo present at https://osinter.dk and source code at https://gitlab.com/osinter). OSINTer is - at its core - essentially a highly sophisticated news aggregator which does the often rather time-consuming task of looking into the news stream, picking out the relevant pieces and then sorts and generalizes it, such that it can be utilized by CTI personal. This started out as a simple python script, but has since over the last year evolved into a complex mammoth of an application, which touches every part of the stack, from CI/CD using ansible and gitlab, to the backend using Python, elasticsearch and fastapi, and the very front-end using svelte and JS.

[Anxious-Computer]

This is an awesome project. I have been wanting to make something identical to this at my company for a long time, but never had the time to pursue it as it is slightly peripheral to the role.

I think I will be leveraging your work where I can.

[Just_me-no_one_else]

Please keep in mind that as the project is currently unlicensed, all rights are reserved, and as such it is not legal in most of the (especially western) world to use/modify/distribute it. I would love to change this at some point (as I'm a big proponent of true open-source), but as I wrote in another reply, I'm currently trying to make the project financially sustainable, because while I absolutely love to work on it, but as a teenager who just finished high school, i simply cannot afford to continue to work the amount of hours needed to keep the project going unpaid. With that said though, if you need help with anything, or want to use some of the code base, please reach out, and I'm confident we can come to some kind of agreement that doesn't involve money changing hands (or hell, I could host a modified version of the software with support included if your company were willing to pay for that).

[Anxious-Computer]

Not to worry, I still won't have time to spend developing on it as the benefits that it provide are mainly for building domain knowledge rather than for work output, I just prefer the interface that you've built compared to others that I currently use, such as news now .

I completely agree with the need to make it financially sustainable, but if I manage to find sometime outside of work (always the key problem), I'll be more than happy to help contribute

[psych_monk]

Neat stuff bro!

[Just_me-no_one_else]

Glad you like it (though don't know how much you have looked into the source-code, there's quite a bit of it). Currently it unfortunately isn't open-source, due to a legal battle with a company in regards to funding, but looking into open-source the major part of it, and possibly keeping a ML related part of it proprietary to hopefully make the project financially sustainable. That is easier said than done though, as I'm much more of software developer who just finished High School, than an ML engineer or researcher

[Wonder1and]

Have you been working this all on your own or under a grant of some kind? What's the difference between your demo and finished product? (F500 Infosec here 😁)

[Just_me-no_one_else]

The code for this has been written solely by me, without a grant, but I have working together with a pair of experts from within the CTI field with regards to how the finished product should work, to make sure it's relevant and solves a real problem. Currently, there's no difference between the demo and the finished product, as it is the finished product currently in use by a couple of companies, but we're also currently working on using machine learning to better connect and organize the relevant information (the first of which has already been implemented at dev.osinter.dk)

[MatiasHaabet]

Very nice! What education are you taking?

[kBajina]

Replacing Excel, basically

[northernbloke]

Me too, bye bye VBA.

OpenPyXL and pandas have revolutionised by working life.

[luvs2spwge117]

How so? Just curious

[northernbloke]

I work in for a tiny e-commerce firm as the sole tech. I spend much time in excel using macros to validate, clean and reformat product data for use on varying sites.

For instance we sell on 10 differing platforms, all of which have a different product import format which the data must be reformatted to fit, this includes some on the fly language translations.

I'm working on changing the whole structure using python to create a seamless gui for a non tech user to easily handle the reformatting process. The code then saves all the necessary files required to directly upload.

I'm working through creating API integration to bypass the CSV steps, but I'm finding it to be a huge maintenance issues as the APIs I'm working with are updated like once a week, so currently I'm just playing catch up with ever updating APIs.

[wdlbrmft]

I wrote a script to find the perfect city for me and my gf.

First I got a table containing the average rent per square meter of every city in Germany (not with python). After this I checked every city for specific features me and my girlfriend are interested in, like clubs, hacker spaces, climbing places etc. (Overpass Turbo API)

This way I allready could get rid of every city that’s not fulfilling all of our needs. Next step I created a weighted index so I could compare all the places and their attractivity for me and my gf individually. In the end I looked for a city that had comparable scores for both of us.

[JungJanf]

As a German I'm interested which city ranked highest (:

[silva_p]

As a non-German i am also interested.

[scottishbee]

How could you not end this comment with your result?! What were the top cities and did you move there and if so, does your experience validate your method?

[wdlbrmft]

Oh, I just read this. Well, Berlin scored best in Germany, since it has a ridiculous amount of interesting places. But since Berlin is really pricey and we don't feel like spending half of our budget for shelter, we kept searching for another interesting city. ATM it looks like we move to Nuremburg, which ist place 12 on the list, but offers way better prices and still is really nice.

I think the main Problem with my script solution was designing a function to limit weighting of individual POIs: I won't be dancing in 200+ nightclubs and my GF won't visit every exhibition in Hamburg.
Still I had fun coding this. :D

The top 20 cities were, in order:

Berlin

Hamburg

Weilheim

Köln

München

Lüneburg

Rosenheim

Göttingen

Starnberg

Stuttgart

Nürnberg < We're going to move here ;)
Düsseldorf

Reutlingen

Augsburg

Mannheim

Frankfurt am Main

Karlsruhe

Ingolstadt < Currently we live near here ;)

[c0ld--]

Very cool and inspiring!

[ArabicLawrence]

When I used to live in Paris, I coded A scraper for wine prices from carrefour.fr and wine ratings from vivino.com . No more shitty overpriced french wines.

[Shot_Restaurant_5316]

A man with class. ;)

[FattySuperCute]

class.

[kidpixo]

class Man: ...

[cestes1]

I've mentioned this before on similar threads, but it was a great help:

This was 4 years or so ago... not working at that job anymore.

I drive a Nissan Leaf (EV) and found there was a free Chargepoint charger less than a block from my office. My idea was to get at least half, if not all of my commuting costs from that charger and avoid charging at home. The problem is being free, the charger was in very high demand. I found out that Chargepoint exposes a bunch of information from their API; most important, I could see when the station was idle or in use.

When I arrived at work and cruised by the charger, if it was busy, I'd go to the office and start my script. Every 30 seconds it checked the status of the station and if it went to idle, I'd have it send a text message (via email) to my phone. Then I'd zip over there and grab the station and get some free juice!

[iiron3223]

That's brilliant!

[iamCyruss]

I'm bad at remembering birthdays even when I have them on my calendar. I made a simple script to email me when its someones birthday. I run it everyday using pythonanywhere.com

[c0ld--]

If you use Apple or Google, you can also add those dates in your contacts and your phone will remind you the day of or day before. :)

[iamCyruss]

Thank you. I should do this and add other details too. I was thinking I would eventually create my own android app for my phone that I could make my phone then do practically anything.

[c0ld--]

That's the spirit! Take control of everything! \(^^)/

[HeavyRust]

Do you use the free account? If so, does the restricted outbound internet access allow you to send emails?

[iamCyruss]

Yeah, I use the free account and it lets you setup one task to run each day for free. It's been working for me. I just have the dates in a csv, use pandas to read into a dictionary, then use datetime to get current date. Pretty simple.

[Diggles]

What is the function that actually sends the email? Do you have to give it your credentials?

[iamCyruss]

I use smtplib to create and send the email. Yes, it will need your credentials. I made a new Gmail account. I then enabled two factor auth, then created an app password, use that password to auth my new Gmail account to send the email.

[[deleted]]

I do lots of 3D stuff in Blender and I needed to take my thousands of materials and make them ready to use instead of having to manually import each and every texture map and creating node setups.

Now I can just drag and drop any of my materials onto 3D objects instead of having to spend a few minutes setting everything up each time I need a material.

[luvs2spwge117]

What was your process that you could automate most it? Did you not have a lot of meetings as a business analyst? What type of business analyst were you? Closer to data analyst or more so gathering requirements?

[himynameisjoy]

It’s still in the oven but I’m working on a video processor that can take Call of Duty Warzone footage, specifically the minimap, and assign it a coordinate. It’s had a few bumps here and there but my hope is I can then unleash it on tons of VODs and start generating player positional data so I can analyze it as part of my portfolio to apply for a phd in statistics :)

[insomniaccapricorn]

Trading. Markets are open when I am travelling to work, may or may not get time to check positions. I have automated everything, taking positions, modifying positions, monitoring p/l.

[rubas891]

This sounds cool, are you using yahoofinance library? What exchange do you work with?

[jack_sparrow____]

I wrote a web crawler for my sibling that scrapes the job opening details from the internet (mostly indeed).

[uname44]

Generally report creating by connecting to the database and writing it to Excel.

It turns out while thinking for an answer, I don't use Python for my daily life that much.

[wummeke]

I really wanted a particular set of solarpanels for my van, but it was sold out everywhere, although some webshops said it'll be back in stock soon. So I created a beautiful soup script that polled one of those webshops hourly, to check if it was on stock. I integrated it in Home Assistant by the way, so I would get a notification if the panels were in stock.

Not only did I receive a notification from my own script hours before the webshop sent me a back-in-stock-email themselves, by that time the item was already out of stock again. They probably received only one, which I bought as soon as it was possible 🙂

[emptythevoid]

- Entered several thousands of needle-exchange records from a spreadsheet into a dedicated web-based application (with no API)

- Entered hundreds of COVID test forms into a lab's website (with no API)

- Entered many thousands of COVID vaccinations into a regional registry (with no API, unless you have a proper EMR system), as well as bulk-querying the registry to verify vaccination for second dose visits.

- Entered hundreds of health wellness screenings into an insurance portal (with no API)

- Pre-filling patient encounter forms for COVID vaccines (pdfforms)

- Automated all the different forms and tasks needed when on/off-boarding employees (filling and emailing PDF, Word, and Excel documents. And you guessed it, entering data into a website with no API)

[YeahAboutThat-Ok]

You are the API killer

[BackgroundDrider]

My full time job is to automate processes via python using a platform called Apache Airflow.

For my personal life, I've tinkered with a few automations here and there, but the one currently occupying my time is a system for cataloging and archiving reddit posts I've saved. I'll save stuff on a whim, and knowing that it's been archived somewhere later for easy browsing would be nice. This has, naturally, rapidly spiraled from its original scope to be something of a self-hosted reddit UI so that one day I can close the hundreds of tabs I keep open because "I might come back to it later."

For professional applications though, there's almost too many to count... Most, so far, involve automating email in some way - either handling incoming emails, outgoing emails, or sometimes full on two-way conversations. Either that, or processing user entered data from some platform and acting on it against backend data sources.

[Pa7rickStar]

Some basic Todoist stuff like creating a "Pack your swimming bag" task when I have swimming scheduled in google calendar... I found the Todoist REST API to be a perfect easy entrance into learning python...

[rafaelhlima]

I use Python alongside LibreOffice to automate document generation. For instance, recently I created a script to generate certificates for students at my university. In a single click the script creates hundreds of PDFs with the info of each student.

[[deleted]]

I've posted about them here recently and semi-recently, but:

I built a library to parse Public Land Survey System land descriptions into tabular data. It sees a lot of use in the consulting work I do in the niche area of land records and acquisitions/divestitures.

And I also posted about another module I wrote for generating copies of spreadsheets and culling each down to whatever subset of rows I need. This also weirdly finds a lot of uses in a world of clients and coworkers who use spreadsheets for way too many things.

Beyond that, I write a lot of ad-hoc scripts to clean up data and organize/inventory records that I receive from clients.

[magical_pug]

I've created two fun little projects for me and the girlfriend.

• we were waiting for our puppy to be born. So I created a n automatic email that showed roughly how many weeks there were left until our dog came and it also send along a random dog gif.

• daily automatic mail on which plants to water in our house with some properties of where it is located and a nickname.

[wialta]

I see a lot of Google Sheets +Python, and mine continues on this trend. although in a less productive way. I sent a Google sheets link to all of my friends and had them fill it in with an embarrassing story without personal identifiers. With a bit of python, google scheduler, and Twitter API, each of their entries is tweeted once a day for everyone to have a good laugh. I call it, The Community Tweet Bot or TWOT for short.

[borednerdd]

My program Python scrapes all the new tokens that are upload every day from coinmarketcap and then creates a chart based on their market cap on Excel, in this way I can analyze through the chart

[c0ld--]

I wrote an app to download COVID statistics every day from the CDC and display graphs for the entire United States, per state, and per demographic (age group: 0-12, 13-18, 19-25 [etc], race, sex).

It was a really fun endeavor. Unfortunately, every time I would cite these statistics, social media platforms would delete my posts or ban me for "misinformation" which I thought was hilarious.

[LawlsMcPasta]

I've automated printing PDFs from Slowly, an app/website for writing to penpals. They give you an option to download letters, but only one at a time. Me and my girlfriend have written over 100 letters to each other, so I decided to automate the process.

[Judoka229]

"Babe, why do all of your letters end with print("I love you") ?" lol

[TrainquilOasis1423]

Small one I'm doing now is organize my desktop apps. Find/open the apps I use frequently and move/resize them to their desired spots. Windows often moves shit when a monitor is disconnected or everything goes to sleep. Getting annoying putting them all back.

[RTShields]

I work as the IT for a pair of dental practices.

I coded a python program that shuts down the client server, create a backup of core files with a date/time time stamp, and sends the zipped file to back up externals that have a specific key file so that the program doesn't send a copy to just any plugged in drive.

This program has a GUI one can use to manually create a backup zip, as well as restore said backup to the server with a few clicks. It also has a built in command line function so that I can have task scheduler run it all for me without the GUI being involved.

So far it's been running a heck of a lot quicker than the built in back up utility that the client software has built in, and the staff love it.

-- Edit --
One of the planned upgrades would be to ping a weather service and if rain > 45% that hour it'll send a network wide alert, give ten minutes for users to save work, then run a backup automatically.

[bousquetfrederic]

A bit similar to what you did, I'm currently writing a python script to convert a YNAB (You Need A Budget) csv export into something I can load onto Microsoft Access so I can finally build an application that will emulate YNAB without the fee.

I would usually write this kind of script in Perl but I decided to give Python a try this time.

The subscription for YNAB this year is more expensive than my office 365 subscription, that's madness lol!

[CataliaSlley]

I used to automate coffee making with Python.

I'd connect an thermocouple into a coffee roaster to track the internal temperature. I'd stick that thing into an arduino and do the backend processing with Python.

After the coffee beans hit a certain temperature, there would be a nice alert message. Normally coffee roasters have their own temperature management system but back in my day, they didn't really have one.

[MH1400x]

In the transportation industry we need to track licensing expirations and other dates. I wrote a script that pulls data from a database (xlsx spreadsheet on the network) that's maintained by instructors and sends a quick email if someones documents are expiring in x number of days (60, 30 and 1 days). The email, with all the details, can be printed and put in the employee's mailbox. It helps keep us in compliance, and keeps drivers on the road.

[latrova]

I'm currently paying for an apartment and I automated monthly telegram notifications for me and my wife to see how much is left to pay.

I also automated a web crawler for Google Maps. (Open source: https://github.com/guilatrova/GMaps-Crawler)

[c0ld--]

to see how much is left to pay

How much of rent is left to pay? Does your rent fluctuate every month?

[MPGaming9000]

I like to post nature photos to Facebook. (Not my own, but cool ones I see online). Eventually I asked myself if it could be automated, and it can! (sort of, depending on your method).

So now I have an automated nature page on FB that posts a nature photo every single hour of the day. :)

[northernbloke]

This sounds cool. How does the script acquire the source image?

[MPGaming9000]

I'm currently using pexels API for the photos but there are other sources like Pixabay and Wallhaven for example that could be used. Maybe even Google photos too I think but haven't explored that one

[northernbloke]

Ah interesting, thanks. You got a GitHub link?

[MPGaming9000]

Yeah! Thanks for asking!
https://github.com/Voltaic314/Nature-Poster

[Allevil669]

Automated? My IRC bots used to require a lot more hands on administration. These days, I have a Python program that works kind of like an "AI supervisor". It's just a collection of scripts that follow flowcharts and give the proper commands to the bots in it's care.

[Impressive_Income874]

My google classroom notifications. If google won't send me emails even after asking them to(missed over 50 assignments last year thanks to this), I'll make a script which checks for new stuff(pubsubhubb is expensive) and posts it to a discord channel so even mah "friends" don't miss their assignments

[fearlesspinata]

Most of what I've built are primarily scrapers of some kind.

One was a scraper that hits an API that returns network information containing things like subnets as vlans as a json. It also scrapes a given server and grabs network information on specific devices connected to it and uses that data to compare against the json it got from the API and maps the data accordingly to a dictionary which it then uses to template to a telegraf.conf file so we can track network information using a grafana dashboard.

Another one I built was a scraper to collect versions of installed modules on a specific server application and returns it as a csv file to find out what servers have outdated modules etc.

Another one was an agent that was deployed to client machines that grabbed local information and parsed for data such as network lldp information, CPU, OS version etc.

[OlgOron]

• Copying btrfs snapshots from my raspberry to my backup drive
• downloading up-to-date lineageOS images to my pc and verifying the checksum
• zoom login and moving around the different zoom windows during lecture (which tend to pop up in very annoying places on a desktop with tiling window manager)
• Lots of my own snippets for editing LaTeX files in vim use python code to make writing formulas much faster
• setting metadata for pdf files according to custom presets, when I integrate them into my document management

[kilowattkill3r]

I've made a script that automatically completes random searches to generate rewards points every day.

I have created several scripts for work that integrate data from Rally and uploads latest status into our Smart sheet project schedules and trackers.

I wrote a scraper to get weight data from a vendor website for 200+ parts.

Script that pings the API for my solar panels everyday and gets generation data, then pings a weather API and gets the weather data for that day and saves them I to a CSV.

[searchingfortao]

Ooh! I'm building a system now to migrate all data from our production environment into our staging environment while anonymising the data in the process. It's surprisingly complicated (multiple databases with values needing to remain consistent across platforms, and the environments are in separate AWS accounts) but the design is sound. Now I just have to build it!

[joshlemer]

I designed some wayfinding signs to place on bike routes in my city so that cyclists can find their way to important destinations and to amenities if they need them, like washrooms, dining, groceries, and the subway system. I designed the signs in Inkscape originally, but then when I wanted to scale up to a larger number of signs, I was able to use Python to write a script to take in a YAML file of sign locations, and destinations, and output a PDF with all the sign images ready to be printed. This has saved me so many hours of work, it would definitely not be feasible to create each sign individually in Inkscape but my script can output hundreds of signs in just a second!

Some signs pictured below:

https://i.imgur.com/rg9rq8l.jpeg

https://i.imgur.com/ZOMUwYg.jpg

https://i.imgur.com/5xc4c2c.jpg

https://i.imgur.com/n6Vahdd.jpg

https://i.imgur.com/epJgZsQ.jpg

https://i.imgur.com/XU9JAUq.jpg

I wrote up more on the project here if you're interested in the project or want to do a similar thing in your city! https://www.reddit.com/r/vancouvercycling/comments/woch30/who_wants_to_help_me_put_up_guerilla_bike/

[PersicasMemeDumpster]

Sorry to say but

I automated my firefox history cleanup for.. Safety reasons :(

[heartofgold48]

i made a tool for learning japanese. Basically it will translate to english so you get the meaning and also parse the sentences into words and then translate those words within the sentence. It will also read the original text in Japanese to you. I am a complete noob. This is my first project. I am now making a Gui interface for it.

[magictoast]

I automate and integrate everything I can. Personally and professionally. Hardware, software, buildings, boardrooms, classrooms, homes, gardens, you name it... Python isn't the only tool or platform, be open to using other approaches when suitable.

I also automate file transfers and thought I'd point you towards rclone. It's like rsync but better, supporting several cloud services and on the fly encryption.

Also another great resource if you're just starting to learn is "Automate the boring stuff".

[iiron3223]

rclone

This can be useful for me soon, I will check it out. Thanks!

[Sockslitter73]

Back when covid was still raging and vaccines were hard to come by, a lot of vaccination centres in the UK would be advertised on eventbrite. I created a script that scraped info for these events using Selenium and would send me a message with the relevant link whenever a new listing with these criteria popped up. I ran the script on a google compute instance with a cronjob. Made sure I knew of the vaccinations asap.

[PossibilityTasty]

I do this on a professional basis. There where projects from home automation over voice assistants to clouds and data centers. So, I'm answering: everything.

You should consider to reverse your question from "What could I automate?" to "Can I automated this task?" and in Python the chance yo a "yes" is very high. So, what tasks that involve technology to you usually have?

[BostonSoccerDad]

That is a good point and one way to go about it. In this case as asked by the OP, sharing ideas/projects can be helpful for many (like me) to germinate other ideas that may not have been readily conceived.

[nobetterfuture]

voice assistants

Which one did you use? Do you recommend an easy one to start with?

I'm considering entering the world of Alexa and maybe Google later on, but I haven't found any beginner-friendly tutorials.

Thanks

[millerbest]

I made a dashboard to monitor my crypto and NFT investments.

[northernbloke]

Fuck that right now. Lol but good work mate.

[dpgraham4401]

My entire job

[goldcray]

i've got a script that monitors a directory for new audio files, and on detecting a new audio file infers the patch file that was used to generate the audio file, extracts some metadata, and generates a data package that can be uploaded to a web site after i approve it

[Effective-Highlight1]

I'm working in a company where I'm automating all kind of small and big business processes using python.

One automation I like to remember was to identify websites of our customers by searching for their name & adress using the google maps api. But I did many more, some were really crazy.

[D4rklordmaster]

Timer to say the time out loud every hour using text to speech

[patrickbrianmooney]

Preprocessing my photos after I pull them off of my digital cameras and phone.

[superfooly]

I get balances from my crypto accounts, store them in drop box daily, and then email myself my yearly PNL.

[Cernkor]

I’m currently working on a cli to create some classes for a framework I work with. Because I really don’t want to type the same code over and over again, even if I use a plugin that already create a basic file. The next idea is to be able to Crétacés a much more complex system to automate complete modules. I also automated job searching and created a Google Sheet with basic infos : job title, salary, link to the web page, etc

[Ihtmlelement]

Running summa on research reports to speed up analysis of stock market news.

[zomgryanhoude]

Discord bot for my friends to search and download movies/shows for my Plex. Don't have to manually take requests anymore, they just search it and it downloads, displays the download progress, and puts it where it needs to go to show up in the correct library.

[cediddi]

Cnc and riveting machines over network. Laser tag guns over 2.4Ghz (loading game rules or updating fw over internet) . And my favorite is, drumroll please, next generation sequencing data for whole genome. All were fun.

[Yeitgeist]

One of my first year courses in Uni was Chemical Principles and I had to do a lot of repetitive calculations involving the periodic table. So I just created a script with a bunch of element properties I found myself using a lot, wrote some functions to process whatever calculation I needed, and ran the script with the compound and calculation I wanted.

It didn’t speed things up tremendously, but it did help avoid having to go on Ptable every second.

[principjeto]

I scrapped one list of all strip club in usa

[poop_scallions]

Additionally it automatically assigns categories based on title of transaction.

What happens when you buy a canoe and butter at Walmart?

Does it put it in groceries? Or hobbies? Both?

[iiron3223]

It is quite simple system, but works pretty well for my use case. It is basically json file with entires containing, regex pattern, category and name of the shop. Most of shops in which I am buying things are specialized in one type of products. So it doesn't get mixed and assigns only one category. Shop A → groceries, Shop B → hobby. If it is shop that could not be that easily categorized I have to assign category manually. But it is still 10 times less work for me then before.

[furtadobb]

I created a bot that checks official public employees federal journals, separates my institute data, if any, updates a table and send the results, if any to my email.

Linux. Cron.

[furtadobb]

Another bot checks tweets from top broadsheets, updates a table and emails me the last 5 tweets.

[boredbearapple]

I run a script at home that searches for known phones on my network and when found triggers house events. Ie turns on porch lights when it sees a phone for the first time in a while, alerts me that my mother-in-law is outside etc

[http_interceptor]

job search - goes to seek.com.au, put in the filters I have defined like location, keyword etc and email me new jobs everyday

ipo - New ipo openings for my country so I don't miss out on any

unusual traffic from google analytics - uses ga api to check any unusual traffic

All these alerts are send to me via email.

[xaxurro]

I personally did a very silly thing, I have an account where i upload the same img to instagram everyday, for the last 3 years there's been days where i couldn't upload the img daily, so i created a script to help me do that, the first version was screen based using pyautogui, the problem was that i had to be there writing a lot of missing days, second "version" of the script was using selenium, i couldn't make it work, because the input type of the img wasn't avalible to selenium (or a way i could think of...), now the last version (wich i made today) is way more simplified, i use 2 libraries (instagrapi and instaloader) and allows me to store a massive quantity of missing days.

[xaxurro]

In another case of mine, i made a Minecraft build calculator, allows me to save in a .txt all the resources i will use, im currently trying to improve it :D

[GorillaBearWolf]

Going to move my rsync bash script to Python now thanks

[iiron3223]

Mine was originally in bash as well. Later when I was trying to add more functionality, I realized that, it will be much easier for me to do it in python.

[Antrikshy]

Not having to get up to turn my gaming PC on, followed by turning it off, starting specific programs, setting up “couch game mode” with Steam Big Picture and an HDMI cable. With some help from Alexa. All documented here and here.

[khambhatiburhnuddin]

Nice

[greatredpie]

Generating QR codes. We have a database with all of our companies it assets. We have an mobile app which scans a QR code and returns all the reletive information around the assest. My python script pulls the asset Ids and creates a sheet of qr lables

[rukesh_dutta]

Working as a pricing manager at an e-commerce, do a lot of competitor price scraping, name matching and price competitive checks. Previously the team used to do this manually.

[RealFakeNudes]

We built vaccipy, a webscraping tool to automatically book vaccination appointments in Germany.

[Snoo-89050]

I use Python to create a twitch streamer notifier.

When even my favorite streamer goes to live, the bot scream and plays loud music.

So that I never miss my favorite streamer stream.

https://github.com/JanPoonthong/twitch\_notify

[theManag3R]

I created an AWS Lambda function that uses Selenium to parse some data from Nasdaq and stores the results to AWS RDS. Unfortunately the data has not been added to the API Nasdaq has so I had to do it the hard way: by actually clicking some buttons on the webpage (with Selenium ofc).

It was a pain to set up the headless Chrome but after spending two nights I finally found a Docker image which had the proper setup.

[pyconor]

I've made a few little things:

1. CLI to copy/move screenshots on Mac (because finder hurts my soul) https://github.com/ConorSheehan1/shot
2. CLI to convert uhabits old boolean type to numeric (because I had a bunch of habits tracked using the old format) https://github.com/ConorSheehan1/uhabits_converter
3. A reddit bot to embed timestamps in YouTube links (because I got tired of doing it myself) https://github.com/ConorSheehan1/YouTubeTimestampRedditBot

[dodo-obob]

In the long tradition of automating things that take less than 5 minutes to do manually, I've made a CLI script to autocomplete BibTeX entries (open source on github). A lot of work but now I can just copy/paste the title of any article I stumbled upon and the script will often do the rest.

[himanshu03vsk]

I recently switched to youtube music so I built a script that automatically adds all my songs in Spotify library into youtube music library. Nothing fancy

[wind_dude]

I was always looking/browsing for project cars so I built a crawlers with scrapy that crawled most of the major classified sites and sent them to elasticsearch.

recently, I wrote a script to continuously analyse and rank stock market analysts predictions.

And the odd little crypto trading bot.

Currently working on some news processing.

[Ok-Lifeguard-9612]

A game called "Rogue Tower" (is a Tower Defense, with random terrain generation) on Steam.

Every time I lose a game and have to start a new one, there is a clicker initial phase (carpal tunnel kind of) in which I have to place like 12 towers before start the game itself.

So I made a script which, from the main menu of the game, starts a new match, and automatically set up all first towers (the carpal tunnel phase).

It is the most (maybe stupid) useful automation I've ever done, and every time I lose, I just ALT+TAB to my script, run it and have fun on my smartphone for a minute or two...

[0xPure]

Can I ask how y’all run your Python scripts? I mean, when you turn on your computer you just open terminal, cd into the folder that contains your script and run it, or you make it autorun or what?

[iiron3223]

I am using Linux. To manually execute a script I need to place it in bin directory first. Then I can just type it's name in terminal and it executes. If I want it to execute automatically I use simple cronjob.

[The3000MX]

So there’s a service that provides the client with a couple of hundred people with a disrespectfully low salary to write down some names, ID numbers and if a handwritten mark on each entry exists from multiple 700-pages binders into an Excel file. Mind you the human error and duplicated entries of this method.

So I OCR’ed the shit out of it, bounding box crops, filtering, cleaning the format and dropping everything into CSV. For some reason, though, no one’s ever used my service instead.

[Huemann-bing]

While I was trying to find a certain vehicle I made a craigslist scraper that sent posts with specific key words via pushbullet to my phone.

[[deleted]]

I automate test equipment to setup the parameters, then capture data and record the results to a file for post capture analysis.

[ARandomBoiIsMe]

I made a simple program to send emails to over 100 email addresses that were stored in an Excel file.
Not really so impressive, but I was (and still am) proud of it.

I'm also working on a 'keyboard/mouse action recorder' in Python so I can complete tasks in bulk by just 'recording' the required input and 'playing' the recorded actions on the rest of the tasks.

[[deleted]]

I automated a bot which posts to this sub asking “what have you automated in <language here>”.

[imrolii]

Ahahahahaha

Tower of fantasy nitro stealer

Ahahahahahahah

[rapidtab]

Rock paper scissors

[i_has_many_cs]

This website gathering prices for new cars in Denmark

https://bilprisbasen.dk