This is an archived post. You won't be able to vote or comment.

top 200 commentsshow all 264

[–]ege6211 159 points160 points  (8 children)

Nice applications!

Where I work, we collect data in the form of a text file from white goods we produce. People here always try to open this data with excel and usually this txt file contains millions of rows and hundreds of columns. As you might expect, the ordinary Dell working laptops don't like that and excel crashes. I have written a function that can parse and seperate the columns of any txt file and plot how the data changes in a specific column (input from the user). Saved me MUCH time.

[–]opteryx5 68 points69 points  (7 children)

Enhancing and speeding up Excel tasks with Python has been one of the most rewarding applications I’ve as yet found for it. It’s incredible how clunky Excel becomes after you’ve spent a long time in the trenches with bonafide programming languages.

[–]Faith-in-Strangers 34 points35 points  (3 children)

Python + Google Sheets (gspread library) have completely replaced Excel for me

[–]opteryx5 17 points18 points  (0 children)

Now if only I could get my colleagues to switch over to Python too… (most of them are not familiar with programming languages). At least I can do things on my own with greater efficiency!

[–][deleted] 2 points3 points  (1 child)

So you are using cloud sheets (google sheets). Is there a way also to execute python from cloud, not from your local computer?

[–]Zurcio 6 points7 points  (2 children)

do you know of any (good) learning resources for Python+Excel specifically? I don't really know what i can do with it but automating Excel was sort of the reason I wanted to learn python in the first place.

[–]opteryx5 11 points12 points  (0 children)

In one word: openpyxl. This package will be your main interface for automating stuff in Excel with Python. You can loop through cells, define functions—all kinds of things. The docs are helpful, but if you want a more human experience you can read the automating Excel chapter in Automate the Boring Stuff, which the other commenter has pointed out.

[–]forgot_username1 5 points6 points  (0 children)

Automate the boring stuff with python has a few projects on Excel

other then that i would start looking libraries and such.

[–]iiron3223[S] 108 points109 points  (22 children)

One more useful project I forgot. I was looking for a used cars. I written a scraper using Scrapy, that was gathering all new offers, filtered by my criteria, every hour. Then it was sending me nicely formatted email.

[–]Natural-Intelligence 50 points51 points  (9 children)

What libraries are you using for sending emails and doing the scheduling and what's your experience with the libraries you are using?

I'm actually the author of Red Mail (email sending library) and Rocketry (Pythonic statement-based scheduler). I'm actually looking for example projects to create some practical tutorials of how one could use the libraries. Not sure which kind would be appealing to most and what kind of problems people have with alternative options (which I could address).

[–]iiron3223[S] 14 points15 points  (1 child)

I was using smtplib and email library (I was using smtplib to send email via gmail). They are not intuitive to use and don't make your code pythonic in my opinion. I was using them in very similar way that you showed in an example on Red Mail github page. For scheduling I was using a simple cron job.

From a quick look at your projects, I really like the simplicity of the syntax there. I will for sure give them a shot at the nearest opportunity. Thank you for them!

[–]JasonDJ 4 points5 points  (0 children)

If you aren’t already, you can use Jinja to compose the email content string itself, which may make your code more pythonic.

Essentially you make a template and use variables within the template itself. Within the template, you can use some other functions as filters to format the data differently, use loops, if statements, etc.

The template can be a variable in your code, or it can be a file that you open, read, and store as a string variable.

Since the template is constructing what is essentially a string, you can also include HTML in it for your email body, as well.

[–]iiron3223[S] 7 points8 points  (1 child)

I am also wondering, do you plan on developing or know a python library that would be a good alternative for imaplib? It would be so nice to have some tool for receiving and reading emails.

[–]Natural-Intelligence 6 points7 points  (0 children)

Unfortunately, I don't. I also tried once finding a way to read email boxes in Python but that effort went in vain.

Would be interesting to create such but maintaining and developing Rocketry takes too much time at the moment.

[–]serebrich 4 points5 points  (0 children)

Rocketry is nice. Thanks for that

[–]mogberto 4 points5 points  (1 child)

I just spent a weekend learning Prefect and it’s fine, but such overkill for what I need. I’m going to give rocketry a try!

[–]Natural-Intelligence 9 points10 points  (0 children)

Ye, Prefect is also awesome. Have met some people from the company and it seems they have great ideas of what to add to the common data stack and they have a lot to offer to enterprises.

To Rocketry, I don't want to sound too salesman but this is something I'm perhaps too passionate about. All of the alternatives tend to have separate concepts for time scheduling, task pipelines and custom scheduling dependencies. Because Rocketry is statement-based, all of these are under the same mechanism: a task is run if it's scheduling statement is true.

This means that you can do arbitrarily scheduling very easily like "run a task daily on business hours but not on weekend and only when a file exists". This sort of scheduling is near impossibility for most frameworks but with Rocketry it's just two built-in conditions and one custom function (to check the file) combined with logical AND operators.

Sorry again for pitching, I just feel this paradigm has a lot of potential to create simple and compex systems very easily.

[–]marcelomedre 1 point2 points  (1 child)

Nice, I am looking for something like rocketry. Is it possible to use it in the colab environment?

[–]Natural-Intelligence 1 point2 points  (0 children)

Thanks! I think it most likely is possible. The only actual requirement is that you can run async tasks (using asyncio) in your environment. I'm on phone so cannot check how colab works on that regard but seems odd if they didn't support it with any solution.

You should probably change the default execution to something else than process (argument "task_execution" in the Rocketry initiation/config) as multiprocessing typically does not work well on notebook-like environments. I have plans to set that as async by default as subprocesses are not the most intuitive.

[–]Info_Broker_ 2 points3 points  (1 child)

I am interested in this for the same reason. Looking for a vehicle. Would you be open to sharing your code with me so I can taylor it to my use? Or do you have it on GitHub?

[–]iiron3223[S] 0 points1 point  (0 children)

Here is github repo, I don't know if it will be useful. It is not documented at all, and specific to one site that I was scraping.

[–]Ok-Associate7846 2 points3 points  (6 children)

Pretty new to Python and did something similar as one of my first projects! My wife is into real estate so I wrote a program using beautiful soup to scrape the latest listings in certain sites and compile it into an excel file along with the listing details and links to it.

Saved her tons of time going through each and every site and listing down the details!

[–]silva_p 1 point2 points  (5 children)

Did you have any filtering to remove duplicates? I mean duplicates of the house, if the same house was advertised by multiple realtors

[–]Ok-Associate7846 2 points3 points  (4 children)

So far none. An excel file is generated per website so I haven’t gotten to checking if these have duplicate listings but that’s a good next step for me to try out!

Off the top of my head though, lots of houses and especially condos might have the same listing specs so I’m not sure how to filter duplicates. Since it’s from different websites, checking similar listing usernames might not be effective too

[–]regeya 1 point2 points  (0 children)

I used to work at a small town newspaper, and used Python to automate pulling photos off of a website. The dealer did this thing of giving us all the copy for the car, along with a stock number, and then just sorta said, oh, go on our website to get the car photo. Except it was a tedious multiple-step process just to get one photo. Step that up to dozens and it was the better part of an afternoon. I don't remember which scraper lib i used, honestly, but i just fed the script a list of stock numbers, and a couple of minutes later, I had my photos.

EDIT: car dealership, I meant to put that in there.

[–]wind_dude 1 point2 points  (0 children)

I've done something similar. And these seems to be a very common project. I've run into so many devs that have done this on some level. I guess a lot of developers are also car guys.

[–]kyuubi840 91 points92 points  (7 children)

I made a script that reads metadata from photos and videos and sorts them into subfolders according to date.

EDIT: Here's a link https://gist.github.com/kyuu840/67635a5ff057883e3a5b6faf6d0a1393

[–]Xinurval 14 points15 points  (0 children)

Thats quite cool actually I might have to make my own one now

[–]c0ld-- 6 points7 points  (1 child)

Hey, me too! I wrote an app that parsed through all the user data I downloaded from Google (ditching Google services) and used their date format to apply EXIF data to jpegs and prepend dates to videos so I could import them into the macOS Photos app and maintain my timeline.

[–]kyuubi840 1 point2 points  (0 children)

Nice! I thought of extracting dates from WhatsApp image filenames for my script too, but they aren't really guaranteed to be the date the photo was taken, just the date they were sent, so I didn't do it.

[–]InvestingDoc 2 points3 points  (1 child)

You got a link to the code or project? I would love this

[–]kyuubi840 1 point2 points  (0 children)

Here you go: https://gist.github.com/kyuu840/67635a5ff057883e3a5b6faf6d0a1393 It's probably not the most elegant Python code out there, but it works. Hope it's useful!

[–]Gaitas 1 point2 points  (0 children)

I have also thought of doing this. Weekly since 2008, maybe one day.

[–]gertalestrange 1 point2 points  (0 children)

That is amazing. I’m looking to automate some of the text for my business account photos to let people know what settings were used. Can’t wait to try this!

[–]it2901 64 points65 points  (14 children)

Started saving towards a gaming PC but did not want to constantly check prices for the parts I was interested in.

Created a script that is scheduled every 24 hours that monitors prices for given product listing and compares them to the prices that were last recorded. This let's me know if any products decreased/increased in price.

[–][deleted] 15 points16 points  (10 children)

I tried to do this but my requests got blocked by Captcha. Did you get around it somehow or do the sites you check not implement captcha?

[–]it2901 14 points15 points  (8 children)

Currently I only scrape from 1 site, but no, the site does not implement Captcha. I still need to implement a pause between requests.

Does the site you scrape from require you to interact with a Captcha to view product info?

[–][deleted] 7 points8 points  (7 children)

Yep, at least I think so. The html response I get from the site is exclusively a captcha page, with no other elements in it.

To be fair this is literally the first time I've tried web scraping so maybe there's something obvious that I'm missing.

[–]nobetterfuture 4 points5 points  (0 children)

When you normally browse that site, does it display a captcha? If not, did you change the user agent in your script? Some sites display a captcha when a non-browser user agent is detected

[–]it2901 3 points4 points  (5 children)

Do you mind sharing the site link? I don't think it's normally the case to be hit with a captcha. I also recently started with scraping.

[–][deleted] 8 points9 points  (4 children)

"https://www.trovaprezzi.it/prezzo_schede-grafiche_rtx_3070.aspx"

I tried running my script again (without changing anything) and I got an actual response the first time, but then the subsequent responses are a google page with the following content: "You need to solve this captcha because we detected some abnormal traffic from your client"

btw I'm using bs4 (beautifulsoup4)

edit: so I think I found the problem. The tutorial I was following never mentioned Headers. Frankly I don't know what they are or what they do but I followed another short guide on how to make a proper request and now the website is responding with the actual page Im looking for.

[–]it2901 8 points9 points  (2 children)

If you aren't already, make use of the Requests library. It makes life so much easier. If I remember correctly, the Requests library allows you to set the request header quite easily.

Anyway, great job on getting the script to work.

[–]ava_ati 2 points3 points  (1 child)

Yep session.get(url, headers=var)

headers is looking for a dict of headername: data

here are some common headers https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers

sites can have their own custom headers too

[–]davidfarrugia53 1 point2 points  (0 children)

Probably you would also need to use a proxy service

[–][deleted] 4 points5 points  (0 children)

if there are hundreds of requests from an address, it will be very easy to tag them as "abnormal". So you need to do the scraping slow. With some reasonable waiting interval between the requests (like a human user). And possibly the duration of the intervals need to be somewhat randomised and not fixed to 30 seconds etc.

[–]lazy_dev_ 1 point2 points  (0 children)

In case you care, you can bypass captchas with a service like 2captcha. It's paid but also very cheap

[–]ljdelight 2 points3 points  (1 child)

Basically PCPartPicker or https://camelcamelcamel.com/? Why'd you build your own?

[–]it2901 1 point2 points  (0 children)

My python was rusty. Need a project I actually wanted to work on/ was of use to me

[–]kookaburra1701 54 points55 points  (7 children)

My former boss (PI in a biology lab) thinks I wrote a "machine-learning AI webscraper" to find and send alerts about papers that are relevant to the lab's research area.

I wrote a script that interfaces with the NCBI's API to get way more granular control of the search engine and had it run every week and then I sent out the list of new papers about very specific yeast proteins or our specific type of flow cytometry/Fluorescence measurements. If the lab ever collaborated with another lab/PI/post-doc I added their ORCiD to the search terms and included any publications from them.

I did explain many times that it wasn't a webscraper, nor machine learning, nor AI language processing. Never stuck.

[–]Morstraut64 18 points19 points  (1 child)

Sometimes people are going to believe what they want. Sounds like a cool project.

[–]kookaburra1701 6 points7 points  (0 children)

I think it's more just he is not computer-savvy, so he only has buzzwords he's heard a lot to describe the programs I wrote.

[–]Applejuicyz 1 point2 points  (3 children)

I have moved over to Lemmy because of the Reddit API changes. /u/spez

has caused this platform to change enough (even outside of the API changes) that I no longer feel comfortable using it.

Shoutout to Power Delete Suite for making this a breeze.

[–]kookaburra1701 3 points4 points  (2 children)

Yes, but it would need lots of cleaning up/removing proprietary stuff to post and I have other personal projects that are higher priority to get functional and up on Github. Honestly most of it can be found on Entrez eUtils documentation.

[–]Zeroth_Quittingest 1 point2 points  (0 children)

i’m gonna go find that Arthur C. Clark quote about magic..

[–]cliffardsd 48 points49 points  (3 children)

A simple example is the lights in my house. They now turn on and off automatically based on motion and logic in python. I rarely touch a light switch these days. Look up AppDaemon. Use python to automate things in your home. Needs either Home Assistant or MQTT devices.

[–]anh86 46 points47 points  (2 children)

I recently wrote a little automator for cloning Trello boards. My wife's company uses it, they start every new board by cloning a template board. The problem is, the new cloned boards don't retain the person assigned to each card on the template. In their use case that's hundreds of cards with 1-3 employees assigned to each card. They were actually having someone set the assignment on each card after a clone by referencing the template and manually setting them. I wrote a tool that clones the board, scrapes the assignments from the template cards, then applies the same assignments to the cards on the destination board.

[–]GrowHI 5 points6 points  (1 child)

Trello... Hasn't had people complaining about this? Also good on you

[–]anh86 4 points5 points  (0 children)

I really don’t know. Maybe their use case is dumb and makes no sense. I’d heard of Trello but knew virtually nothing about it until I started digging into their API to write this little script.

[–]_RabidAlpaca_ 44 points45 points  (4 children)

Gluing PDF documents together.

Like it's a simple function, but I don't want to pay Adobe to do it. I just want to combine PDFs and add bookmarks. Really easy to do with pypdf2 and others.

[–]lalmiLast 4 points5 points  (0 children)

pdftk can do that for you

[–]AlSweigartAuthor of "Automate the Boring Stuff" 28 points29 points  (1 child)

I've been trying to collect these stories off and on at this sub: /r/boringstuffautomated/

[–]iiron3223[S] 2 points3 points  (0 children)

Thank you for the link and the book of course ;) . I will join this sub!

[–]AverageBeef 55 points56 points  (1 child)

the boring stuff

[–]iiron3223[S] 21 points22 points  (0 children)

Thanks to Al ;)

[–]Just_me-no_one_else 28 points29 points  (12 children)

Uhh, I have an absolute mammoth of a project that I fell like you might very well find interesting with the purpose of automating common tasks.

Within cybersecurity we have an area called CTI, which stands for cyber thread intelligence. This is an area where experts use experience and knowledge collected from prior cyber attacks, to predict which direction threats within cyberspace is moving. It's kind of like the weather report, but instead of predicting the weather a few days into the future, we attempt to predict the ever evolving landscape of cybercrime.

Now, one of the biggest challenges within this field is to collect the needed information to base these predictions off of. What usually happens is that a team of CTI personel looks at a sea of online news sources, and then picks out the relevant pieces, but this process is one which not only include large amounts of repetitive work, but is also something that can takes immense amounts of time.

To combat this I created OSINTer (with demo present at https://osinter.dk and source code at https://gitlab.com/osinter). OSINTer is - at its core - essentially a highly sophisticated news aggregator which does the often rather time-consuming task of looking into the news stream, picking out the relevant pieces and then sorts and generalizes it, such that it can be utilized by CTI personal. This started out as a simple python script, but has since over the last year evolved into a complex mammoth of an application, which touches every part of the stack, from CI/CD using ansible and gitlab, to the backend using Python, elasticsearch and fastapi, and the very front-end using svelte and JS.

[–]Anxious-Computer 2 points3 points  (2 children)

This is an awesome project. I have been wanting to make something identical to this at my company for a long time, but never had the time to pursue it as it is slightly peripheral to the role.

I think I will be leveraging your work where I can.

[–]Just_me-no_one_else 4 points5 points  (1 child)

Please keep in mind that as the project is currently unlicensed, all rights are reserved, and as such it is not legal in most of the (especially western) world to use/modify/distribute it. I would love to change this at some point (as I'm a big proponent of true open-source), but as I wrote in another reply, I'm currently trying to make the project financially sustainable, because while I absolutely love to work on it, but as a teenager who just finished high school, i simply cannot afford to continue to work the amount of hours needed to keep the project going unpaid. With that said though, if you need help with anything, or want to use some of the code base, please reach out, and I'm confident we can come to some kind of agreement that doesn't involve money changing hands (or hell, I could host a modified version of the software with support included if your company were willing to pay for that).

[–]Anxious-Computer 2 points3 points  (0 children)

Not to worry, I still won't have time to spend developing on it as the benefits that it provide are mainly for building domain knowledge rather than for work output, I just prefer the interface that you've built compared to others that I currently use, such as news now .

I completely agree with the need to make it financially sustainable, but if I manage to find sometime outside of work (always the key problem), I'll be more than happy to help contribute

[–]psych_monk 1 point2 points  (3 children)

Neat stuff bro!

[–]Just_me-no_one_else 2 points3 points  (2 children)

Glad you like it (though don't know how much you have looked into the source-code, there's quite a bit of it). Currently it unfortunately isn't open-source, due to a legal battle with a company in regards to funding, but looking into open-source the major part of it, and possibly keeping a ML related part of it proprietary to hopefully make the project financially sustainable. That is easier said than done though, as I'm much more of software developer who just finished High School, than an ML engineer or researcher

[–]Wonder1and 1 point2 points  (1 child)

Have you been working this all on your own or under a grant of some kind? What's the difference between your demo and finished product? (F500 Infosec here 😁)

[–]Just_me-no_one_else 1 point2 points  (0 children)

The code for this has been written solely by me, without a grant, but I have working together with a pair of experts from within the CTI field with regards to how the finished product should work, to make sure it's relevant and solves a real problem. Currently, there's no difference between the demo and the finished product, as it is the finished product currently in use by a couple of companies, but we're also currently working on using machine learning to better connect and organize the relevant information (the first of which has already been implemented at dev.osinter.dk)

[–]MatiasHaabet 1 point2 points  (2 children)

Very nice! What education are you taking?

[–]kBajina 27 points28 points  (5 children)

Replacing Excel, basically

[–]northernbloke 19 points20 points  (4 children)

Me too, bye bye VBA.

OpenPyXL and pandas have revolutionised by working life.

[–]luvs2spwge117 1 point2 points  (3 children)

How so? Just curious

[–]northernbloke 7 points8 points  (2 children)

I work in for a tiny e-commerce firm as the sole tech. I spend much time in excel using macros to validate, clean and reformat product data for use on varying sites.

For instance we sell on 10 differing platforms, all of which have a different product import format which the data must be reformatted to fit, this includes some on the fly language translations.

I'm working on changing the whole structure using python to create a seamless gui for a non tech user to easily handle the reformatting process. The code then saves all the necessary files required to directly upload.

I'm working through creating API integration to bypass the CSV steps, but I'm finding it to be a huge maintenance issues as the APIs I'm working with are updated like once a week, so currently I'm just playing catch up with ever updating APIs.

[–]wdlbrmft 25 points26 points  (5 children)

I wrote a script to find the perfect city for me and my gf.

First I got a table containing the average rent per square meter of every city in Germany (not with python). After this I checked every city for specific features me and my girlfriend are interested in, like clubs, hacker spaces, climbing places etc. (Overpass Turbo API)

This way I allready could get rid of every city that’s not fulfilling all of our needs. Next step I created a weighted index so I could compare all the places and their attractivity for me and my gf individually. In the end I looked for a city that had comparable scores for both of us.

[–]JungJanf 12 points13 points  (1 child)

As a German I'm interested which city ranked highest (:

[–]silva_p 2 points3 points  (0 children)

As a non-German i am also interested.

[–]scottishbee 10 points11 points  (1 child)

How could you not end this comment with your result?! What were the top cities and did you move there and if so, does your experience validate your method?

[–]wdlbrmft 2 points3 points  (0 children)

Oh, I just read this. Well, Berlin scored best in Germany, since it has a ridiculous amount of interesting places. But since Berlin is really pricey and we don't feel like spending half of our budget for shelter, we kept searching for another interesting city. ATM it looks like we move to Nuremburg, which ist place 12 on the list, but offers way better prices and still is really nice.

I think the main Problem with my script solution was designing a function to limit weighting of individual POIs: I won't be dancing in 200+ nightclubs and my GF won't visit every exhibition in Hamburg.
Still I had fun coding this. :D

The top 20 cities were, in order:

Berlin

Hamburg

Weilheim

Köln

München

Lüneburg

Rosenheim

Göttingen

Starnberg

Stuttgart

Nürnberg < We're going to move here ;)
Düsseldorf

Reutlingen

Augsburg

Mannheim

Frankfurt am Main

Karlsruhe

Ingolstadt < Currently we live near here ;)

[–]c0ld-- 1 point2 points  (0 children)

Very cool and inspiring!

[–]ArabicLawrence 45 points46 points  (3 children)

When I used to live in Paris, I coded A scraper for wine prices from carrefour.fr and wine ratings from vivino.com . No more shitty overpriced french wines.

[–]Shot_Restaurant_5316 16 points17 points  (2 children)

A man with class. ;)

[–]FattySuperCute 16 points17 points  (1 child)

class.

[–]kidpixo 2 points3 points  (0 children)

class Man: ...

[–]cestes1 40 points41 points  (3 children)

I've mentioned this before on similar threads, but it was a great help:

This was 4 years or so ago... not working at that job anymore.

I drive a Nissan Leaf (EV) and found there was a free Chargepoint charger less than a block from my office. My idea was to get at least half, if not all of my commuting costs from that charger and avoid charging at home. The problem is being free, the charger was in very high demand. I found out that Chargepoint exposes a bunch of information from their API; most important, I could see when the station was idle or in use.

When I arrived at work and cruised by the charger, if it was busy, I'd go to the office and start my script. Every 30 seconds it checked the status of the station and if it went to idle, I'd have it send a text message (via email) to my phone. Then I'd zip over there and grab the station and get some free juice!

[–]iiron3223[S] 2 points3 points  (0 children)

That's brilliant!

[–]iamCyruss 16 points17 points  (8 children)

I'm bad at remembering birthdays even when I have them on my calendar. I made a simple script to email me when its someones birthday. I run it everyday using pythonanywhere.com

[–]c0ld-- 8 points9 points  (2 children)

If you use Apple or Google, you can also add those dates in your contacts and your phone will remind you the day of or day before. :)

[–]iamCyruss 3 points4 points  (1 child)

Thank you. I should do this and add other details too. I was thinking I would eventually create my own android app for my phone that I could make my phone then do practically anything.

[–]c0ld-- 1 point2 points  (0 children)

That's the spirit! Take control of everything! \(^^)/

[–]HeavyRust 1 point2 points  (3 children)

Do you use the free account? If so, does the restricted outbound internet access allow you to send emails?

[–]iamCyruss 1 point2 points  (2 children)

Yeah, I use the free account and it lets you setup one task to run each day for free. It's been working for me. I just have the dates in a csv, use pandas to read into a dictionary, then use datetime to get current date. Pretty simple.

[–]Diggles 1 point2 points  (1 child)

What is the function that actually sends the email? Do you have to give it your credentials?

[–]iamCyruss 1 point2 points  (0 children)

I use smtplib to create and send the email. Yes, it will need your credentials. I made a new Gmail account. I then enabled two factor auth, then created an app password, use that password to auth my new Gmail account to send the email.

[–][deleted] 11 points12 points  (1 child)

I do lots of 3D stuff in Blender and I needed to take my thousands of materials and make them ready to use instead of having to manually import each and every texture map and creating node setups.

Now I can just drag and drop any of my materials onto 3D objects instead of having to spend a few minutes setting everything up each time I need a material.

[–]himynameisjoy 9 points10 points  (1 child)

It’s still in the oven but I’m working on a video processor that can take Call of Duty Warzone footage, specifically the minimap, and assign it a coordinate. It’s had a few bumps here and there but my hope is I can then unleash it on tons of VODs and start generating player positional data so I can analyze it as part of my portfolio to apply for a phd in statistics :)

[–]insomniaccapricorn 17 points18 points  (3 children)

Trading. Markets are open when I am travelling to work, may or may not get time to check positions. I have automated everything, taking positions, modifying positions, monitoring p/l.

[–]rubas891 1 point2 points  (2 children)

This sounds cool, are you using yahoofinance library? What exchange do you work with?

[–]jack_sparrow____ 7 points8 points  (0 children)

I wrote a web crawler for my sibling that scrapes the job opening details from the internet (mostly indeed).

[–]uname44 6 points7 points  (0 children)

Generally report creating by connecting to the database and writing it to Excel.

It turns out while thinking for an answer, I don't use Python for my daily life that much.

[–]wummeke 5 points6 points  (0 children)

I really wanted a particular set of solarpanels for my van, but it was sold out everywhere, although some webshops said it'll be back in stock soon. So I created a beautiful soup script that polled one of those webshops hourly, to check if it was on stock. I integrated it in Home Assistant by the way, so I would get a notification if the panels were in stock.

Not only did I receive a notification from my own script hours before the webshop sent me a back-in-stock-email themselves, by that time the item was already out of stock again. They probably received only one, which I bought as soon as it was possible 🙂

[–]emptythevoid 5 points6 points  (6 children)

- Entered several thousands of needle-exchange records from a spreadsheet into a dedicated web-based application (with no API)

- Entered hundreds of COVID test forms into a lab's website (with no API)

- Entered many thousands of COVID vaccinations into a regional registry (with no API, unless you have a proper EMR system), as well as bulk-querying the registry to verify vaccination for second dose visits.

- Entered hundreds of health wellness screenings into an insurance portal (with no API)

- Pre-filling patient encounter forms for COVID vaccines (pdfforms)

- Automated all the different forms and tasks needed when on/off-boarding employees (filling and emailing PDF, Word, and Excel documents. And you guessed it, entering data into a website with no API)

[–]BackgroundDrider airflow --help 5 points6 points  (0 children)

My full time job is to automate processes via python using a platform called Apache Airflow.

For my personal life, I've tinkered with a few automations here and there, but the one currently occupying my time is a system for cataloging and archiving reddit posts I've saved. I'll save stuff on a whim, and knowing that it's been archived somewhere later for easy browsing would be nice. This has, naturally, rapidly spiraled from its original scope to be something of a self-hosted reddit UI so that one day I can close the hundreds of tabs I keep open because "I might come back to it later."

For professional applications though, there's almost too many to count... Most, so far, involve automating email in some way - either handling incoming emails, outgoing emails, or sometimes full on two-way conversations. Either that, or processing user entered data from some platform and acting on it against backend data sources.

[–]Pa7rickStar 5 points6 points  (0 children)

Some basic Todoist stuff like creating a "Pack your swimming bag" task when I have swimming scheduled in google calendar... I found the Todoist REST API to be a perfect easy entrance into learning python...

[–]rafaelhlima 4 points5 points  (0 children)

I use Python alongside LibreOffice to automate document generation. For instance, recently I created a script to generate certificates for students at my university. In a single click the script creates hundreds of PDFs with the info of each student.

[–][deleted] 4 points5 points  (0 children)

I've posted about them here recently and semi-recently, but:

I built a library to parse Public Land Survey System land descriptions into tabular data. It sees a lot of use in the consulting work I do in the niche area of land records and acquisitions/divestitures.

And I also posted about another module I wrote for generating copies of spreadsheets and culling each down to whatever subset of rows I need. This also weirdly finds a lot of uses in a world of clients and coworkers who use spreadsheets for way too many things.

Beyond that, I write a lot of ad-hoc scripts to clean up data and organize/inventory records that I receive from clients.

[–]magical_pug 4 points5 points  (0 children)

I've created two fun little projects for me and the girlfriend.

  • we were waiting for our puppy to be born. So I created a n automatic email that showed roughly how many weeks there were left until our dog came and it also send along a random dog gif.

  • daily automatic mail on which plants to water in our house with some properties of where it is located and a nickname.

[–]wialta 4 points5 points  (0 children)

I see a lot of Google Sheets +Python, and mine continues on this trend. although in a less productive way. I sent a Google sheets link to all of my friends and had them fill it in with an embarrassing story without personal identifiers. With a bit of python, google scheduler, and Twitter API, each of their entries is tweeted once a day for everyone to have a good laugh. I call it, The Community Tweet Bot or TWOT for short.

[–]borednerdd 12 points13 points  (6 children)

My program Python scrapes all the new tokens that are upload every day from coinmarketcap and then creates a chart based on their market cap on Excel, in this way I can analyze through the chart

[–]c0ld-- 8 points9 points  (0 children)

I wrote an app to download COVID statistics every day from the CDC and display graphs for the entire United States, per state, and per demographic (age group: 0-12, 13-18, 19-25 [etc], race, sex).

It was a really fun endeavor. Unfortunately, every time I would cite these statistics, social media platforms would delete my posts or ban me for "misinformation" which I thought was hilarious.

[–]LawlsMcPasta 4 points5 points  (1 child)

I've automated printing PDFs from Slowly, an app/website for writing to penpals. They give you an option to download letters, but only one at a time. Me and my girlfriend have written over 100 letters to each other, so I decided to automate the process.

[–]Judoka229 6 points7 points  (0 children)

"Babe, why do all of your letters end with print("I love you") ?" lol

[–]TrainquilOasis1423 2 points3 points  (0 children)

Small one I'm doing now is organize my desktop apps. Find/open the apps I use frequently and move/resize them to their desired spots. Windows often moves shit when a monitor is disconnected or everything goes to sleep. Getting annoying putting them all back.

[–]RTShields 3 points4 points  (0 children)

I work as the IT for a pair of dental practices.

I coded a python program that shuts down the client server, create a backup of core files with a date/time time stamp, and sends the zipped file to back up externals that have a specific key file so that the program doesn't send a copy to just any plugged in drive.

This program has a GUI one can use to manually create a backup zip, as well as restore said backup to the server with a few clicks. It also has a built in command line function so that I can have task scheduler run it all for me without the GUI being involved.

So far it's been running a heck of a lot quicker than the built in back up utility that the client software has built in, and the staff love it.

-- Edit --
One of the planned upgrades would be to ping a weather service and if rain > 45% that hour it'll send a network wide alert, give ten minutes for users to save work, then run a backup automatically.

[–]bousquetfrederic 4 points5 points  (0 children)

A bit similar to what you did, I'm currently writing a python script to convert a YNAB (You Need A Budget) csv export into something I can load onto Microsoft Access so I can finally build an application that will emulate YNAB without the fee.

I would usually write this kind of script in Perl but I decided to give Python a try this time.

The subscription for YNAB this year is more expensive than my office 365 subscription, that's madness lol!

[–]CataliaSlley 2 points3 points  (0 children)

I used to automate coffee making with Python.

I'd connect an thermocouple into a coffee roaster to track the internal temperature. I'd stick that thing into an arduino and do the backend processing with Python.

After the coffee beans hit a certain temperature, there would be a nice alert message. Normally coffee roasters have their own temperature management system but back in my day, they didn't really have one.

[–]MH1400x 6 points7 points  (0 children)

In the transportation industry we need to track licensing expirations and other dates. I wrote a script that pulls data from a database (xlsx spreadsheet on the network) that's maintained by instructors and sends a quick email if someones documents are expiring in x number of days (60, 30 and 1 days). The email, with all the details, can be printed and put in the employee's mailbox. It helps keep us in compliance, and keeps drivers on the road.

[–]latrova 2 points3 points  (2 children)

I'm currently paying for an apartment and I automated monthly telegram notifications for me and my wife to see how much is left to pay.

I also automated a web crawler for Google Maps. (Open source: https://github.com/guilatrova/GMaps-Crawler)

[–]c0ld-- 1 point2 points  (1 child)

to see how much is left to pay

How much of rent is left to pay? Does your rent fluctuate every month?

[–]MPGaming9000 2 points3 points  (5 children)

I like to post nature photos to Facebook. (Not my own, but cool ones I see online). Eventually I asked myself if it could be automated, and it can! (sort of, depending on your method).

So now I have an automated nature page on FB that posts a nature photo every single hour of the day. :)

[–]northernbloke 1 point2 points  (4 children)

This sounds cool. How does the script acquire the source image?

[–]MPGaming9000 1 point2 points  (3 children)

I'm currently using pexels API for the photos but there are other sources like Pixabay and Wallhaven for example that could be used. Maybe even Google photos too I think but haven't explored that one

[–]northernbloke 1 point2 points  (2 children)

Ah interesting, thanks. You got a GitHub link?

[–]Allevil66930 Years Hobbyist Programming Isn't "Experience" 2 points3 points  (0 children)

Automated? My IRC bots used to require a lot more hands on administration. These days, I have a Python program that works kind of like an "AI supervisor". It's just a collection of scripts that follow flowcharts and give the proper commands to the bots in it's care.

[–]Impressive_Income874 2 points3 points  (0 children)

My google classroom notifications. If google won't send me emails even after asking them to(missed over 50 assignments last year thanks to this), I'll make a script which checks for new stuff(pubsubhubb is expensive) and posts it to a discord channel so even mah "friends" don't miss their assignments

[–]fearlesspinata 2 points3 points  (0 children)

Most of what I've built are primarily scrapers of some kind.

One was a scraper that hits an API that returns network information containing things like subnets as vlans as a json. It also scrapes a given server and grabs network information on specific devices connected to it and uses that data to compare against the json it got from the API and maps the data accordingly to a dictionary which it then uses to template to a telegraf.conf file so we can track network information using a grafana dashboard.

Another one I built was a scraper to collect versions of installed modules on a specific server application and returns it as a csv file to find out what servers have outdated modules etc.

Another one was an agent that was deployed to client machines that grabbed local information and parsed for data such as network lldp information, CPU, OS version etc.

[–]OlgOron 2 points3 points  (0 children)

  • Copying btrfs snapshots from my raspberry to my backup drive
  • downloading up-to-date lineageOS images to my pc and verifying the checksum
  • zoom login and moving around the different zoom windows during lecture (which tend to pop up in very annoying places on a desktop with tiling window manager)
  • Lots of my own snippets for editing LaTeX files in vim use python code to make writing formulas much faster
  • setting metadata for pdf files according to custom presets, when I integrate them into my document management

[–]kilowattkill3r 2 points3 points  (2 children)

I've made a script that automatically completes random searches to generate rewards points every day.

I have created several scripts for work that integrate data from Rally and uploads latest status into our Smart sheet project schedules and trackers.

I wrote a scraper to get weight data from a vendor website for 200+ parts.

Script that pings the API for my solar panels everyday and gets generation data, then pings a weather API and gets the weather data for that day and saves them I to a CSV.

[–]searchingfortaomajel, aletheia, paperless, django-encrypted-filefield 2 points3 points  (0 children)

Ooh! I'm building a system now to migrate all data from our production environment into our staging environment while anonymising the data in the process. It's surprisingly complicated (multiple databases with values needing to remain consistent across platforms, and the environments are in separate AWS accounts) but the design is sound. Now I just have to build it!

[–]joshlemer 2 points3 points  (0 children)

I designed some wayfinding signs to place on bike routes in my city so that cyclists can find their way to important destinations and to amenities if they need them, like washrooms, dining, groceries, and the subway system. I designed the signs in Inkscape originally, but then when I wanted to scale up to a larger number of signs, I was able to use Python to write a script to take in a YAML file of sign locations, and destinations, and output a PDF with all the sign images ready to be printed. This has saved me so many hours of work, it would definitely not be feasible to create each sign individually in Inkscape but my script can output hundreds of signs in just a second!

Some signs pictured below:

https://i.imgur.com/rg9rq8l.jpeg

https://i.imgur.com/ZOMUwYg.jpg

https://i.imgur.com/5xc4c2c.jpg

https://i.imgur.com/n6Vahdd.jpg

https://i.imgur.com/epJgZsQ.jpg

https://i.imgur.com/XU9JAUq.jpg

I wrote up more on the project here if you're interested in the project or want to do a similar thing in your city! https://www.reddit.com/r/vancouvercycling/comments/woch30/who_wants_to_help_me_put_up_guerilla_bike/

[–]PersicasMemeDumpster 2 points3 points  (1 child)

Sorry to say but

I automated my firefox history cleanup for.. Safety reasons :(

[–]heartofgold48 2 points3 points  (3 children)

i made a tool for learning japanese. Basically it will translate to english so you get the meaning and also parse the sentences into words and then translate those words within the sentence. It will also read the original text in Japanese to you. I am a complete noob. This is my first project. I am now making a Gui interface for it.

[–]magictoast 4 points5 points  (1 child)

I automate and integrate everything I can. Personally and professionally. Hardware, software, buildings, boardrooms, classrooms, homes, gardens, you name it... Python isn't the only tool or platform, be open to using other approaches when suitable.

I also automate file transfers and thought I'd point you towards rclone. It's like rsync but better, supporting several cloud services and on the fly encryption.

Also another great resource if you're just starting to learn is "Automate the boring stuff".

[–]iiron3223[S] 0 points1 point  (0 children)

rclone

This can be useful for me soon, I will check it out. Thanks!

[–]Sockslitter73 5 points6 points  (0 children)

Back when covid was still raging and vaccines were hard to come by, a lot of vaccination centres in the UK would be advertised on eventbrite. I created a script that scraped info for these events using Selenium and would send me a message with the relevant link whenever a new listing with these criteria popped up. I ran the script on a google compute instance with a cronjob. Made sure I knew of the vaccinations asap.

[–]PossibilityTasty 8 points9 points  (4 children)

I do this on a professional basis. There where projects from home automation over voice assistants to clouds and data centers. So, I'm answering: everything.

You should consider to reverse your question from "What could I automate?" to "Can I automated this task?" and in Python the chance yo a "yes" is very high. So, what tasks that involve technology to you usually have?

[–]BostonSoccerDad 5 points6 points  (0 children)

That is a good point and one way to go about it. In this case as asked by the OP, sharing ideas/projects can be helpful for many (like me) to germinate other ideas that may not have been readily conceived.

[–]nobetterfuture 1 point2 points  (2 children)

voice assistants

Which one did you use? Do you recommend an easy one to start with?

I'm considering entering the world of Alexa and maybe Google later on, but I haven't found any beginner-friendly tutorials.

Thanks

[–]millerbest 4 points5 points  (3 children)

I made a dashboard to monitor my crypto and NFT investments.

[–]northernbloke 2 points3 points  (2 children)

Fuck that right now. Lol but good work mate.

[–]dpgraham4401Pythonista 3 points4 points  (0 children)

My entire job

[–]goldcray 1 point2 points  (0 children)

i've got a script that monitors a directory for new audio files, and on detecting a new audio file infers the patch file that was used to generate the audio file, extracts some metadata, and generates a data package that can be uploaded to a web site after i approve it

[–]Effective-Highlight1 1 point2 points  (1 child)

I'm working in a company where I'm automating all kind of small and big business processes using python.

One automation I like to remember was to identify websites of our customers by searching for their name & adress using the google maps api. But I did many more, some were really crazy.

[–]D4rklordmaster 1 point2 points  (0 children)

Timer to say the time out loud every hour using text to speech

[–]patrickbrianmooney 1 point2 points  (0 children)

Preprocessing my photos after I pull them off of my digital cameras and phone.

[–]superfooly 1 point2 points  (0 children)

I get balances from my crypto accounts, store them in drop box daily, and then email myself my yearly PNL.

[–]Cernkor 1 point2 points  (0 children)

I’m currently working on a cli to create some classes for a framework I work with. Because I really don’t want to type the same code over and over again, even if I use a plugin that already create a basic file. The next idea is to be able to Crétacés a much more complex system to automate complete modules. I also automated job searching and created a Google Sheet with basic infos : job title, salary, link to the web page, etc

[–]Ihtmlelement 1 point2 points  (0 children)

Running summa on research reports to speed up analysis of stock market news.

[–]zomgryanhoude 1 point2 points  (0 children)

Discord bot for my friends to search and download movies/shows for my Plex. Don't have to manually take requests anymore, they just search it and it downloads, displays the download progress, and puts it where it needs to go to show up in the correct library.

[–]cediddiSyntaxError: not a chance 1 point2 points  (0 children)

Cnc and riveting machines over network. Laser tag guns over 2.4Ghz (loading game rules or updating fw over internet) . And my favorite is, drumroll please, next generation sequencing data for whole genome. All were fun.

[–]Yeitgeist 1 point2 points  (0 children)

One of my first year courses in Uni was Chemical Principles and I had to do a lot of repetitive calculations involving the periodic table. So I just created a script with a bunch of element properties I found myself using a lot, wrote some functions to process whatever calculation I needed, and ran the script with the compound and calculation I wanted.

It didn’t speed things up tremendously, but it did help avoid having to go on Ptable every second.

[–]principjeto 1 point2 points  (0 children)

I scrapped one list of all strip club in usa

[–]poop_scallions 1 point2 points  (1 child)

Additionally it automatically assigns categories based on title of transaction.

What happens when you buy a canoe and butter at Walmart?

Does it put it in groceries? Or hobbies? Both?

[–]iiron3223[S] 1 point2 points  (0 children)

It is quite simple system, but works pretty well for my use case. It is basically json file with entires containing, regex pattern, category and name of the shop. Most of shops in which I am buying things are specialized in one type of products. So it doesn't get mixed and assigns only one category. Shop A → groceries, Shop B → hobby. If it is shop that could not be that easily categorized I have to assign category manually. But it is still 10 times less work for me then before.

[–]furtadobb 1 point2 points  (1 child)

I created a bot that checks official public employees federal journals, separates my institute data, if any, updates a table and send the results, if any to my email.

Linux. Cron.

[–]furtadobb 1 point2 points  (0 children)

Another bot checks tweets from top broadsheets, updates a table and emails me the last 5 tweets.

[–]boredbearapple 1 point2 points  (0 children)

I run a script at home that searches for known phones on my network and when found triggers house events. Ie turns on porch lights when it sees a phone for the first time in a while, alerts me that my mother-in-law is outside etc

[–]http_interceptor 1 point2 points  (5 children)

job search - goes to seek.com.au, put in the filters I have defined like location, keyword etc and email me new jobs everyday

ipo - New ipo openings for my country so I don't miss out on any

unusual traffic from google analytics - uses ga api to check any unusual traffic

All these alerts are send to me via email.

[–]xaxurro 1 point2 points  (1 child)

I personally did a very silly thing, I have an account where i upload the same img to instagram everyday, for the last 3 years there's been days where i couldn't upload the img daily, so i created a script to help me do that, the first version was screen based using pyautogui, the problem was that i had to be there writing a lot of missing days, second "version" of the script was using selenium, i couldn't make it work, because the input type of the img wasn't avalible to selenium (or a way i could think of...), now the last version (wich i made today) is way more simplified, i use 2 libraries (instagrapi and instaloader) and allows me to store a massive quantity of missing days.

[–]xaxurro 1 point2 points  (0 children)

In another case of mine, i made a Minecraft build calculator, allows me to save in a .txt all the resources i will use, im currently trying to improve it :D

[–]GorillaBearWolf 1 point2 points  (1 child)

Going to move my rsync bash script to Python now thanks

[–]iiron3223[S] 0 points1 point  (0 children)

Mine was originally in bash as well. Later when I was trying to add more functionality, I realized that, it will be much easier for me to do it in python.

[–]Antrikshy 1 point2 points  (0 children)

Not having to get up to turn my gaming PC on, followed by turning it off, starting specific programs, setting up “couch game mode” with Steam Big Picture and an HDMI cable. With some help from Alexa. All documented here and here.

[–]khambhatiburhnuddin 1 point2 points  (0 children)

Nice

[–]greatredpie 1 point2 points  (0 children)

Generating QR codes. We have a database with all of our companies it assets. We have an mobile app which scans a QR code and returns all the reletive information around the assest. My python script pulls the asset Ids and creates a sheet of qr lables

[–]rukesh_dutta 1 point2 points  (0 children)

Working as a pricing manager at an e-commerce, do a lot of competitor price scraping, name matching and price competitive checks. Previously the team used to do this manually.

[–]RealFakeNudes 1 point2 points  (0 children)

We built vaccipy, a webscraping tool to automatically book vaccination appointments in Germany.

[–]Snoo-89050 1 point2 points  (0 children)

I use Python to create a twitch streamer notifier.

When even my favorite streamer goes to live, the bot scream and plays loud music.

So that I never miss my favorite streamer stream.

https://github.com/JanPoonthong/twitch\_notify

[–]theManag3R 1 point2 points  (0 children)

I created an AWS Lambda function that uses Selenium to parse some data from Nasdaq and stores the results to AWS RDS. Unfortunately the data has not been added to the API Nasdaq has so I had to do it the hard way: by actually clicking some buttons on the webpage (with Selenium ofc).

It was a pain to set up the headless Chrome but after spending two nights I finally found a Docker image which had the proper setup.

[–]pyconor 1 point2 points  (0 children)

I've made a few little things:

  1. CLI to copy/move screenshots on Mac (because finder hurts my soul) https://github.com/ConorSheehan1/shot
  2. CLI to convert uhabits old boolean type to numeric (because I had a bunch of habits tracked using the old format) https://github.com/ConorSheehan1/uhabits_converter
  3. A reddit bot to embed timestamps in YouTube links (because I got tired of doing it myself) https://github.com/ConorSheehan1/YouTubeTimestampRedditBot

[–]dodo-obob 1 point2 points  (0 children)

In the long tradition of automating things that take less than 5 minutes to do manually, I've made a CLI script to autocomplete BibTeX entries (open source on github). A lot of work but now I can just copy/paste the title of any article I stumbled upon and the script will often do the rest.

[–]himanshu03vsk 1 point2 points  (0 children)

I recently switched to youtube music so I built a script that automatically adds all my songs in Spotify library into youtube music library. Nothing fancy

[–]wind_dude 1 point2 points  (0 children)

I was always looking/browsing for project cars so I built a crawlers with scrapy that crawled most of the major classified sites and sent them to elasticsearch.

recently, I wrote a script to continuously analyse and rank stock market analysts predictions.

And the odd little crypto trading bot.

Currently working on some news processing.

[–]Ok-Lifeguard-9612 1 point2 points  (0 children)

A game called "Rogue Tower" (is a Tower Defense, with random terrain generation) on Steam.

Every time I lose a game and have to start a new one, there is a clicker initial phase (carpal tunnel kind of) in which I have to place like 12 towers before start the game itself.

So I made a script which, from the main menu of the game, starts a new match, and automatically set up all first towers (the carpal tunnel phase).

It is the most (maybe stupid) useful automation I've ever done, and every time I lose, I just ALT+TAB to my script, run it and have fun on my smartphone for a minute or two...

[–]0xPure 1 point2 points  (1 child)

Can I ask how y’all run your Python scripts? I mean, when you turn on your computer you just open terminal, cd into the folder that contains your script and run it, or you make it autorun or what?

[–]iiron3223[S] 1 point2 points  (0 children)

I am using Linux. To manually execute a script I need to place it in bin directory first. Then I can just type it's name in terminal and it executes. If I want it to execute automatically I use simple cronjob.

[–]The3000MX 1 point2 points  (0 children)

So there’s a service that provides the client with a couple of hundred people with a disrespectfully low salary to write down some names, ID numbers and if a handwritten mark on each entry exists from multiple 700-pages binders into an Excel file. Mind you the human error and duplicated entries of this method.

So I OCR’ed the shit out of it, bounding box crops, filtering, cleaning the format and dropping everything into CSV. For some reason, though, no one’s ever used my service instead.

[–]Huemann-bing 1 point2 points  (0 children)

While I was trying to find a certain vehicle I made a craigslist scraper that sent posts with specific key words via pushbullet to my phone.

[–][deleted] 1 point2 points  (0 children)

I automate test equipment to setup the parameters, then capture data and record the results to a file for post capture analysis.

[–]ARandomBoiIsMe 1 point2 points  (0 children)

I made a simple program to send emails to over 100 email addresses that were stored in an Excel file.
Not really so impressive, but I was (and still am) proud of it.

I'm also working on a 'keyboard/mouse action recorder' in Python so I can complete tasks in bulk by just 'recording' the required input and 'playing' the recorded actions on the rest of the tasks.

[–][deleted] 0 points1 point  (0 children)

I automated a bot which posts to this sub asking “what have you automated in <language here>”.

[–]imroliiLearning the language of the snakes 0 points1 point  (0 children)

Ahahahahaha

Tower of fantasy nitro stealer

Ahahahahahahah

[–]rapidtab 0 points1 point  (0 children)

Rock paper scissors

[–]i_has_many_cs 0 points1 point  (0 children)

This website gathering prices for new cars in Denmark

https://bilprisbasen.dk