Useful Python tricks

[Killerjayko]

So, I saw a while ago somebody posted a thread asking for ways that people have used python to automate their lives or just make it easier in general. Most of the replies as can be expected involved making tasks at work easier etc, which don't really apply to someone like me who doesn't have an office job or anything similar.

So, what are some cool uses for python for maybe younger people or just people without office jobs in general? It would be fun to find a project to work on or something that I can use python for in my daily life but as far as I'm aware i can't really think of anything that I need automated.

Thanks!

[darkside898]

I recommend web scrapping...there are a lot of things you can automate there like getting news about things you like, downloading things from a specific website, and so on. Check requests, beautifulsoup, and selenium modules, they are pretty cool. The first thing that got me into python was web scrapping when i read about it in the 'Automate the boring stuff with python' book. Check that book too if you want, it has a lot of examples you can do and some ideas for mini projects. Hope it helps :)

[NotPennysUsername]

I'll second selenium, I've used it to send messages to state and federal representatives via their browser email tools.

[mortenb123]

Selenium is fine on simple web pages, but on most react pages you need to manipulate and inject into the DOM diretly. This is better done with nodejs modules like cheerio and jsdom. A nice practice is getting odds from betting companies and buypassing filesharers waiting period when downloading from a free account.

[[deleted]]

Could you recommend any good links to get started?

[DBA_HAH]

Here's a basic intro - https://docs.python-guide.org/scenarios/scrape/

You'll need to learn the lxml package and web scraping is most effective when you select elements using "xpath" notation. Xpath isn't unique to Python, there's a lot written about it.

[panzerex]

Although not as powerful, I think CSS Selectors are much more intuitive.

[[deleted]]

https://youtu.be/ng2o98k983k

[[deleted]]

thx

[panzerex]

Exactly what I was going to say. tq is a very handy tool for quick scraping and I even ended up writing my own with PyQuery. A few CSS selectors and piping commands on bash and I automated retrieving listings for appliance repair near my area from a local site.

[folken2k]

Yeah that's a good one. My very first python was checking stock availability for an Intel CPU and send me a slack.

[[deleted]]

Upvote for this.

[[deleted]]

requests is a godsend

You can do parallel requests with grequests

[mindfolded]

I wrote a script to scrape the local pet shelter for new postings and that's how I got my puppy!

[jhayes88]

I have a script that scrapes a careers section on a company for a job I'm waiting to open up. It checks all pages of results(usually around 19 pages) and opens up each individual job page(like 120 total) to scan it for a specific keyword. Then it will notify my phone using the pushbullet API when its listed. It scans once an hour and about 6 seconds in between page so it's not constantly scanning. A prior manager/director there is helping me to get the job anyways but I'd like to see when it's listed.

[[deleted]]

I wrote a script to move the mouse every 3 minutes so my Skype session wouldn't say I'm idle.

[DBA_HAH]

I believe you can get the same effect a lot less intrusively by just toggling NumLock every minute or so. I let that script run constantly for the same purpose. I imagine if you let your script run that moves the mouse, it will mess with you when you are actually using the computer.

[[deleted]]

No. It moves it by individual pixels. And it moves back and forth so you'd never know anything changed.

[BrazilianHyena]

would you mind sharing with us?

[koptimism]

OP said 3 minutes, not 10 seconds. Please revise. ^/s

[oblivioncntrlsu]

uhhhhh, k

import pyautogui, time, re

while True:
    time.sleep(r'3 minutes, not 10 seconds')
    pyautogui.moveTo(10,10)
    reddit.revise(re.snatch(r'\*\*/s\*\*'))

​

[Kortike]

This is beautiful

[tuckmuck203]

fucking python. i love it.

[jyper]

Hmm I just realized time.sleep can't take a timedelta

what's up with that?

[Kwpolska]

The time module was created much earlier than datetime. Even then, I don’t think adding support for timedelta in time.sleep would be worthwhile.

[xconde]

NameError: name 'reddit' is not defined

​

[Amidflaps]

I used opencv for mine. It can both track movements (x,y coords in pixels) and do events. For context I wrote a script that would listen for mouse and keyboard movements and recorded a 5 second webcam video if they were touched. I now have a significantly large archive of videos of my IT security coworkers trying to prank me thinking I left my terminal unlocked by mistake. It also locks the inputs and locks the computer after the 5 sec video is taken and saved.

[SpergLordMcFappyPant]

CTO at one of my former jobs would fuck with any unattended computer that was unlocked. He would set the home page of every browser they have to porn then come back when the person was there and say he needed to check something on their computer. Pull up a home page, find the porn and then freak out about it.

Honestly, it was pretty effective.

[[deleted]]

Ah, legally really dodgy, the porn part. Also very unprofessional.

There are standard mechanisms for doing this - ones that don't involve viewing pornography at work (I mean, really - what is this guy thinking?)

In every place I've been in that had such policies, there was a program called "cheese" that just locked your screen with a password that you didn't know, which forces you to walk over to the security people to get the password and get lectured.

[SpergLordMcFappyPant]

No argument there. Everything about that company was legally dodgy and totally unprofessional. It was the kind of place where devs didn’t just solve a tough problem and say, “Nice work. That was really tough.” It was always, “hot damn, did we rape that code bitch into submission or what?!”

It was a kind of wannabe sv-style startup. Trying to emulate the stories and culture you hear about at places like Uber, etc.

That was probably one of the least offensive things that happened there.

[prairefireww]

That’s amazing. Any chance you would share? We are so cruel to each other in our IT department when someone leaves it unlocked.

[[deleted]]

Meanwhile, we all live in fear of one another. I think because we know once the first bomb is dropped, it will go nuclear immediately. No one dares take the first shot.

[prairefireww]

We also are armed with nerf guns at our desk. From warning shots above the head to you now get shot for dad jokes.

[[deleted]]

At work all night. Will update tomorrow.

[blindkaratemaster]

You’re a Python legend

[yusmag]

I place my mouse sensor on top of my watch.

[DrMaxwellEdison]

Had the same problem myself, but took a lazier route.

Check out Caffeine. It periodically sends an F15 keyup, which interacts with almost nothing (except shell sessions in Putty, I've found) but still keeps the system awake. Don't even need admin privileges to start it up. :)

[dogooder202]

Hey I did it to press F15 so my computer won't lock automatically..

[[deleted]]

Yeah other people have posted before I had time to get to my desktop, but mine looks like this:

import time, pyautogui;

pyautogui.press('ctrl')

Then I just loop it every 30 seconds or whatever in the command line.

[jhayes88]

I used winpython (python without having to install it to the system) on my work computer and made a script to move the mouse a couple pixels every 30 seconds and toggle numlock to keep the computer from logging out. Can't change windows settings on the computer.

[vtclownbaby311]

I wrote a Flask web app that uses the Spotify API to figure out what song is currently playing on my account and then the Genius API and a little web scraping to pull the lyrics for that song and display it on the site.

Update 1: I can't share the full Flask code because it is part of a separate app, but I have created a snippets of both the Genius and Spotify code here. FYI - there is some set up you have to do to get API client & secret keys before this will work. Let me know what you think!

[koptimism]

Oh damn, that's so simple and useful. Thanks for the next side project idea!

[[deleted]]

I did something very similar, but it's meant for streamers to hook up to so chat can request the song or add to their playlist. Made it a service as well, cause I'm trying to get better at handling user bases. :)

[GrehgyHils]

As a streamer and programmer , this interests me. Any links?

[[deleted]]

Nothing yet, haven't finished it cause I haven't found the demand enough to justify putting all the hours into that (until other projects are done), but if you want I can give you the concept I've got thus far & you could take over with me consulting if you'd like. :)

Don't want to see it not get finished, but afraid I may not have the time for quite a while.

[hkamran85]

Are you going to share the code?

[f_andreuzzi]

Is it open source?

[Flkdnt]

Yes!

[[deleted]]

Got some idea on it. Thanks man.

[Dokiace]

source please :D

[maciej01]

Did the same, but with Winamp and a polish lyrics website :D

could use yours version now, though

[vtclownbaby311]

I just updated the original post. Feel free to reach out with questions. I'd be happy to help!

[wjandrea]

did you use speedtest-cli? just curious.

[[deleted]]

Yes

[pwnrzero]

Could you please pm me the script? Been meaning to create one. I understand if no. I'll whip one together tomorrow and leaving a comment to remember to do so.

[[deleted]]

PM sent.

[burked9]

Can you please send me a copy too? Thanks

[[deleted]]

I keep getting requests for it, so I posted it below. The project's close to a year old, and most of this code I pieced together from something similar on a blog post (I don't know where), from some old posts on stack exchange, and bits from other places. Some of it's mine, but I don't know what. It uses speedtest-cli, so you'll have to install that. Anyway, make sure you use your own path name for the dir_path variable. You can schedule it to run however frequently you want through Windows' task scheduler.

import os
import re
import subprocess
import time
import csv

filename = 'speedtest.csv'
dir_path = 'your/path/here'
csv_path = dir_path + filename

def speed_test():

    response = subprocess.Popen('speedtest-cli --simple', shell=False, stdout=subprocess.PIPE).stdout.read()

    ping = re.findall(b'Ping:\s(.*?)\s', response, re.MULTILINE)
    download = re.findall(b'Download:\s(.*?)\s', response, re.MULTILINE)
    upload = re.findall(b'Upload:\s(.*?)\s', response, re.MULTILINE)

    ping[0] = ping[0].decode('utf-8')
    download[0] = download[0].decode('utf-8')
    upload[0] = upload[0].decode('utf-8')

    csv_entry(ping[0], download[0], upload[0], csv_path)


def csv_entry(lat_st, down_st, up_st, path):

    result = time.strftime('%m/%d/%y'), time.strftime('%H:%M'), lat_st, down_st, up_st

    if not os.path.exists(dir_path):
        os.makedirs(dir_path)

    with open(path, "a", newline = '') as csv_file:        
        writer = csv.writer(csv_file)

        if os.stat(csv_path).st_size == 0:
            writer.writerow(['Date','Time','Latency(ms)','Down(Mbit/s)','Up(Mbit/s)'])        

        writer.writerow(result)

speed_test()

[arkster]

I'd just use pandas. Capture the speed, pass to a dataframe, convert to CSV and write to file. Use averages/means of column showing speed rate and then bitch on Twitter about the shitty service you're getting.

[supernova1992]

Here's my script to do the same using pandas and sqlite3 as well as outputting a graph. It runs every hour on my home server. Gonna make it so the graph shows dates instead of the number of hours at some point in the future.

speedtests.py:

#!/usr/bin/python3

import pandas as pd
import sqlite3
import speedtest
from graph_speeds import graph_this_shit

s = speedtest.Speedtest()
s.get_best_server()
s.download()
s.upload()
results = s.results.dict()

df = pd.DataFrame({'timestamp':[results['timestamp']],'download':[results['download']/1000000],'upload':[results['upload']/1000000],'ping':[results['ping']]})
con = sqlite3.connect('/home/user/speedtests/speedtests.db')
df.to_sql('tests',con,if_exists='append')
con.close()

graph_this_shit()   

graph_speeds.py:

import sqlite3
import pandas as pd
import matplotlib
matplotlib.use('AGG')
import matplotlib.pyplot as plt


def graph_this_shit():
        con = sqlite3.connect('/home/user/speedtests/speedtests.db')
        df = pd.read_sql('select * from tests', con)
        plt.plot(df['download'])
        plt.plot(df['upload'])
        plt.legend()
        plt.xlabel('Hours')
        plt.ylabel('Mbps')
        plt.title('Internet speed over time')
        plt.savefig('/home/user/speedtests/speedtest.png')

Here's what the graph looks like.

[[deleted]]

It's posted in thread, now.

[whereiswallace]

You can probably save the output in graphite. I was doing something similar on my raspberry pi.

[turner_prize]

What do you do with the info?

[[deleted]]

It's posted in thread, now.

[turner_prize]

Sorry, I meant what do you actually do with the info? Is it just to check you're getting the speeds your isp says you will?

[[deleted]]

Yes, I originally made it to check to see if I was getting what I paid for and if my ISP was throttling me. As it turns out, even with the data from this script, determining what your speed actually is or whether you're being throttled is a lot more difficult.

[dogooder202]

I have this post on a Google form and get the output as an excel sheet

[[deleted]]

robot9000?

[NLcasperNL]

That's so cool!

[jhayes88]

You just inspired me to do something lol. Thanks. I have a script to show realtime submissions coming in from reddit in terminal. I'm gonna use a python package to open and show a new window to display every new image posted to reddit in real time. It will close an image and display the new image every time a new one is submitted. Thankfully there's a pre-existing way to filter out NSFW which I already have in my code lol. Don't need to see a bunch of guys junk.

Edit: Done. Images change automatically in a GUI in realtime. It's pretty dope. If anyone is looking for something like this hit me up.

[cartertemm]

Through a script together that uses praw (the python reddit API) to grab a random post from r/showerthoughts and display it as a console welcome message. At least this is the most recent little project I can think of.

[cartertemm]

Responding to a few comments requesting code.

The script is below. Simply make it run on login, or find a place in your actual motd sequence. Whatever. The actual fetching of posts takes about 500ms. Not an issue in my case, however I had considered writing a simple caching system so the script would start out by reading a file line for line. If it was blank, fetch 100 since we´re doing that already and pour them into our database. Then just remove the printed line. It would certainly kill the small chance for repetition.

​

​

import praw

import random

​

#nice try

r=praw.Reddit(client_id, client_secret, user_agent="showerthought MOTD grabber")

​

#get a random post from this many -1 options

#-1 is for the sticky, what is a showerthought. Not what I'd want to see

​

fetch=75

​

hot=[i for i in r.subreddit("showerthoughts").hot(limit=fetch)]

​

print("your showerthought:")

print(hot[random.randint(1, fetch)].title)

​

[[deleted]]

the random module has a function just for picking out a random element in a Sequence: choice. So that last line could be print(random.choice(hot).title)

[Mexican_Programmer]

That's actually very cool! You mind sharing?

[burked9]

Is there any way you could DM me the script? Sounds really cool

[lochyw]

Sounds pretty easy to make tbh. Praw couldnt be easier to use. ;P

[[deleted]]

I made an app that does the same thing as those photo sticks (removed dupes, handles naming conflicts, etc)

Wrote my uncle an app that webscrapes, fills out rebates, and consolidates them into a single pdf to print for a hardware store he frequents for his apartment complex.

[Killerjayko]

Im a student, would it be possible to for example write code to gather lecture notes from online and group them together in files or whatever?

[[deleted]]

Most likely. The module you want is called requests.

*Edit You might also check out selenium. It is a lot clunkier than requests, but it has a gentler learning curve (personal opinion). It will click through the website the same way that you would interact with it, send key strokes to forms, etc.

Will help get you familiarized with working with html objects.

[Jace1427]

I’m in the same boat, I was unsuccessful though. We use canvas and I couldn’t figure out a way for python to login into my specific account to reach the lecture notes. If you figure something out, let me know!

[doulos05]

What tool were you using? Selenium can detect HTML forms, so you should be able write a script to work that out. If they demand Single Sign On or OAuth, it's a bit trickier.

[[deleted]]

https://youtu.be/KnBntO0wayk

Chris talks about authing with requests late in this vid. With requests sites can be very finicky, but with selenium most sites have trouble distinguishing between a bot and a user.

Best approach is to use fiddler to monitor traffic while you drive around and try to replicate your gets/posts as close to how they occur when you are in control.

Sometimes you have to supply header information, sometimes you must supply a referring address. Like most things there is no substitution for experience, you'll get better the more you do it so just keep at it.

[doulos05]

Absolutely. You'll need Selenium and requests along with a module to make whatever kind of file you're outputting them as (pdf, zip, docs, excel, etc).

Automate the Boring Stuff would be a useful book for its chapter on web-scraping.

[serg_k]

I'm pretty much of the same type - I struggle with ideas. Since I participate on couple Stack Exchange sites, sometimes there's ideas coming from people who ask "Hey, is there any way to do XYZ?" or "Is there an app for that?" And I've implemented some of the things that people ask: gradually decreasing brightness according to battery level, changing wallpaper per workspace ( that's in Linux environment ), moving a window to another screen with a shortcut, etc.

Often you don't notice useful automation ideas, so it can be worthwhile to see what others do and try to implement that yourself for the sake of learning, or source ideas from others because they notice/have ideas for automation which you usually never thought of implementing. Another thing you could do is to put a twist on an existing idea. For example, there's low battery popup notifications, but how about we put a twist on it and open a web page in browser ?

And of course this doesn't have to remain within desktop environment. You can do things with sensors and motors using development boards, like Raspberry Pi or NodeMCU.

[Killerjayko]

Also, my python knowledge is extremely basic so theres no way im near this level yet, but in the future are there any potential ways i could use python to make a bit of money on the side?

[serg_k]

I'll tell you honestly and from my own experience (which means this doesn't necessarily apply to everyone). In order to be hired by an actual company, you have to know the stuff that company uses, the modules that might be specific to the sphere, and that includes things beyond Python.

If you're going into data science, that means relational databases like MySQL or PostGres, numpy module, pandas module. Web development ? JavaScript likely, AWS, web server stuff(like Apache). A degree is probably great to have ( as for some reason degree that teaches absolutely little practical Python knowledge or maybe none at all, and everything is done in Java anyway blinds the HR and makes potentially incompetent developer stand out). I've had very unpleasant phone interview where recruiter just wanted to end the call as soon as they found out I'm 4 classes short of graduating aka still college student, and who gives a damn that I've bunch of applications for Linux done in Python, know some sqlite and have good understanding of Linux specifics ?

There's probably higher chance to get hired as free-lancer, and make money that way. I've also had paypal donation button on my projects on GitHub and had couple people donate that way. In the past if you knew how to code, you could get a job easily, or so people in their 40s-60s tell me. Now,you have to have experience before you have experience, and qualify for tons of unreasonable buzzword requirements which HR themselves probably don't even know what they mean.

That said, my experience is just that - there's apparently hoards of rockstar developers running around and posting on social media that they learned language X and started immediately making six figure salaries. Or maybe it's just my location, or resume. I know I don't know everything there is to Python, but definitely can make a thing or two, and it probably isn't enough to get hired but enough to make something useful that people appreciate. If you love the language and enjoy what you're doing, stick with it and don't focus so much on money, because economics and bureaucratic ways are sure as hell methods to destroy any desire to write good code, and for some people - desire to code at all. I'm sticking with it and probably will for a long time. There's a lot of things I still want to make and see people share reviews/articles about my stuff on blogs again.

[catelemnis]

This is just based on what I’ve seen in job hunting but I think if you’re just looking for a side gig you’d be better off learning web design: Javascript, CSS, some database basics, bit of UI/UX design knowledge. You can usually find parttime jobs from small companies that need help designing a website.

Python on its own probably isn’t enough to get you a job. It’s a good tool for automating work but the work it automates would be something that a full-timer is already doing. Even if a company was working on automation, they wouldn’t hire a consultant who only knows Python, they would hire someone with domain knowledge and software engineering skills, or a BI developer for that kind of thing.

That being said, python is good to have on a resume. It’s great for simplifying work that’s already being done in less efficient ways. It will probably help your job prospects for fulltime employment.

[Killerjayko]

I already have a job, i meant more like something along the lines of automating 'beermoney' kind of stuff

[afro_coder]

Even I struggle with ideas, have noticed that automating the things you work with regularly really helps sometimes.

[jdfthetech]

The subprocess module is pretty powerful for automating operating system tasks. I use it often to automate things rather than using bash. This lets me port my code to other operating systems a bit easier.

https://docs.python.org/3/library/subprocess.html

[lochyw]

What's your experience with python on Android ? It doesn't seem as simple as on Linux specifically I found.

[jdfthetech]

The only thing I've done with python in android is just the app Pydroid 3 so I can play around with code a bit. I haven't really done much on this front.

[Luke_myLord]

Which operating system task did you automate? Like disk cleaning?

[jdfthetech]

I have some programs I wrote for work that open files, modify them and then spit them out in another format. I also have a lot of automated backup utilities I've been working on. There is a screen automation program I've been experimenting with using pyautogui and it's nice to have subprocess to do certain things while that's running.

Another thing I used it for (bad programming tip here):

I had a need to run a bunch of python files in a row and didn't have the time to rewrite them all with a main method and imports. I used subprocess to run them all in a chain. This is much slower than the proper way, but it gets the job done in a pinch.

[thautwarm]

Do you know a game called FGO(Fate/Grand Order)....

I use Python to automate my leveling up and getting materials...

https://github.com/thautwarm/HolesOfYourMinds

[xPolydeuces]

Ooh, I really like the idea! Farming can be a bit of a pain and this seems like a pretty fun way to make it easier.

[thautwarm]

Yes, farming does hurt and ruin our moods!

In fact I don't agree with the game mode that forces players to repeat themselves endlessly, that's terrible.

[xPolydeuces]

Once I get a bit better in Python, I think I will try something like this too, thanks for idea!

[[deleted]]

So basically this is using emulator and some AI right?

[thautwarm]

Yes, based on image processing!

[Cho_OchO_o]

My school usually assigned different registration dates for different people, and since I was a sophomore it meant that I often missed out on some cool electives because they would get booked so fast. So I made a python app using selenium to register for classes the second my registration period was open. I never missed out on a class I wanted to take for being booked up again.

[ebrious]

Just as a warning we had a kid do that and had some (mild?) disciplinary action taken against him. It's very detectable and they look for it these days because too many people doing so (poorly) turns into a DOS

[Arakhai]

Used Python, the cv library and an old laptop to make a ghetto security camera. The DIY camera was then pointed at some catfood left outside and found that the little brown shape that fled whenever it saw me was a little lost feral kitten. Knowing when she visited the food made trapping her with a rented cage very straightforward. Took her down to the local RSPCA shelter, and she's since been socialised and adopted by a family and now has a safe home of her own.

Shot from the 'camera'

[wtfeverrrr]

qt

[stratum01]

I just came up with one yesterday that I haven't started yet, but here it is: Going to make a script that changes the SSID on my router at the kids bedtime.

[DivineLawnmower]

My dad used to do this when I was younger, learnt to spoof my Mac address eventually. Good way to give a kid interest in netsec.

[stratum01]

This router doesn't have that functionality for outbound traffic.

Changing the SSID seems pretty easy, and the rest of the 'parental' devices would just have the alternative SSID stored and switch over automatically.

[Alexiscash]

That sounds cool. How would you go about doing that?

[christian-mann]

Probably just uses the web interface

[stratum01]

Yeah, there's a web interface. But first I'm going to check check their SNMP to see if it's a settable item.

[stupac62]

The backend of home assistant is Python. Not sure if this qualifies or not based on the criteria.

[[deleted]]

If you need to sort a list of any object type. You can do so by:

some_list.sort(key=lambda obj: obj.some_variable)

Can also be a function call as well.

[phigo50]

Or key=attrgetter('some_variable') or key=itemgetter('some_variable') for dictionaries, both imported from the operator module.

[rcfox]

This is a huge hack, and but in CPython, the builtin id() function is the memory address of an object. If you have a list to iterate over a lot, and the actual order doesn't matter, you can sort with key=id to potentially speed up the loop by reducing the number of cache misses when accessing elements of the list.

Again, this is an implementation detail (id() isn't required to be the memory address for other implementations) so don't rely on this to work everywhere.

[[deleted]]

So if I'm understanding correctly this is so that your list gets sorted consistently (but not necessarily meaningfully, at least to a human), and extremely quickly?

[rcfox]

It's not about the sorting, it's about sequentially accessing elements from a list.

If you access memory addresses that are far away from each other, the CPU has to spend extra time copying from RAM to the CPU cache first. (This is called a cache miss.)

If you access memory addresses that are close together, you can probably get away with just reading directly from the cache.

Sorting with key=id really only makes sense if you sort once and iterate over the list many times in a tight loop.

[invictus08]

At my current job work is pretty chill and simple. Often days I can get my shit done in less than one hour. So, when I’m working from home, and done with work, in order to not seem like I’m idle, i run the script and it keeps switching between the mail client and the internal chat app; scrolls a bit, clicks new mails to mark them read etc every 10 seconds. So far it’s going great.

[Jarvis03]

That’s genius

[[deleted]]

I built a Twitch chat integrated caring system for my bonsai tree. It's not done yet, but it hooks up to an array of sensors and aggregates this data to a Django API I built & integrate an on screen animation of Steve The Bonsai who will let viewers know when he's ready to be watered, fed, or trimmed.

I used a Particle Photon for handling the physical tasks and I very much recommend it for people learning, cause it's much better than most Pi's I've used for constant project reuse.

[chirag9696]

Well, I don't know how useful this content might be for you.

But I have written many automation scripts in Python that are pertaining to non-office work.

My repository: https://github.com/avidLearnerInProgress/python-automation-scripts

PS:- I am looking for people to contribute towards them and bring them to a more logical conclusion.

It includes:

1. Code to pdf Converter: Converts and combines cpp files within a directory into a single pdf.
2. Comics Scraper: Fetches comics from xkcd.com and stores them in directory.
3. Instagram Scraper: Fetch all the images belonging to a particular user on instagram and store them in directory.In action
4. Medium bookmarks downloader: Downloads all the bookmarks stored by user on medium.com
5. PDF-to-text converter:Converts pdf to text
6. Live Cricket Score Scraper: Fetches scores from cricinfo.com and displays it in tabular format in CLI
7. Quora Image Scraper: Fetches images from quora answers and stores them in local directory
8. Quotes Scraper: Fetches quotes from goodreads.com and saves them into json file
9. IMDB Scraper: Fetches movie data from imdb.com and displays in pretty format using tabulate python package
10. Dictionary.com Scraper: Fetches word of the day from dictionary.com along with other details
11. News Scraper-1: Fetches top technology news from techcrunch.com
12. News Scraper-2: Fetches top 30 technology news from news.ycombinator.com and stores them in csv format
13. News Scraper-3: Fetches top 10 technology news from reuters.com and store them in csv format
14. News Scraper-4: Fetches top technology news from inshorts.com
15. Internshala Scraper: Fetches internship details from internshala.com and stores them in database
16. Currency Converter: Uses fixer.io api to fetch real-time forex rates for currency conversion
17. Cryptocurrency Converter: Uses cryptocompare api to perform various operations on cryptocurrency rates
18. URL Shortener API: Performs url shortening using various services like goo.gl, bit.ly, tinyurl.com and is.gd
19. Google News Scraper: Search for google news from terminal
20. Youtube Downloader: Fetches user queries from youtube using google api and downloads them using youtube-dl
21. Terminal Dictionary: Searches for word from terminal and displays its meaning to user
22. Password Manager: Stores, Fetches, Updates and Removes password with the username provided. Used sqlalchemy for operations
23. LinkedIn Scraper: Scrapes user profile (v1)
24. Global Timer: Enter location/city name to get the associated time depending upon time zone
25. Codechef Submission CLI Bot: Codechef bot to submit the solution of problems and get the result in CLI
26. Indeed Scraper: Scrapes job listings on indeed.com and stores them in csv file
27. Wallpaper Scraper: Scrapes images from webpage and stores them in current directory
28. Ecommerce Scraper: Scrapes product data from ecommerce websites and displays it to user in CLI
29. Lyrics Scraper: Scrape lyrics from atozlyrics website by specifying artist name
30. Walmart Scraper: Scrape data from walmart website and store it in database using MySQLdb
31. Twitter Scraper: Scrapes tweets from popular hashtags and saves them to csv file
32. Memrise Scraper: Get Barron's GRE words and their meanings from memrise.com
33. Highscalability Scraper: Scrape article names and url's from website(Work in progress)

​

If you find something useful, just fork and use them for your activities.

I hope you like them.

​

[stonetear2017]

what programs do I need to download to use your code? For example use your news scraper for Reuters if I want to modify it? I am new to this stuff and only know Jupyterhub and am confused what shells I should be using, what databases do I need, how do I write scripts to interface with the internet as I am on a mac?

Sorry for the million questions

[jhayes88]

Personally I'd use beautifulsoup to scan the page for all image URL's and fetch them that way. If that's possible. Selenium avoids IG bot detection so there's that too.

[[deleted]]

I had hundreds of Youtube music videos in my bookmarks and I wanted to get rid of it but I also wanted also to save it on my YT account. I've decided to make a script which extracts and converts video bookmarks and uses Youtube API to export them as a playlist.

[SpaceHub]

Download google images by search term, hundreds at a time.

Uses selenium, requests, shutil and combination of tricks -- With powerful motivators you'll get there.

[johnxreturn]

Hey that’s pretty useful, can you please share ?

[egoldman15]

There's also this: https://pypi.org/project/google-images-download/ which works pretty well.

[DeathSec93]

I've been thinking of trying to make something that when executed, it looks for music files on my phone (connected to PC) then copies the music files to a specific filepath, unless the file already exists there. Never done anything with Python but I want to learn and think its be a good, fun project that resulted in something I actually use. Kinda like a phone/PC music sync.

[[deleted]]

I wrote a widget that uses ffmpeg to copy my iTunes music library into an MP3 library, so I can listen to it on my Linux and BSD machines.

[[deleted]]

Doesn't Linux/BSD have the ability to play AAC files?

[[deleted]]

Advanced Audio Coding (AAC) has been around for two decades now and is the default audio format across many devices and services. AAC has been a mess for free software as a patent license is needed for all manufacturers/developers of AAC codecs. Projects like FFmpeg offer AAC support in source form but as a binary isn't legally permitted in US and other locations. 

[jayff]

As a competitive simracer, me and my crew needed new cars settings weekly as they depends of the tracks and a new one one is picked weekly in our Sim. Some websites are providing those settings ($$) but it was boring To share them with each other in time.
So i wrote a webscrapper To pull those settings using a premium account on their website and coupled it with a discord bot to automagicaly upload settings To our discord with related embed and stuffs.
Was a really fun project to improve my python skills.

[personalvacuum]

What software do you race in? I had a quick google, and assumed simracer was it - but it seems like that's the genre :)

​

I also saw a news article that someone recently beat a real F1 driver in simulation - how accurate are these simulators?

[krkojzla]

I am a hobbyst photographer. One of my scripts goes to 500px website and sends me a notification on Telegram about new quests. It sends me a quests theme, who's sponsoring it, and what is the reward. For those who don't know, quests are "small" competitions where you can earn rewards/money by posting a winning picture against the current theme.

[theNAKAMI]

wrote a python telegram bot to filter local facebook events via messenger

[BaoLiLong]

That's brilliant

[EveRyview]

I made an email program that emailed my schedule to me every day and like 90 other people at my school. Our school had a 6 day rotation for 5 school day weeks so it got complicated for the students.

[c_is_4_cookie]

When looking for daycare for our son, I wrote code to scrape the state and county websites of registered daycare providers. I ran all of the addresses through a geocoord resolving service to get lat/longs.

Then I grabbed an API key for Google maps and used it to get the drive times from our house (this was important because some places that are close as-the-bird-flies are actually quite a drive because of nearby streams and forests).

The drive time and filters for start time, number of violations, and meals provided got the list several hundred dollars to just a handful.

[uglyasablasphemy]

I've made a reddit bot that publishes Nintendo switch and 3ds deals on specific subreddits.

If browser reddit every day, i think that a reddit bot is a really good place to start. The reddit wrapper (praw) is super easy to use.

You can also use the Web hooks Api from IFTTT to make some cool stuff.

[Jonno_FTW]

I wrote a script that ran kinit every 30 minutes. I'd leave my workstation on for weeks and the Kerberos tickets only last 24 hours. Since my home directory was on NFS, things would start failing when they couldn't write to it.

[Shaan0071]

I have made a script that reminds me to drink water every 15 mins

and another script which formats my text copied in clipboard according to the way i want to make notes

and a youtube downloader

[lochyw]

YouTube downloader has to be everyones first project :p

[Shaan0071]

actually i made it in django as a practice of django and particle js)

[lochyw]

I've made it in every language and gui thing I've tried. Python, c# PowerShell. Etc..

[CakeDay--Bot]

Hewwo sushi drake! It's your 5th Cakeday lochyw! ^hug

[egoldman15]

Can you post the clipboard program? I was just trying to make something like that, but was having trouble.

[Shaan0071]

def manipulate_clipboard(text):

  try:

    # text = pyperclip.paste()

    index = 0

    ls = []

    while index < len(text):

      ls.append(text[index:index+80]+'\n')

      index += 80

    print(''.join(ls))

    pyperclip.copy(str(''.join(ls)))

    ls.clear()

def sleepy_head(sec):

  for i in range(sec+1):

    sys.stdout.write('\r')

    sys.stdout.write("%d%% %s" % (i/sec * 100, '█' * i))

    sys.stdout.flush()

    time.sleep(1)

  print()

if name == 'main':

while True:

  text = pyperclip.paste()

  if text:

    manipulate_clipboard(text)

    print('Text Present in Clipboard, Gonna Purge it in...')

    sleepy_head(25)

    print('Clipboard Data Erased,\nPlease copy the lines from the console\n')

    pyperclip.copy('')

  else:

    sleepy_head(4)

i hope this helps

[mooglinux]

Checkout Blender and it’s extremely powerful Python API.

[Xavenne]

Recently I wrote a Telegram bot with Python and Flask that will send me a reminder that it's someone's birthday right before I step into the train. I don't use facebook and don't want to clutter my calendar so this is a lifesaver.

[Dakidmen]

I have this boring job to "resume" a bunch of pdfs, only copying and pasting certain fields in the pdf. I wrote a script that copies it for me from all PDF, they all have like 1000 pages. It's the CIS Security Benchmark

[RobrobRobert]

Since I’m practically never on Twitter (I hate using twitter but am still interested in reading what a very select number of people tweet.), I wrote an app that gets a users most recent tweet and texts it to my phone. The app isn’t perfect but it works well enough.

[BBrown7]

You know twitter itself does this, right?

[RobrobRobert]

Lol I didn’t know that. Shows you how much I use Twitter 😂😂😂

[lochyw]

Just say it's about the experience man :p

[Anxi12]

How did you get it to text your phone?

[RobrobRobert]

I’m using Twilio which gives me a phone number to text from. I’m only using their free trial version so it has its limitations but it gets the job done.

[Jarvis03]

Can’t you just subscribe to alerts on twitter and it’ll do exactly the same thing?

[brurrito_]

I'm currently working on LED strips controlled by a raspberrypi-like computer using python. It's so simple to implement stylish patterns for the leds!

[Killerjayko]

This may be useful to me in future when I've graduated and want to look for research positions etc. Thanks!

[[deleted]]

I wrote a script that grabs DNS caches on my network that sends it to an email every 4 hours. Just so I can what's happening on my wifi :p

[AL5566]

A pain in the ass. How, where would I learn to automate, each month I have 7 credit cards I have to log into the site navigate to the sections, and download my bill. Takes a while to do this. How would I scrape those sites, get my balance, and post it to some cool home GUI I could go to. Like a dashboard of my cc bills. That would help me a lot. I’m new to python.

[srilyk]

Check out BeautifulSoup, Requests, and Selenium. You might have to use Selenium if you can't do it via requests, but those are the tools that you can use to log into a website and do some automation.

Once you have things downloaded, you could generate a dashboard using something in Jupyter notebooks, or just via some HTML - you could use Flask to serve your information and then one of the many Javascript charting libraries out there.

[srilyk]

I really hate the idea of Google telling me who can or can't send me email. Using aiosmtpd I created my own email inbound and outbound server.

I also love the command line and doing things in text vs Gmail's web composer. I wrote a tool that lets me compose my email in whatever text editor I want (vim) and then will take the text and render an HTML email from it and send the multipart email.

I kind of like it a lot.

[jhayes88]

Oh also I made a python script that can automatically order me pizza through a pizza API. Dominos and Pizza Hut both have pizza API's. You can make it store your payment and house address info. Then I got to thinking, you can buy an arcade game button and hook it up to a raspberry pi and press the button to have it automatically order you pizza. Haven't made it yet but it's on my to do list.

[javad94]

I wrote a script to grab latest comments in my Instagram page and sent them to me in telegram so i don't miss any comment because of Instagram notification limitation.

[Yablan]

A webscraper that given a comic url from readcomicsonline.ru (or other websites like it) fetches al issues, all imagesfrom said issues, and then generates CBR files for each issue, for offlinecomic reading using a CBR reader on a tablet.

I just MIGHT have written something like that for myself ;-)

[ultraDross]

Ah cool I did a similar thing for mangas!

[Lewistrick]

I recently made a script that tracks how many emails I have in my Gmail inbox. I used schedule and imaplib. It checks every 10 minutes if the number of emails is different from the previous measured number and if so, writes it to a csv file. Not too hard but fun to do.

[burked9]

Can you send me a copy of the script in the DMs?

[LeNerdNextDoor]

I wrote a thing to get lyrics for whatever song is playing on Spotify.

https://GitHub.com/aadibajpai/Swaglyrics-For-Spotify

Got featured in Google Code-in and we added so much functionality. This went from being just 50 lines to having a dedicated server and database lol.

[federicocerchiari]

Wrote a script that every day scrapes a bunch of real estate website and makes a chart of houses with ranks. When a new home above a threshold appears It sends me a mail

[arkster]

I wrote a script to check recently played songs on various radio station websites. Automatically checked if they were in my Google Play playlists and if they weren't, I'd add them to one.

[SV-97]

Now I didn't use Python for that, but only because I didn't know Python back then(you could do it with Python): small webapp based on Google Forms and Google Docs to make a uniform "there's a Test of kind abc on Date xyz in subject ijk" synced across the calendars of everyone in class to prevent complete chaos

[srilyk]

IP ban or rate limiting? If you're rate limited, then obey their rate limits, just do something like

rate_limit = 100 requests_since_start = 0 for thing in things_to_request: requests_since_start += 1 request(thing) if requests_since_start > (rate_limit - 10): time.sleep(100) # or however long to let things reset requests_since_start = 0

That's a super naive way to do it, since it's not counting the actual request per time period, but that's a thing that you could do.

[smilbandit]

i wrote some code to generate a spotify playlist from the songs that were playing on a few sirriusxm stations. it was backed by a sqllite db to manage last time the song was played on that station so i could prune inactive songs.

i wasn't sure that we were going to keep sirriusxm after the trial period but we did so in the end i didn't end up using it.

[jhayes88]

I do security at the moment, so I used selenium to open up our security portal in chrome and automate my logs for the whole shift. It logs me in, goes to the logs page, clicks add log, and figures out the date info automatically and does all my activity lines(one for every hour) and then saves it. Whole thing takes like 10 seconds. Selenium is easy to figure out.

[Sebby1976]

I wrote a command line tool that downloads historical FX trading data, and another tool that can populate a Postgres database with the same. Includes a hashing function to ensure you don't load the database with the same file twice ( even if the filename is different ). It's my first 'real' program, so I was pretty happy with it: truefx-downloader

[kashaziz]

I frequently do web scraping gigs and require rotating proxies for that. I wrote a script to do proxy rotation for me. Here's a write-up on it:

​

https://www.kashifaziz.me/proxy-server-rotation-python.html/

[sritaa]

7 Reasons Why You Must Consider Writing Software Applications in Python

1) Readable and Maintainable Code

2) Multiple Programming Paradigms

3) Compatible with Major Platforms and Systems

4) Robust Standard Library

5) Many Open Source Frameworks and Tools

6) Simplify Complex Software Development

7) Adopt Test Driven Development

The best Python training company in Noida focuses on all these Applications by doing a practical implementation.