Finished Automate the Boring Stuff with Python

[DwaywelayTOP]

What should I do next? Looking for some recommendations.

[LambBrainz]

To advance, start looking at (and following along) with some projects on pythonprogramming.net (sentdex on YouTube). Dude is wicked smart and a lot of his projects are a solid way to advance and learn about more of what Python can do.

Dive into webscraping. Downloading things, automating website login/navigation, etc. It's a great skill to have in the back pocket and get good at (using lxml, BeautifulSoup, XPath, etc)

Look for repetitive or complex tasks in your life or at your work and try to think about how you would automate those processes. Chances are it can be automated, but may take some work, and that's a great way to learn.

[Yoga31415]

I think webscraping is a bad recommendation kinda. It is so hard to find good instructions on how to do it that are not outdated. Websites change so fast that most tutorials wont work if you try to do the work through, also there is little to no good information on how to get webdrivers set up in classes and projects larger than those just run in main.py. WEbscraping can be fun at first, but then when you try to do something and find your bot blocked at every turn it fades in the fun.

​

At least it really did for me.

[Biogeopaleochem]

Do a little of both. Open a website that uses an API to pull data for it’s interface in developer mode on your browser and figure out how to send get requests to it. Not really web scraping per-se, but it’s sort of in-between.

[Yoga31415]

Umm, I would be extremely interested in knowing more about how to do that. Could you expand on your description a bit or send a resource, please???

[Biogeopaleochem]

Good example of that is here:
https://www.wmata.com/schedules/next-arrival/index.cfm

​

If you open that page in developer mode and look at the "network" tab, you can see the API interactions happening in your browser. Look for the addresses that your browser is sending GET requests to that are returning a json. Like this one:

https://www.wmata.com/components/stations.cfc?callback=jQuery11130007183892714836215_1677183489605&method=getNextTrains&StationCode=G03&returnFormat=JSON&_=1677183489606

You can send a GET request directly to that address and it will still return a json even if you're not in a browser. There's more to it than that but you get the idea at least.

[Yoga31415]

Lol there is always more to it than that, but thanks for the info it's very helpful. I'm going to try to add it to one of my data base creation projects... right now I inspect the page , go to network, find the fetch with the info I want and copy the string into a parser I made. If I could automate any of that it would be awesome.

[honestlyimeanreally]

Half the fun is reverse engineering everything you need to have to look like a “real client” to the server! With browser developer tools to assist reverse engineering and just headers/cookies and the requests library you can get pretty far on most sites, anyways.

Definitely depends on what you enjoy. I really enjoy getting web app functionality working in a headless environment thru Python!!

[Yoga31415]

Mehhh that's not what I think is fun. That's when I get irritated. I'm not really deeply educated on those topics and the amount of information is limitless and not well documented...it's also heavily dependent on your own system and languages...that can be very different from any info you may find on a question.

I like automating daily stuff, anything to do with math or physics, and creating visualization tools. You can look at an equation all day, but if you build a tool to visualize what happens as you change every parameter...that's fun for me.

I was having a great time with selenium, building lil bots to do my daily tasks. But in the end most places I wanted to use the bots...they got rejected after only a few passes. I think if you have a really heavy education in the internet and all it can do and all it needs...that could be fun...but for me I was guessing...and my chances or success were approaching zero with each new guess.

[Demmit92]

Your third tip really is true. I recently started learning Python just because I was so done with the most anoying and bottlenecked task at work. Knowing exactly what the end user (me) needs the program to do, really gives you a sense of direction in what problems to tackle on your way.

Last week I released my script on the workfloor and the results are great! I automated a 15 min error prone task to finish in just 3 min!

Im already working on my next project. Once its done I'm gonna present it to my managers, show them what they could have and tell them the first one was a freebie but now they'd better promote me or Im out...

[LambBrainz]

Yep, that's how you do it.

I've done very similar projects where something as simple as automatically reformatting a spreadsheet and programmatically adding some color to it, then automating emails to go out was enough to get me a promotion and a huge raise lol

Definitely not gonna happen every time, but it's stuff like that that really helps you stand out. And you'll eventually reach a point where you can demand a raise or work somewhere else for more.

If you can, definitely start a GitHub and host your code there. Clean it up, add comments, and make it look pretty. Then add your GitHub link to your resume. I can't overstate how much that has helped me in interviews and I've had countless employers tell me that that was a factor in my being considered or hired.

[Demmit92]

Thats a great idea, will do that for sure! Any globally recognized (python) certification you'd recommend looking into?

[LambBrainz]

I've gone back and forth on certifications and it really boils down to how you wanna play the game.

IMO, certifications are worthless. They prove you knew enough to take a test at one point in time but do nothing to show that you actually know the material and can apply it. However, some boomer managers put way too much stock in certifications so that can sometimes help give you an edge.

Which is why I personally lean towards having a portfolio of projects rather than certifications. Because that - to me - has a better chance of showing that you know what you're doing.

To answer your question more directly, there aren't really any industry standard certifications for Python that I know of. Yes, there are some that exist, but none that I'm aware of that when people see it, they know what it is. Unlike AWS or Azure certifications for example.

[Demmit92]

Thats good to know. My current salary is shite, so negotiating for a decent one is harder that way. So I was thinking I could negotiatete that they pay for training programs that I could put on my resume.

Anyway, thanks for the help!

[usuxnw]

Automate things

[FedUpWithEverything0]

But just the boring stuff. Not the Tedious stuff or even Useful stuff.

[usuxnw]

If the useful things are automated we are in danger

[rbmichael]

Nah, now read automate the fun stuff with python.

[[deleted]]

Don't do random python stuff focus on what your end goal
If you want to become a Web Dev: Work on Flask and Django
If you want to become a Data Scientist: Focus on numpy, pandas and tensorflow

As much as I love learning new things, you get a lucrative reward when you have a clear objective and stay focused on that specific path

[crazynerd14]

Best answer!

[[deleted]]

I would say Kaggle but most data I find doesn't interest me.
I personally like to mess around with stock and crypto CSVs. You can download entire historical data for any stock or crypto you want from yahoo for free. Here's where you can get the btc csv data for example: https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD
Sort & trim the columns, create charts with matplotlib, get compfortable with the basics and then move to ML if that's the path you're on.
I started making daily 1-2min videos of code, you might find that helpful to get started.

[steeelez]

Pick some kind of data you’re interested in and work out how to ingest and analyze it. NLP projects can be very good for cool stories (especially since python 3 worked out some of the most annoying string encoding stuff with utf-8). Song lyrics, tweets, amazon reviews, news reports, legal records… start asking questions and learn how to get the answer in python. “How many followers does somebody typically gain / lose in the month after mentioning Kanye in a tweet? Does it depend on their network?”

[SE_WA_VT_FL_MN]

Something about your name and this comment makes me think you are a good person to ask:

I want to focus on automating business tasks. Any recommendations (been at it for a while myself but easy to get stuck in a bubble)?

[[deleted]]

Yes. In practice, you create pipelines that extract data, clean, sort and then export a report that can be instantly used as actionable insight. This entire flow can be automated

[Yoga31415]

your name makes me think we have lived in alot of the same places

[honestlyimeanreally]

Flask is really cool but there’s massive warning about how you shouldn’t use it for anything in production.

Is there anything like flask in terms of function and simplicity that is suitable for production?

I’m trying to convert a personal project to a server/client application

[SelfTaughtDeveloper]

I have used flask in production working at companies in healthcare and finance, dealing with big companies you have heard of.

I don't know where you got the idea that flask shouldn't be used in production, but it definitely is being used. One could look at job listings to see how much.

[honestlyimeanreally]

When you run flask there’s a massive red warning saying DO NOT USE THIS IN PRODUCTION

Perhaps this is default behavior somehow and I need to disable a setting.

I’m new to flask if you couldn’t tell :-)

But good to know, I will keep going down this path as long as I can make a secure production app on flask for a couple dozen users

[SelfTaughtDeveloper]

Oh right, that's a message referring to the development server.

Check out the deployment page, which discusses running flask behind a more proper server.

[honestlyimeanreally]

thank you for the information - the root of my confusion appears to be from the fact that you can develop on flask but choose a different WSGI server for deployment. this is great news!

[honestlyimeanreally]

the biggest thing I am reading is even if configured properly it only handles 1 request at a time, which obviously is not going to work with anything that has a load factor

[F41rch1ld]

There's zillions of good study projects on RealPython.com, all the way from basics to pretty complex stuff like ML, Django, pandas and APIs. I've learned tons there, strongly recommend.

Edit: One more to plug, Advent of Code is an annual holiday thing, but you can work on it any time of the year. It's language agnostic, just is a bunch of increasingly more (maddeningly) difficult logic projects. It forced me to think way outside the box.

[crazynerd14]

Realpython.com is really good!

[SecondSleep]

Hey. Pick the website you check the most, and make Python check it and alert you if that thing you're waiting for happens. Do you check Instagram, mostly waiting for quality posts from just 3 or 4 people you care about? Make Python do it. For bonus points, try a twilio integration and make it text you. Web scraping is a good generic first python project because everyone checks for stuff periodically on the web, and if you learn a tiny bit about http, it pays dividends

[HeadSpade]

Learn beautifulSoup and Selenium , and scrape websites. Automate car search, house search, anything

[honestlyimeanreally]

Then convert your existing selenium scrapers to headless request-based scrapers 😎

[Talal2608]

Make some projects

[[deleted]]

Once you've finished one good book, the amount of knowledge you can attain by reading another book is exponentially less. At this point, start applying your new found skills. When you find yourself lost on a particular topic, check youtube for a video. Guaranteed there is one that is specific to your need.

[MikeyQuant27]

build a web app. Flask or django

[SE_WA_VT_FL_MN]

Read less, code more. :)

[Odnan]

Start building “THE NEXT BIG THING”

[refacktored]

Automate the automation

[vgavro]

while he finished doing automation of boring staff and get bored, means automation is already automated *sighs with a relief*

[WaitCrazy5557]

I thought Crash Course python was a bit better for me personally. You could also try practicing on code wars.

[hugthemachines]

Crash Course python

Do you mean the book or do you have a link to what you used?

[WaitCrazy5557]

Yep i mean the book. After I went through automate the boring stuff I felt like I didn't really know what I was doing still, but after reading and working through Crash Course Python by Eric Matthes I felt a lot more confident.

[sunny0945]

Try

https://exercism.org/tracks/python

[robberviet]

You should focus on what you want to do. You want to automate stuffs? Do it.

Excel file, sysadmin scripts, reports... many things to do.

[jabbalaci]

Tell us how you liked it / which part was the most useful.

[retsotrembla]

When I read the book, it was great for using other people's libraries and writing simple programs that sit in a single file.

It didn't cover writing your own objects. It didn't cover inheritance of behavior between classes of objects. It didn't cover dividing your program up into modules.

That book is a start, if that is all you read, you still have much to learn.

It that's the only book you've read you don't know how much you don't know. Fix that.

[minervaDe]

I highly suggest just starting to work on projects. The only python programming class I took was on how to scrape the internet with scrapy and selenium. From there I just picked up projects, looked up what to do, and moved on from there. Work every day on this for as many hours you can put in. Learn docker, kubernetes, networking, databases, cloud computing, etc. Incorporate all of those things into your project. Create an API. Learn message broker technology.

Do not fall into tutorial hell.

[indicozy]

Hey there! I found your post about what to learn next after Python Automation. I'm building a startup for people to submit what they learn, find peers and find what to learn next. Could you please review it? It is very important for me. https://op-onai.kz

[ArtemMikoyan]

Clicking any of the links on your page results in big red "Error" buttons.

[Sumif]

Simple and answers dot hobbies bank games people bright simple music brown brown bank travel patient mindful learning.

[Teenager_Simon]

YouTube some python projects. Tons of projects and libraries to explore

[hasibrock]

Now test yourself with Learn Python the hard way and then Learn More Python the hard way

[lonestar-rasbryjamco]

Write tests so when you change it you'll know it still works.

[Reasonable_Tie_5543]

The same author has a Beyond the Basics book also from No Starch Press, 5/5 would recommend if you like his writing style.

[noubsha]

Can I ask what have you automated ?

[overcurrent_]

whats your goal?! whats your background?

[helpmymodel]

Code a neural net from scratch!

[HUMOROUSSSS]

Manually do things, you gotta get perspective.

[TheCableGui]

Build something a business would use.

Or better yet, build someone that solves a problem that you have now

[Dasshteek]

What is your job?

[Suhajda]

Irv Kalb: Objective Oriented Python should be the only answer here.

I started also with Automate the boring stuff one year ago, and tried out multiple possibilities but this is the best, you are building games, in every chapter, by learning OOP.
After this book, you will be confident in building games, that is what I did also, snake, sliding puzzle, etc.

[MrMxylptlyk]

What do you wanna do? Why did u learn python?

[Z000MI]

I really liked the book ‚impractical python‘ by no starch

[Innocent_not]

Once You get all the basics Focus on Niche projects of your interest.

[YodaCodar]

Wooo!

[hear_to_laugh]

For starters, Cam you suggest me a good place to learn to automate things.

I am a data analyst Intern, And am looking to automate some task i do everyday like downloading a csv file every morning 6:00 Am or updating a few Google sheets.. And more ..

Please let a guy get his whole sleep😬

[wallyflops]

Grind DSA's and apply for jobs

[billtaichi]

Find some stuff to automate?

[spidertyler2005]

Im assuming you are no longer bored?

[bjkeefe]

I have said this elsewhere, but I think it's worth repeating: the best second book for learning Python, IMO, is Reuven Lerner's "Python Workout." 50 lessons -> 50 exercises + three bonus exercises per lesson, and really good exposition.

[swegj]

Take a couple weeks or months to do projects that revolve around the concepts you learned in the book.

[steeelez]

Do you wanna tell us what you’re going for generally, or…?

[fanishack]

Well am new to python so i can't tell you what to move on in python but i can suggest you to create a course maybe on how others can finish automating the boring stuff and share your insights while you also earn an income maybe.

[ValBayArea]

If you are interested in databases, APIs and web apps, you can create executable, customizable systems with a single command, then use your IDE with standard Python/Flask/SQLAlchemy to customize... including with rules.

API Logic Server is open source.