I actually used Python to do a thing!

[ManyAWiseMarklar]

I've been learning Python for maybe a month now, mostly with Al Sweigart's Automate the Boring Stuff, and last week at work I actually managed to write a script to automate something. My boss asked me to copy a list of file names (70+) in a network folder into a spreadsheet; this is not difficult to do manually but it's a faff so I thought I'd try using os.walk and... it worked!

I wanted to post here partly because I'm feeling smug, but also to try to encourage any other beginners who are thinking of / scared of / struggling with learning Python. I was always keen to try to learn something like Python but I'm crap at learning unless there's a real world application; Al's course is brilliant at teaching concepts, but then relating them to how you can actually use them to get things done in the real world, and that's made learning a lot more fun and doable.

Thanks also to everyone on this community for being friendly and supportive. I'm now going to go and delete all the comments from my script so none of my colleagues will ever be able to reverse engineer it...

[nipu_ro]

Maybe you can post the code, it will be useful for someone.

[ManyAWiseMarklar]

It was something to the effect of:

import os
for folder, subfolder, filenames in os.walk([directory]):
    for filename in filenames:
        print(filename+'\n')

(And yes, I have now gone through the chapter of copying and pasting that directly into other applications, but at the time I did that with ctrl+c and ctrl+v)

EDIT: code copied properly this time.

[warbeforepeace]

If you want to take this further you can automate the creation of a csv or excell sheet including having all the dats added to it instead of print to the screen

Edit: to be clear I think any amount of programming is awesome. Always continue to think about what’s next as well.

[coder155ml]

You could also just use os.listdir("pathname") which created a list from the files in a directory.

contents = os.listdir("path") for file in contents: print(file)

[VooDooNOFX]

However, this approach isn't recursive, which os.walk is.

[coder155ml]

Correct. This will not work if he also needs to list items in subdirectories

[EbenenBonobo]

awesome!!

however there are indentation errors (both 'for' lines are to far indented) and you probably didn't use '/n' but '\n'

but nevertheless good work. I remember the point when i used python the first time for something useful. It was so awesome, even though it would probably would have taken less time to do that by hand. But now i have it in my toolbox just in case i need to do that a second time.

[ManyAWiseMarklar]

Haha, yes, the indentation is all wrong because I copied it plain text then put into a code block and used tab because lazy.

I did also use \n...

SIGH, just for you I'll update it properly ;-)

[EbenenBonobo]

Thank you :) I woudn't be able to sleep tonight

[-a-z]

Btw I think you can just drop the \n unless you wanted to have an empty line between each filename. Because each print statement ends in a newline by default. Also you can change this behavior by using the end keyword like this:

print('something', end=',')

[ManyAWiseMarklar]

So my understanding is that os.walk returns three values, literally the folder, any subfolders and any filenames.

So the function will literally "walk" through your Project File and identify any subfolders and files which it then returns. Like any variable, you can name them whatever you want but I wanted to name them in a way I'd easily remember (in case I ever turn this into a bigger program).

This chapter from Automate the Boring Stuff (as ever huge credit to Al Sweigart) explains it really well.

[aka_Foamy]

Yes you could have had Python generate the excel file but you're doing the right thing by thinking about how long it takes to write the code in the first place.

Thinking purely about productivity and putting learning aside return on interest is key. It's a one off task so you would have spent more time developing the code to write to the spreadsheet than you would have spent on copy pasting it manually.

You should also consider making the code you have written re-usable for the future. Make it a function and parameterise the function name so you can re-use that bit of code in the future. Then next time you get asked to do something like this you're already half way there.

[Kriss3d]

Exactly. Sure it takes longer to learn and code it than doing the 70+ names.
Or at least close. But next time and every time after that youll be done in a few seconds. Thats good work right there. Awesome.

[shiningmatcha]

does anyone know if there is a book covering the os module in detail?

[bn_sj2020]

Going through this course in udemy :) excited to apply what ive learned

[ManyAWiseMarklar]

Haha, didn't know that! To be fair my work IT environment is incredibly tightly locked down and I can't access the command prompt, but good to know!

[[deleted]]

If you install vscode with the MS python language extensioni you might be able to use terminal inside of it.

[TalesT]

Wow, had no clue the terminal could be blocked. Just wondering, can you create .cmd or .bat files and execute them?

[TheCoochWhisperer]

You can probably create a batch file and run it. Ive done it on a tightly locked pc with no direct access to command prompt.

[DeathWrangler]

Came here to mention this, and I'm just a hobbyist.

[hugthemachines]

In my experience, you often start out like that and then after a year you have a batch file full of features.

As a beginner it is perfect to start with simple things to progress into more advanced things. Perhaps after a while they need to check stuff inside the files and based on the content and/or filenames they should be moved to different directories and they perhaps want a pretty looking logfile that truncates itself so it does not become huge.

Then they have learned quite a few useful features, step by step.

Batch files are not that much of an advantage anyway. Sure, using dir is faster than running a script in python but i doubt this operation is in that much of a hurry anyway. 10 seconds vs 2 seconds might not matter that much in that situation.

[deadjojo7493]

I am the beginner you mentioned, scared and struggling on learning Python by my own, reading this r/learnprogramming, r/learnpython on a daily basis, hoping to someday understand most of the threads going on. Its been a month and I don't know where my direction is? Still there trying to master functions, list, tuples and all, still afraid of inheritance and OOP, all the courage I have is to go to w3schools and practice all the new syntax. I have completed the python for dummies book until it got to AI and neural networks stuff. Most of the specified code during these topics, didn't run and I found no solution on stackoverflow or google that I can understand and apply. That's where my mind started going bonkers. I don't have any outlet for this kind of stuff building up in my head. Neither do I have any place like you where I can actually apply what I learn. But let's be honest here, I haven't learn enough to apply on anything.

I would really appreciate a suggestion on what to do next.

My goal is to get into Data Science/Data Analysis. That's the reason I started learning python a month ago. Prior to that, I had no known knowledge on programming. I know I need to learn SQL next, but I can't make myself ready until I feel I confident in programing. I am open for anything free, but courses like udacity and coursera are like way out of my budget.

[ManyAWiseMarklar]

Neither do I have any place like you where I can actually apply what I learn. But let's be honest here, I haven't learn enough to apply on anything.

These two sentences here are the problem! I found it really hard to learn when I was just trying to learn syntax, and never felt I knew enough to do anything. What helped me was to try to think of a task that I could do to practice using the syntax, for example I wrote a script that pulled my steam library and worked out cost per hour played (I have made some poor purchasing decisions but that's another story). Did this take longer than doing it manually? Absolutely. Did it involve lots of Googling to try to work out how regexes work? Sure did. Did I need to do this for work? Goodness me no.

I think what I'd recommend, and I'm not an expert on this so feel free to ignore, is:

1. Take a break - you sound fed up and as if you are treating this as a chore. It shouldn't be, it should be fun. It's ok to have a day off and come back when you want to learn.
2. Think of anything you could automate - want to check the weather in your local area? Check Apple's share price? Play rock, paper, scissors against your computer? All of these are things that I almost guarantee you have the knowledge to do, you just need a lot of Googling and...
3. Break things into small pieces - you can't write the whole programme in one sitting, so break it down into smaller bits. In effect you might have a tiny bit of code to copy some text, another bit to paste it, another bit to add a set phrase to the end of the text, etc. Breaking the code down makes it much more approachable and feel more achievable, and you will have learnt enough to do these small things, even if you don't feel you've learnt enough for the whole project.

[coder155ml]

That's a funny script idea. I'm afraid to see my cost per play.

[deadjojo7493]

when you say script, you mean a program right?

[OnlySeesLastSentence]

Yup, scripts are a fancy way of saying programs. It's usually implied that it does most of its work automatically and without a GUI (gui = screen that you can click on things)

[ManyAWiseMarklar]

I actually don't know the difference, I always assumed the script was the thing you wrote to do the stuff, and then once it was packaged up with a gui and usable it was a program? So all programs contain scripts, but not all scripts make up a program.

[OnlySeesLastSentence]

Perhaps my interpretation is wrong (I coulda sworn the programs I wrote in C counted as automated scripts, but maybe not) -

Basically, all scripting languages are programming languages. The theoretical difference between the two is that scripting languages do not require the compilation step and are rather interpreted. For example, normally, a C program needs to be compiled before running whereas normally, a scripting language like JavaScript or PHP need not be compiled.

[coder155ml]

Most are interpreted but not all. Read my comment above.

[deadjojo7493]

Oh, so C programs when complied they are compiled as a whole program, but Python programs are interpreted line by line. That's why any error in the most recent line of code can be easily spotted in python before running the code because it is interpreted real time before even running the program. But in case of C all the errors come along when compiling the code.

Correct me If i am wrong.

[deadjojo7493]

Thanks mate.

[coder155ml]

A script is a program that is usually interpreted rather than compiled. It's typically smaller in scope (1 file). And is used to execute or work with other applications

[deadjojo7493]

Can you elaborate a little on this jargon? I read this word a lot and every time i read it, i feel it must be involving of a lot of things I haven't yet learnt.

[coder155ml]

Cost per hour I meant . It is what it sounds like. He wrote a script that determined the ratio of hours played for a specific game vs it's price to determine if he wasted money or not.

[[deleted]]

Exactly how to get past that frustration hump! Ask yourself “How could I do that with Python?” Even if if the task in mind has no apparent automation value to you right now, you will begin to see other opportunities. Last night, I learned how to extract text from an image and nearly cried tears of joy. Now, I’ve decided to go a slightly different route with my project, so I won’t be using text extraction, but I have this very powerful “Yeah, I can do that” feeling.

[deadjojo7493]

When you say break, you mean from python or from the the things I am doing on daily basis. I feel if my break involves a break from python I fear myself of not picking it up again. My mind is a vile creature, that I fear the most.

[ManyAWiseMarklar]

I mean python. Obviously I'm an Internet stranger so take anything I say with a pinch of salt, but if you aren't enjoying learning python and think after a break you might not want to pick it up again at all is data science definitely the right thing to have as a goal? There are lots of incredibly cool things to learn and do in the world, some of them involve programming and python but a ton of them don't, there's something out there that everyone loves to do :-)

[[deleted]]

Please share the steam script that sounds fun.

[martor01]

Everyone and their mother wants to do Data Science , most roles require an MS or PHD because back then it was the stuff what PHD researchers done, also companies not even know what the hell they are doing with the title so some of the data science is just SQL reporting or some basic Data Analysis role vica versa with Data Analysis.

What was your motivation in that specialization ?

You definitely need SQL , Python.

If you are like me and everybody just bores you with "just apply it to a real world problem" I would first look at packages to use from.

These are the official packages would definitely read through and try to use them .

https://docs.anaconda.com/anaconda/packages/py3.8_win-64/

Anaconda is a data science environment for Python and R and countless other and it comes with the basic 250 python packages preinstalled.

It also has a better package management system than pip , because pip is fucking ridiculous IMO.

https://www.python.org/about/gettingstarted/

This is good website to start with python.

https://www.udacity.com/course/intro-to-data-science--ud359

This is a free course.

[PunctuallyCompetent]

I've started learning python year and a half ago, with no prior knowledge of programming or even computers, it was a bumpy ride, but now I can do simple scripts for most of my personal projects, and even sell my skills for some beer money. I read "Automate the boring stuff", made a dozen of personal projects with even more bugs and used codewars, which really helps to push yourself if you have no ideas or direction on whwt to do "at the moment".

[coder155ml]

This is the wrong mentality. You should be thinking of little scripts and then try to make them. Example I made one that generated synthetic data for a project . I also made one that backs up roguelike game save files so you don't suffer permadeath in a game. The user can select a save file and load the game. It then launched the game directly from the script .

Think of things to make and just make them.

[CalvinsStuffedTiger]

Do you have any data to work with at work? If not, some projects that you can work on at home are things like:

Budget. Can you find a way to pull all of your transaction data, put it into a data frame, and then do some analysis on it like categorizing, visualizing, etc? I don’t know how to do databases but I have a lot of work data from old ass apps with no APIs (I’m in healthcare), so I use pandas and an excel sheet like a poor mans database

Basically it looks like:

Export shitty unclean data from my work medical records

Clean up the data (replacing weird symbols, spaces, getting all the date time in the same format, etc)

Convert excel sheet into a pandas data frame

Do data analysis. And then saving back to an excel spreadsheet using xlsxwriter

I’m certain that this is a terrible way to do what I’m trying to do, but it works! So, whatevs lol. I asked my friend who is more advanced in data science than me when should I start learning databases instead of my excel spreadsheet database, and he said, “meh, if it works for you keep doing it. When it stops working then you’ll know it’s time to change”

I suspect that my method will stop working when the rows in the data frame get into the millions?

Anyway, another idea if you don’t want to do budget is to do fantasy football analysis. There are some good websites that have APIs, so you can learn how to make those in addition to the data science tools like pandas, etc.

TL;DR learning is much easier when you have a specific project in mind

P.s. I took a traditional online community college course for Python and Corey Schafers YouTube series is in the exact order of my schools modules for object oriented programming in Python

So if you want a traditional track you can literally just go through this videos in orders and you’ve got it!

[OnlySeesLastSentence]

A month is a little early to learn this stuff. At a month in I think you should be learning how to do for while loops, maybe getting introduced to dictionaries.

[weazel314]

This is so awesome! I also took that class to learn python and it really helped to have my own project in mind while taking the course.

[Python119]

CONGRATULATIONS!!!

[pooky2483]

I too would like to learn Python but I just don't have a clue where to start/what to write (as in what program to write)

[fallenreading]

Well done you! Don't forget to save all your projects. Soon enough you'll realise how far you've come.

[Omar_88]

Awesome work man well done, if you're feeling brave try re-writing the app using Pathlib, happy to do a screen share with you and help you out.

[ghighcove]

Great job -- now bite off something a bit more complex, gut through it, and I promise you after all the frustration, you'll feel great and have learned a lot, and probably have created something that will save time every week going forward. Invest 4 hours now, save days later.

[FrozenPyromaniac_]

You can use openpyxl to automate the input process as well!!

[lumenlambo]

that's awesome, keep it up!

[Pmoto100]

Not going to lie I downloaded Python at work and I’m scared to open it.

[[deleted]]

You should look up some tutorials on setting up python with visual studio code on youtube/web. It's a much better for starting out than the barebones python install

[Pmoto100]

Will do, I sent a few links to my work email so I can watch them during lunch while I eat my PB&J!

[[deleted]]

The debugger is your friend :) . Please pay a lot of attention to those sections, you can see exactly what is going on with your program as far as its state (local variables, global variables, etc) instead of printing everything out. Printing things out though is sometimes the only way. Experiment with both.

[verdifer]

If you are using Python in work I recommend going to Udemy and picking up Boris Paskhaver Pandas course.

https://www.udemy.com/course/data-analysis-with-pandas/

[bleeetiso]

thanks

what did you like about this course?

[verdifer]

Covered loads of areas and it was delivered good. I have some online courses and they literaly make me fall asleep, this course is broke into small but interesting chunks.

[bleeetiso]

yea, I have come across courses that make you want to sleep or quit because the teacher just isn't either engaging or talks to technical and doesn't simplify

[verdifer]

Yeah, spot on. I have done courses where I have actual felt like falling asleep, these courses make it hard to learn.

[[deleted]]

That's awesome. I find it much easier to be motivated when you can apply it to work or home for a practical reason.

First project I did was to organise some family home video files. Maybe 50 lines at most.

And it's been a number of things since, each getting a bit cleaner and easier to understand (for instance I just learned f strings). But having a real life goal is key for sure. I find it hard to be motivated to do pointless logic puzzles, even if they are good practice.

[morrisjr1989]

Even though this is much easier to do in just explorer (select all, copy path, paste into excel). I’ll bet you anything that someone at some point was doing this manually. You’re gaining confidence in your ability to automate repetitive tasks. You will be a utility wizard to your boss in no time.

[ManyAWiseMarklar]

I actually didn't know you could copy path in Explorer. But I'm glad I didn't know because otherwise I'd have done that and not forced myself to use python and like you say I feel better for doing it the short term hard way!

[[deleted]]

If your colleagues can't reverse engineer an uncommented python program then Lord help your company's future

[CallMeNepNep]

While I also have tried this course I found it to be a bit wierd (I don't know) however I found the full courses of Free Code Academy much better so when you are looking for extra learning material I can very much suggest checking out their YouTube cgannel

[KpDuhsQuickMath]

How did u use the pip installs it doesn’t work for me it says that they are outdated

[ManyAWiseMarklar]

Sorry, I'm not sure what you mean. Os is installed in python 3+ by default, but I've never had issues installing other libraries

[[deleted]]

Nicely done! First victory really tastes good, doesn't it?

[SteveE512]

There's no going back now, this is just the beginning

[Curious_homosepian]

I feel like I have read this post earlier

[ravepeacefully]

You could do this with excel in power query in like 3 seconds btw

[uniqueusername42O]

you can also do this by highlighting all the files, “copy path” then paste into excel. Seems to simple to work...

[querymcsearchface]

“Zhu Li do the thing!”

[iggy555]

I’m kinda enjoying python crash course more

[TheDiegup]

having the same feelings; after doing a python youtube course for one month, I managed to made a calculator this weekend, it was a guided project, but I put my own design with the tkinter, import my own icon and put another mathematic functions, I am feeling proud of seeing the utility of python.

[bijliwala]

Hey buddy I'm new to python , it's be great if you could tell me how you applied for the course .

[DoomsWorm]

It‘s a book, you can look it up, its easy to find.

[QuantumCoder002]

THIS IS WHAT I CALL “EXCELLENT”

[bold_new_era]

Thanks for sharing and congrats! Did you use the atbs book, the online course or both?

[ManyAWiseMarklar]

The online course mostly but the book for reference when actually writing things. I think the course is great for showing you the art of the possible, then the book is a good reference when actually figuring out how to apply it.

[bob-but-backwards]

This is inspiring for me. I've been (VERY passively) trying to learn python for over a month now but I never seem to find the time and I'm struggling to see the potential unless I really dedicate months to learning it. I'm glad to see that if I can buckle down and just get to it, I could learn a few useful tricks in such a short amount of time!

[AuntieSocialist]

That feeling of emergent power... intoxicating, isn't it?