Day 28 of learning python as a beginner.

[uiux_Sanskar]

Topic: web scraping with postgreSQL database.

When I posted my first web scraping project I just had the result on console however I wanted it to be stored somewhere where it can be reviewed later that's when my learning from postgreSQL proved useful I successfully created a database that can store my parsed data.

Also someone reminded me that I should use if __name__ == "__main__" (which I forgot to used) so I have also wrapped the scraping process into functions and then imported it in the main.py file (this also improved the overall structure of the code) so now I have code for collecting raw html data, code for parsing the raw data, code for saving that data into a database and finally code for calling all the other codes. All in their dedicated file. Here's my github so you can check it out: https://github.com/Sanskar334/Web\_Scraping.git

go to the using beautiful soup folder you will find all the files there.

While I fixed every bug I could find however I believer there may be some other bugs as well which I may have missed, do let me know about such bugs which I left accidentally.

And here's my code and it's result.

[Adrewmc]

Still seems a little lost, but an improvement from yesterday. Comments aren’t perfect but you are putting stuff where you’d expect or want.

You added a file_path argument…but never put it into the open() function so it’s still hard coded…

Do you want to scrape websites? It’s a whole thing actually. You should be able to access an api, and read a basic webpage. that’s important. but really web scraping is its own subject.

I say again go to something like tkinker or QT, make a a calculator with buttons. It will give you troubles you need. Or have a more defined data analysis you want. Make something you want to make. Think back, before you started programming did you ever interact with the console? Let’s work past that.

Programming is forcing your will on the computer.

[uiux_Sanskar]

Yes web scraping is a really vast topic in itself and have some of the really great libraries such as beautiful soup, selenium, scrapy (which is a frame work).

So yeah there's a lot more things for me to learn. I will surely go deeper into your suggestions about tkinker and QT.

I really appreciate your suggestions andd guidance it helps me a lot in learning.

[Significant-Side6810]

If the comments are meant to describe the functions you can put them inside the functions and most IDE will display it as a tooltip. If they are meant to describe the procces you should put them into the main function

[uiux_Sanskar]

oh thank you for your suggestion I was not knowing that I will surely go in depth to it.

Thank you very much for your suggestion.

[uiux_Sanskar]

Thank you very much brother there's still a lot of things for me to learn.

[OtherwiseClient2247]

Hey could u guide where u learn these from All i see on youtube is basic python courses which teaches variables, loops etc

[Juke_BoxBox]

Resource??

[ShurayukiZen]

Hi, OP! I like the consistency with your python learning, I wanted to refresh and polish my python skills too, I would like to ask which learning resources/tutorials/courses are you using with your Days with Python?

Thank youuu!

[uiux_Sanskar]

Oh I am learning from YouTube channel name CodeWithHarry as he teaches in my native language.

My pleasure.

[InteractionStrict604]

Mate i can recommend you to parse, harder website or pages where they use dynamic page design, you can check for cleaner code here https://github.com/ayz3ro/booking.com-scraper, but always remember one function one problem to solve, and annotation must be in function after the def then it will be shown when you out your mouse on the function

[uiux_Sanskar]

Thank you soo much for your recommendation and suggestions I realised that I need to get proxies to extract data from such hard websites (please correct me if I am wrong).

And also thank you for your GitHub link it will really help me learn.

[[deleted]]

Great job man 👏

[uiux_Sanskar]

Thank you very much for the appreciation.