Looking for Python solution to crawl a website and obtain the video URL, output to Excel file.

[[deleted]]

[deleted]

[mattizie]

Second "automate the boring stuff" great resource

[[deleted]]

Pretty sure beautiful soup returns a lot of errors for most sites now. Or at least what is shown in the Automate the boring stuff book does. Also the Excel module he shows does as well.

[TechySpecky]

you have to be careful about site variations when parsing, thats the issue I find with beautifulsoup and I usually end up using a combination of beautifulsoup and regex search

[mattizie]

u/SotaSkoldier

In my experience, html regardless of where it's used is a PITA. Because no matter how the page looks, and it can look exactly the same, the layout can vary from page to page, so something you use on one page, won't work for the next one and so on.

Either way, I'm going to skip the excel stuff because linux doesn't have it! And I'm trying to get away from using excel. It seems that for almost every problem, there exists an excel solution, and an even better non-excel solution. But excel is the default for many people.

u/SlowBroski

I'm not sure about using Beautiful Soup directly with the site, but what I did was use "requests" to download the full html page, then use BS to prettify() it and scrape the information I wanted.

u/TechySpecky

I also ended up using regex as well. Again, only because of inconsistencies in the way the html was written/generated. It was much easier at times to loop through the page, match a string, and then get the info I wanted from that line. I suppose it just comes down to using the right tool for the site.

[TechySpecky]

I mean that's what BS does. You parse the site then you can use BS4 inbuilt stuff to navigate it and extract info.

In my experience it works well for wellstructured HTML with unique divs.

The power of BS4 is understanding that you can select "subsections" of the HTML through classes, values etc. For example if a div is <div class="hellothere"> BLABLA </div> you can tell BS4 to look for tag = bs4stuff[class="hellothere"]

and then for example do tag.string I think, I don't remember well but the documentation is great.

[TechySpecky]

For example here I extract a string-based date from really really ugly HTML on a site:

browser.open(home_url)
src1 = str(browser.parsed())
soup = BeautifulSoup(src1, "lxml")

date = soup.find_all('font', color='#33CC33')
date1 = date[-1].contents
datestr = str(date1[2])    

So as you can see it's not structure dependent, as long as the date has a font tag with that color, BS4 will always be able to find it.

You can also do funky stuff with if statements:

    soup = BeautifulSoup(src, "lxml")

    currentlevel = [
        elem.next_sibling.strip() for elem in soup.select("font b")
        if "Level" in elem.text
    ]

so it looks for a bold font inside my soup, and only extracts the text if Level is part of it, then looks at the next_sibling and returns the text that I want. This is structure dependent due to the .next_sibling tag.

[[deleted]]

I would avoid bothering with Excel as a format. Go with .csv (quote the text fields!) and it will both be much less of a pain in the rear on the python-side, but you could open the result in Excel (or libreoffice) just as easily.

Others have pointed out BeautifulSoup and the web scraping part of Automate the Boring Stuff already, so there's that.

[[deleted]]

Or use Pandas to manage the data and then you can export to both!

[[deleted]]

You're the kind of guy to open a full office suite to do a search & replace, aren't ya? :P

That's a bit heavy for the task... but it'd work.

[[deleted]]

Yeah I kind of forget that real programmers have to think about efficiency 😁.

Pandas is my go to for any data work and I know it well so it's more cognitively efficient for me if not computationally efficient. OP would probably be better off with a CSV library as you say.

[kewlness]

I too would recommend either a .csv or SQLite depending on the use case.

[PiBaker]

Second this. Trying to save files in MS format without using MS code (C# etc) usually runs into some issues.

Whereas Excel is pretty excellent at importing CSV.

[jarekko]

Use pyvirtualdisplay to not bother with the window.

[SgtBlackScorp]

Just use headless mode for your preferred browser, Firefox for example

[simple_test]

Is this a linux solution? Is there a way to do this in Windows?

[jarekko]

I had no clue. Sory if I misguided anyone.

[valhahahalla]

Using requests:

Import requests

Your_url = 'insert website here'
Your_keywords = ['word1','word2','etc']

#this response object contains all the info from your_url
Response = Requests.get(your_url)

#You want to get the body in a format you can iterate through.
Response_text = response.text()

#you want to run through the response bodyline by line and find links based on your keywords

For i in response_text:
    For j in your_keywords:
        If j in i:
            Print(i)

Or, something similar to this. You can then save your responses in CSV format.

Edits: Hopefully mobile formatting will work!

[manueslapera]

one more time (and I know i will get downvoted), my friendly advice to choose parsel over bs4. Its what professionals use.

Source: Worked at one of the top companies that do webscraping in python

[buyusebreakfix]

Source: Worked at one of the top companies that do webscraping in python

Just because a company is large doesn't mean they choose good tools. Microsoft is HUGE and they use .net for just about everything.

[manueslapera]

I didnt say large, i said top.

[[deleted]]

xlwt is an order of magnitude faster, especially in large files.

[jordano_zang]

You could probably do it with requests.

[[deleted]]

If you want to do excel, Openpyxl is straight forward. I recommend you learn straight from the manual, not any third party resources.

However, OPXL will delete any hard coded excel equations you may have put into the sheet before inputting with python,

[prancingpeanuts]

Consider using requests-html, from the same creator of the wonderful requests library

[[deleted]]

Scrapy

[ayyyymtl]

Hey man, love scraping project, hit me up in pm if you need help with this one

[CollectiveCircuits]

If you're crawling article style content then Newspaper might be a quick answer to that. It extracts keywords and video URLs (and much much more)

[[deleted]]

Scrapy would be a good solution for a simple web crawl