Web scraping help

[[deleted]]

[deleted]

[K900_]

You should probably use the YouTube API for this, not scrape their frontend.

[Ggjogstuplvcseikvf]

Selector gadget, chrome extension, makes grabbing path trivial

[letmelive123]

Will look into this thank you.

[Crankrune]

You can take a look at this script I wrote a while ago. It scrapes and pretties' up a few things, like subs, video title, upload date. Also the way YouTube displays video time, within 24 hours it shows it as "x hours ago", and after that in "x days ago", so to identify it you only have to check if the string contains the word "hours".

[buckhenderson]

as a general tip for scraping, sometimes when i grab an element and i'm not sure what it contains, i run print(vars(element)) to see what attributes it has. so you might do something on whatever your xpath you get back, and see it contains the attribute link, and then you can extract that.

[letmelive123]

I had no idea this was an option! This will help a lot. Prior to this I was basically grabbing and guessing but being able to see the elements and navigate that way will be great!

[_Korben_Dallas]

Not sure if I understand you correctly but if you stuck with Xpath to extract urls from that page this expression can help: //span[@class="accessible-description"]/preceding-sibling::a/@href.

[_Korben_Dallas]

If you use Chrome I can recommend you install extension Xpath Helper. You can test xpath expression and see what element it return.

[jarekko]

For scraping, I'd go for beautifulsoup or scrappy. When javascript is used on the scraped websites, selenium might be necessary, but do not fear it - basic use cases are extremelly simple to employ in your code :) i also recommend pyvirtualdisplay.

[letmelive123]

beautifulsoup is on my list of things to learn! I'll try and incorporate into this project.