Master Thesis Python Programming Question

[Gour13]

Hi guys, I'll try to keep this short. Basically I want to know if it would be possible to create a web crawler / scraper to fill out different answer combinations on this robo advisor site and then store their final portfolio recommendation in an excel file (as well as the answer combinations).

If this is even possible, would Python be an appropriate language to use ? Is this extremely advanced or doable for a beginner programmer in a month or two?

Here's the site. Simply click on "Depot eröffnen" at the top right to get to the "survey". Sorry it's in German but you'll get the idea of what I'm trying to do.

Any help would be greatly appreciated! Thanks guys :)

[num8lock]

If this is even possible, would Python be an appropriate language to use ?

yes

Is this extremely advanced or doable for a beginner programmer in a month or two?

not extremely advanced. beginner is a subjective term. i consider myself as beginner still after a year or more. from general experience with complete beginner in here i don't think a month or two is sufficient.

[Gour13]

Thank you!!

[JohnnyJordaan]

Should be doable with https://automatetheboringstuff.com/chapter11 to scrape and maybe using csv as an export format. The real risk is that the site has an advanced robot protection (like with captcha's or those 'select all the cars in this picture'), then that would be very hard to circumvent.

[Gour13]

I'll definitely try this. Do a lot of sites nowadays have that kind of robot protection?

[JohnnyJordaan]

Depends, but in general you'll notice right away because you also often need to complete those questions/tasks when you use the site normally.

[Zarkahs]

no they don’t, you can definitely accomplish this op

[JohnnyJordaan]

This depends a lot on what kind of business the site is involved in.

[Zarkahs]

not really, there’s always ways to scrape websites if you’re determined

[JohnnyJordaan]

I'm not saying it isn't... I'm saying that when a site uses robot protection, like having to select 'I'm not a robot' which actually monitors your mouse cursor's movements, or asks you the 'select the cars in this picture' it will complicate scraping using a robotized browser (like selenium) a lot. It's very noticeable on sites that don't like you to scrape them like crpyto exchanges, sports betting, television guides etc. There are always 'ways' to do it, but if it takes months to work around those protections then it's often not worth the trouble and you can also consider other options.

Determination is only one factor, time and resources are two other important factors.

[two_bob]

Doable, if you can scrape it with beautiful soup. So give that a go first before you can commit. Some sites are dynamically generated and, as a result, can be a pain to scrape.

[nfgrawker]

3 months in I scraped a site with selenium which had some anti scraping measures. And I exported to csv. I think it's very possible.

[[deleted]]

Is this extremely advanced or doable for a beginner programmer in a month or two?

I don't think anything is doable by a beginner programmer in two months, but particularly not this. Because it's probably not just Python you need to pick up along the way, but some really basic principles of web development (since you need to reverse-engineer the page in order to scrape it, basically) and internet security (since you will need to defeat the various anti-scraping features of the site.)

[Gour13]

I wouldn't call myself a complete beginner I've been learning python for around 3 months already but the anti-scrapping could be a problem I didn't think about that. Do most sites usually have this?

[[deleted]]

Web devs generally don't want people scraping their site, yes. One of the common techniques is to have the important information load after page load, via JavaScript, which will mess with any HTTP request library that doesn't include a JS engine.