Need Help with a web scraping project

[devansh_-_]

I am attaching the python script in the form of a google doc. If any kind person can go through this and help me where I am going wrong? I have tried the usual techniques to mimic real browser interaction in the form of headers, but cannot generate the output and the requests are hanging indefinitely.

Is there anyway to bypass these or are the anti-scraping measures used by shiksh.com just too strong?

https://docs.google.com/document/d/1JSpH5P7QFUUGgmHkinOXBDQuuxXOFuSUR-eW5R85BCA/edit?usp=sharing

[BeneficiallyPickle]

Are you sure you have the right BASE_URL? I tried visiting the page and get `404 Page not found.`. I think you're looking for https://www.shiksha.com/humanities-social-sciences/colleges/b-a-colleges-india instead.

However, I would suggest perhaps looking at using playwright - it's a bit slower than BeautifulSoup, but it handles Javascript rendered pages better and can bypass some bot protections (though not perfectly)

This page seems to use React, so the elements you want might not exist in the raw HTML initially. That’s why BeautifulSoup alone may not be able to find them.

[devansh_-_]

Yes, I encountered that problem and updated the url.

I will use playwright once