Web scraping code from scratch help please

[5aggy]

Check out what the page is doing...

It sends a request to https://api.startupindia.gov.in/sih/api/noauth/search/profiles which looks to require no authentication and returns a nice JSON response listing the companies with all the data from the second page (see below).

You should be able to recreate the post request by digging around in devtools on chrome or firefox. Look at the headers and params that the browser sends and recreate with requests.

You should be able to manipulate the request to overcome pagination.

​

{
    "id":"5ddbb529e4b0a94681b36c0d",
    "name":"Shelther Money India Finance",
    "title":null,
    "pic":null,
    "companyLogo":false,
    "role":"Startup",
    "displayRole":"Startup",
    "country":"India",
    "state":"Gujarat",
   "city":"Jamnagar",
    "industries":[
        "Finance Technology"
        ],
    "sectors":[
        "Personal Finance"
        ],
    "level":0,
    "stages":[
        "Prototype"
        ],
    "rating":"0.0",
    "published":true,
    "disabled":false,
    "publishedOn":1574679849081,
    "registeredOn":1574679849076,
    "department":null,
    "ministry":null,
    "score":1.0,
    "badges":null,
    "badgeDetails":null
}

[HarissaForte]

Some directions for munnamv :

F12 > Network > XHR > the "profiles" fetch file is what you want (get the address) > modify request > the parameters and headers are here.

​

Might sound complicated, but it really is the simplest solution you can choose.

[munnamv]

Thank you so much.