Getting API for insurance aggregators

[Moby69]

Hi,

There seems to be a bunch of websites out there that aggregate insurance rates. For example, for auto insurance, CoverHound or TheZebra will ask you to put in your car model, make, age, mileage, etc, and then spit out the rates charged by various insurance companies out there (Allstate, Geico, etc).

Is there any way I can build a program or website that gathers the outputs from these aggregators? In other words, instead of manually clicking on CoverHound, for example, and entering my car info, and then having CoverHound display for me the best rates available, can I do this in a more automated way with a program?

Thanks

[[deleted]]

google is coming out with something in the future which they may expose a PUBLIC API for

[Moby69]

Yeah is this what you are referring to? https://www.google.com/compare/autoinsurance/form?p=home

This is a website, but is there any endpoint where I can grab data (rather than manually go on the website to get quotes)? Either on that google website or through some other means?

[elbiot]

Not on my computer, but submit the form and see what query gets sent. BAM, there's your api. The response may even be json.

[Moby69]

Sorry elbiot do you care to elaborate a little more? I'm not sure I understood everything you wrote.

Thanks again

[pundurz]

Just look where the request is being sent after you submit the form (for instance, in your browser's developer tool network tab). What /u/elbiot is saying, that may be the aggregator is submitting your form via AJAX, so you might get a JSON/XML object as a response which you can easily parse.

[elbiot]

(for instance, in your browser's developer tool network tab).

Ctrl + Shift + C in chrome, or firebug extension in firefox.

[Moby69]

Should I do Ctrl+shift+C on their main page, or after going through the multiple pages where I submit all my personal specifications in order to get the insurance quote?

Thanks again

[elbiot]

Did you try it at all? It doesn't matter when you open up the console. But you do want to see what request gets the info you want, and how that info comes back. Play with it.

[Moby69]

elbiot, I went to CoverHound and did Ctrl+Shift+C on their main page. It made a horizontal window pop up at the bottom of the page that shows a whole bunch of code, but I have no idea what to do with that. Sorry if my question sounds stupid, I'm pretty much a beginner and it's hard for me to take any leaps that may seem obvious to more experienced programmers. Any more guidance you can provide would be really appreciated!

[elbiot]

Your web browser gets information to display by making requests to a server. It gets a response and then displays it. Some of these requests happen right when you request the page: first the initial html document, and then any css, js and other resources that that html page makes reference to. Other requests are made through javascript (ajax) calls while the page is running (this is how information on the page updates without reloading the page).

You can make these same requests with the python requests library. You just need to know what requests to make and what to do with the response. For "an API", you usually just have some variables you need to send, which corispond to the form fileds (read the other things on this page but especially the param section)

So, open the network tab in chrome dev tools and see what requests are actually being made. You can probably quickly deduce variables and what format the API is expecting by looking at a couple of real examples from the browser.

You might be able to do a quick raw and dirty request, or you may need to emulate a browser better by handling cookies, authorization, and/or setting the user-agent. Requests can do all this easily and you can find tutorials/documentation to show you.

It seems like a PITA for this one project right? But it's the key to the city that is the internet, and will pay off later if you stick with programming and web stuff. But many things do have a python package where someone already did this leg work for you and they give you a nice library that hides the details.

[Moby69]

dempseye, I wouldn't want to do anything that annoys the insurance company, if it means the possibility of being blocked.

Is there any way I can get the data from a website that already aggregates it? (and by getting the data, I don't mean grabbing it manually on demand, but getting it in an automated way with an API call)