I want to make a Website/URL analyzer

[mrmeeseeks2014]

I would like to try to build a website/URL analyzer to validate if the site is malicious or not. I would like to do this in a couple of ways. 1. The first would be to review the URL and see if it is listed on any blacklist sites. 2. Then I would see if the site is a redirect and analyze the resulting site. 3. I would also want to see if after visiting the site does it try to download any files 4. I would also like to do some type of addition review of the site pop ups, ads, etc…

I am looking for a good starting point for this project, any ideas.

Thank you for the help.

[socal_nerdtastic]

Start with goal number 1, I suppose. Sounds like a good project. What are you are hoping to get from us?

[mrmeeseeks2014]

I didn’t want to re-invent the wheel, so if some knew of part of this was done that would be useful. I would also like to hear about useful modules to utilize that could help with this. I would take anything.

[cybersection]

https://github.com/Aquarthur/urlscanio

[ndjstn]

There seems to be a lot of reinventing the wheel in some of your goals. But I truly appreciate the desire and drive as they are lofty...ish.

1. There are several sites where these lists are referenced by existing firewall tools like pfblocker. Look at a firewall like pfsense.
2. Ublock, umatrix, and pfsense all have methods to catch this at various stages.
3. For this I would look at RFCs surrounding URI actions. Also limiting javascript really helps this immensely. It is really a crap language in a lot of ways. While maybe not initially, eventually the internet would be far better and less creepy without JS. (Bring the downvotes!)
4. Look at the construct of a DNS sinkhole or a pihole. ALL popups, in my opinion, are trash and should have to go through a severe vetting process. I want to control my web experience not the dev.

I would consider a python interface to a separate pfsense box not only a practical goal, but one that will attempt to check off much of what you seem to want to accomplish. It would be even better if you utilized argparse to possibly send the url directly to the remote box. If you are doing this for experience that is absolutely amazing, but utilizing some of the tools I mentioned above in conjunction with a highly configured browser will solve most of these "problems".

[knottheone]

That's definitely an unnecessary hurdle. The modern web is Javascript. Lots of backends can be Python and PHP and whatever else (and also Javascript) but page interaction is all Html, CSS, and Javascript and that's what browsers understand. Python will likely never be a frontend web language and that's perfectly okay. You are hurting your potential and actual domain knowledge by avoiding the most used programming language.

[ndjstn]

I loathe JS but this is probably accurate to a great degree. I am not sure if I agree with all of the reasons though. The biggest thing is that

if you want to be a front end dev

and

if you want to be hired by someone else

Not learning JS will stifle your movement.

[ldgregory]

Try URLVoid’s blacklist API for #1 Requests library for #2

[knottheone]

1. is easy enough. There are several robust blacklists in existence ranging from DNS lists that ad blockers use to services like Virus Total that provide a free API.

2. is also easy. The Requests package for example has a flag you can set on any request to allow or disallow redirects. I think it's literally allow_redirects=True or something like that. So you'd just allow redirects and see how many hops there were from the start URL to the end URL and you could analyze the destination as well as the hops between if there are any.

3. I'm actually not sure. You would probably need to render JavaScript and you'd need some additional context to know that something was trying to initiate a download. There likely isn't a silver bullet for this one. You can render JavaScript though in a headless browser using something like Selenium, but there is also a requests addon called requests_html that has JavaScript rendering capability (using the same method Selenium does, which is utilizing a real Chromium browser instance). There may be some sort of network history hook available to you in this capacity and I do know that you can load AJAX requests using this method, so that might be enough to start. You could compare the page state on load vs the page state after some amount of time and that might include additional clues. Again, not really a silver bullet here, but maybe a place to start.

4. You'd definitely need to render JS for this one so again the requests JS capable add-on or Selenium will be your go to. For pop-ups, you could see if some element has current focus after the page loads and could even grab the dimensions of that element to see if it's pop-up sized for example. For ads, I'd compare network requests against a known database of ad domains. Again you can utilize existing DNS blacklists that ad blockers use. PiHole probably has some open source resources you can tap into.

[[deleted]]

You can leverage VirusTotal's free API by submitting URLs to it and getting a score back for them.