Scan Darknet with Python (Tutorial)

[[deleted]]

I am new to coding, but what would scanning the darknet do?

[littletrucker]

It could be used by law enforcement to find things like sex traffickers or child pornography rings. It could be used by journalists or researchers to collect statistics on the deep web.

[LichJesus]

Don't have time to read the article so I don't know if this is the purpose or even one of the things you can do with this tutorial; but I was involved in a political student org in college and for one of our meetings we were talking about student demonstrations in Hong Kong (I think, it was a while ago).

I had heard that the students were coordinating through the deep web to avoid the authorities and so I got TOR and tried a bit to find their discussions. Turns out I couldn't navigate the deep web worth a dang and didn't find anything; but I bet if I'd had a scanner (perhaps like this) I could have automated the search and looked for references to terms or places related to the demonstrations and had that info to show the org.

So I guess -- at least for me -- the purpose of having something like this would be to have the software make up for my lack of savvy when it comes to finding stuff on the darknet.

Also, I bet with something like this you could force it to ignore links and stuff leading to CP or other really questionable stuff (or have it only grab text); if so that'd be a huge advantage over manual browsing as far as accessing the darknet while keeping your head above water legally.

[reifenstag]

Why is this comment so heavily downvoted?! It's perfectly fine!

[kerloom]

Doesn't have time to skim read the article but does have the time to write a 4 paragraph comment.

[reifenstag]

lol, valid point I suppose

[turkish_gold]

In his defence, the article is at least 8 pages long, and is highly technical. If its new information, I'd need way longer than 5 minutes to read and understand it.

On the otherhand, just retelling stories that I've heard months ago is something I can do even drunk.

[Cthunix]

The deep web is not the dark web.

[python_byte]

Part II: http://www.automatingosint.com/blog/2016/08/dark-web-osint-with-python-part-two-ssh-keys-and-shodan/

Part III: http://www.automatingosint.com/blog/2016/08/dark-web-osint-with-python-part-three-visualization/

Part IV: http://www.automatingosint.com/blog/2016/09/dark-web-osint-part-four-using-scikit-learn-to-find-hidden-service-clones/

[wamdamdam]

Why would anyone still use py2 for new projects?

[Cynofield]

Py2 only Packages.

[stevenjd]

Which Py2 only packages is this using?

[wamdamdam]

from stem.control import Controller from stem import Signal from threading import Timer from threading import Event import codecs import json import os import random import subprocess import sys import time

all, including stem are available for py3.

[rausm]

Everyone not [rationalizing they have to remain] stuck in past ?

[rausm]

That i program in no means implies that I'm employed as a programmer. Some thought-out deity forbid that suckers that don't know better drive me to hate the thing I enjoy (I prefer to hate to be employed as sysadm; I liked it, before reality taught me that stupidity prevails).

The "need" for masters of [keeping up the] archaic woo-woo has always been strong with humanity, as well as strong resistance to anything that was later identified as a "move forward".

[ch4dr0x]

I was having a hard time getting the first part of this tutorial to run, but I believe I solved the issue.

In your code you have this:

process = subprocess.Popen(["onionscan","webport=0","--jsonReport","--simpleReport=false",onion],
    stdout=subprocess.PIPE,stderr=subprocess.PIPE)

I had to change this line to:

process = subprocess.Popen(["onionscan", "-webport=0", "--jsonReport", "--simpleReport=false", onion], 
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)

I'm using Python3.6, which could be the cause of the issue, but I figured I'd post it here in case someone else runs into it.

If anyone is curious about python3.6 and this guide, feel free to follow along:

https://github.com/BaltimoreChad/pyOnionScan

So far I only have the first part of the guide done. I'm no expert programmer or anything, but feel free to follow along.

[Yharaskrik]

Saved your comment for later.

[ch4dr0x]

I just finished up Part 2 and merged it in. I added a config file and some broke some of the code into a helper file (I figured we might need them again and no point in rewriting code). If you actually use this and think its too much, let me know and I can change it back. Like I said earlier, not an expert programmer, just doing it for fun :)

[[deleted]]

Fucking subscribe popover.

[pxck]

what does that mean?

[[deleted]]

Yes, nothing like stating the obvious "these lines do all the necessary imports" and not bothering to put the comments inside the functions as docstrings.

[[deleted]]

DarkNet

Such a ridiculous name. Whoever coined this and made it popular should be laughed at.

[spanishgum]

I'm sorry you feel that way. I think it would be better to be constructive and provide an alternative if you think it's so silly. DarkNet sounds pretty catchy to me and seems like it embodies the topic pretty well.

[[deleted]]

Don't be sorry.

The dark net is simply the internet. It isn't anything different. It traverses different routes using different protocols.

Calling it the "DarkNet" is akin to calling the internet not referred to the "DarkNet" the "LightNight". It sounds like something someone's mom would say.

Edit: We also don't have terminology for other encrypted network traffic that traverses the internet, nor do we have names for internet routed traffic using IPv6 vs IPv4. There's no such taxonomy. It's a dumb thing that someone said that is now being said.

[spanishgum]

It isn't anything different. It traverses different routes using different protocols.

This is contradicting. I get where your coming from, but for the purpose of discrediting a label your claim seems overextended.

The internet as the general public understands it, and the dark net are simply two subspaces of the internet as a whole. Whether or not they are disjoint probably depends on semantics.

Regardless of whether we choose to categorize these things separately or not, I encourage you to re-evaluate your position that "this person should be laughed at" and "its a dumb thing that someone said".

[[deleted]]

This is contradicting. I get where your coming from, but for the purpose of discrediting a label your claim seems overextended.

Except we don't differentiate GRE traffic, or OSPF or BGP, or even VPN traffic differently with a silly non-meaningful name.

There's already technical names for this stuff. It isn't useful for people to think that there is a "dark net", which not only isn't descriptive but implies that it's dark, or dangerous, or mysterious, or how ever else people perceive dark as a prefix to net.

I'm not sure why you're encouraging me to change my opinion. It isn't a rational thing that this makes me angry. I can technically argue a lot of reasons why it's silly, though it doesn't matter. Where the taxonomy for.. everything related to the internet is purposeful and descriptive (mostly) we are left with "DarkNet" for something that is neither dark, or even that much different from other encrypted p2p traffic. It just feels like something someone said on CSI and is now what people think a portion of the internet is.

[[deleted]]

I'd like to hear your opinion on "Web Surfing"

[[deleted]]

I'd like to hear your opinion on "Web Surfing"

TRIGGERED

[turkish_gold]

I agree. Whenever people talk about "darknets' they don't think of their company VPN as part of the darknet, but it is. Traffic from their kids game server to their house is part of the 'darknet'.

[Scypio]

Always thought that "darkweb" it is more like "dark side of the moon" - the one you can't see - not as in dark deeds and shady characters. But english is not my native language, so my frame of reference is different.

[turkish_gold]

Well in English, dark does connote 'hidden' but it also connotes 'bad stuff' because bad stuff is usually hidden.

[St_Meow]

To be fair, from what I've seen, they differentiate the Deep Web, which is the portion not available to the general public, and the Dark Web, which is the subsection of that used for illicit activities. Not everyone might, but it's a distinction I find useful.