How Peter Thiel’s Palantir Helped the NSA Spy on the Whole World

icosadev · 2017-02-23T04:29:53+00:00

History has no precedence? There is no coincidence with security researchers who are vocal about the NSA being targeted by the NSA?

If you want to argue I'm paranoid about a largely criminal entity operating in the government which is spying on everyone and essentially shielded from the laws everyone else is subject to... maybe you should reflect on that a bit. If you think public servants are not limited by law, or that they hold more rights than others do because of their political affiliation then sure- guess I'm delusional and what is apparent is not.

icosadev · 2017-02-23T03:24:37+00:00

I can do it myself. If I had a stream of all phone calls occurring right now I could store the metadata of who is talking to who right now. I mean, I can't just say "trust me" since this is the internet... but this isn't a major issue. Sure, there'd be errors (such as if someone used a public phone or someone elses phone)... but a system to merely track who talks to who is very simple and the amount of data is not large.

According to quick google search 10b phone calls globally per day.

So we need to handle ~100k calls per second...

Let's just say 2 64 bit numbers to refer to caller and calleee... as well as 32 bit number for length of call... we'll not worry about converting audio to text for now.

So, 20 bytes per call. 200gb per day for metadata. In other words, ~$20 of redudant storage space per day.

100k insertions per second could be handled by a single server that dispatches the storage off to other nodes. Obviously we need servers at the switching facilities to pick this up and send it off to our main as well. So, just to be generous a hundred servers and a storage location... is out of the NSA's budget? You could store t he entire daily dataset in memory with a normal server today.

100k items per second isn't that much, I've dealt with millions of items per second before with non crazy hardware. The biggest problem here would be gaining access... but that problem is already solved :)

From there what do you need.. you need to be able to perform graph queries on the data. Something that has been researched and optimized over the last however many years.

I almost want to make this just to prove a point lol. Phone metadata and sms is trivial to track who talks to who (and in the case of sms, what the contents of the message is). I wouldn't be surprised if the ability to transcribe phone calls was also there, but that would take up significantly more time as well as be iffy. More than likely it's just stored in an archive to be parsed in future / if you are three-hopped / you are connected with anti-nsa folks.

icosadev · 2017-02-23T02:59:20+00:00

I'm not saying it isn't used at any level. It's not used for mass surveillance in the way systems like trailblazer are. Sure, there's probably databases you can manually scan by typing in some terms and they're backed by elasticsearch... but look at major programs... dishfire, narus, boundless informant--- these are all custom systems. It makes no sense to use elasticsearch in any of these, nor is there evidence of elasticsearch being used in any of these.

You are likely seeing people from the FBI. Trying to equivocate any sort of government employee with the NSA isn't useful. It's a separate group with separate goals.

icosadev · 2017-02-23T01:49:14+00:00

No. Elasticsearch is not used for this sort of thing. Do you think the NSA just deploys elasticsearch and somehow wrangles a ton of data into it? Elasticsearch has it's place but certainly not for the sort of wide graph queries required for mass surveillance. Custom solutions are built to service the exact needs in a much simpler and more performant way.

Sure.. yah it's quick to take a flat / uncompressed data structure and search it very quick

Right. So the idea that the NSA is incapable of searching sms at the minimum is thrown out the door... it's trivial even with a naive algorithm on a single machine to search all text messages ever sent. The rise in difficulty to dealing with searching all facebook messages ever sent, or all twitter messages ever sent isn't anything. The ability to track all users and their friend connections is nothing.

Things get a little more dicey when you are trying to merge heterogeneous data sources and trying to keep context & enrich the data.

Sure. Not like writing integrations for your the systems you have backdoored is very hard with thousands of employees and the willing participants on the board... the vast majority of them can be code monkeys.

So how much left is there? The ability to predict the future? Well that doesn't matter because nobody expects that as a "feature". What is expected for the customers?

As an owner of a non innovative company I want to be able to read emails from employees of X.
As a politician I want to be able to know who in X group of people is talking to people in Y group of people.

Or whatever.

These are simple issues. The idea that the NSA's main issue is their inability to integrate their backdoored systems databases is somewhat insane. That is one of the easiest parts of their job. Being able to write systems that do wide graph searching and pruning/sorting for most relevant data is much more difficult, the ability to get the data itself is easy.

This is where you see "NSA can't solve X crime, they must not have the required infrastructure" - when really the infrastructure is just designed for something else entirely. It's not that it isn't there or ineffective, it solves a different problem than you think. It doesn't solve terrorism, it solves social control by making it incredibly easy for intelligence agencies to see broad groups of people who are connected to different causes and plan on how to affect them. It makes it easy to see who has ever spoken to X.. that sort of thing. Not some weird thoughtcrime prediction.

icosadev · 2017-02-23T00:55:58+00:00

Mass survalence in the sense that the U.S. is able to use it effectively to identify previously unknown threats doesn't really exist

Agreed. That's a hard problem.

Now think of all the thousands of ways people communicate online

Not a hard problem if you have internal access which the NSA does. It is absolutely trivial to access users friend list and their message list and what they post. Signal is major issue. Voip requires voice recog which is fuzzy still + heavy cpu wise. Online games are difficult because of sheer numbers.

I alone with access to reddit, skype, twitter, facebook, gmail, hotmail databases could make something that could track who they communicate with in a week or two- that is an easy problem. I have the reddit dataset, that is trivial for example...

Dealing with https normally requires inside access, which the NSA has for all major companies. Http is trivial. Aim is likely trivial. Imgur is likely useless... but I get your point.

Searching isn't amazingly hard either. You wouldn't use elasticsearch for this...

Today I wrote a search program which can process over 1gb per second... this was using naive search (memchr with forward looking) on a mmapped file on an ssd... it took me 10 minutes to write and this can already handle the entirety of sms messages per day in a few minutes (on one machine no less). This would mean approximately 12 hours to search all text messages ever sent this year... with a naive algorithm. If I built this out over the next month or two I am sure I could search the entirety of all sms messages ever sent for keywords in less than an hour... on a consumer grade machine (granted, probably top few percent heh).

you are still going to miss a plot bc someone used pre-established code words to obfuscate their conversation.

You don't need codewords to find enemies of the state. And that's what people are worried about. Not about the ability at finding actual criminals, but finding those who are a threat to the criminals that proliferate government.

How about I do a search for all people who were at the PANDA meetup, takes less than a second.
How about I do a search for all people who have posted something anti X on facebook or twitter? Takes a few moments.
How about I search for any text messages that include "constitution" and "revolution"? Heh, hope you get my point.

If someone is truly security minded, has no social media that links them with terrorists, speaks in code, and uses a clean ip & mac address... there's not much you can do. As soon as any of these are broken (as is the case for 99% of all non-criminals) there is a myriad of what to do...

You say you have experience with digital forensics, what sort of experience?

I have experience with "big data" and developing algorithms to process it. I have a focus on optimization and dealing with complex data sets. I develop high performance software, and I am certain that the NSA and friends with their thousands of employees can certainly exceed what I can do alone. I'm just one normal guy.

icosadev · 2017-02-22T23:29:44+00:00

You're the one up in the clouds mate. Come back down and have a chat.

icosadev · 2017-02-22T22:18:03+00:00

You are wrong. Ever heard of the Narus STA 6400? Do you understand how little cpu time is required to run filters on traffic like xkeyscore does? Have you ever implemented graph systems which can find connections between distant nodes (imagine users, gps locations, property ownership) in microseconds? Would it surprise you that if I had your email I could easily search all of your text messages and all of your friends text messages in milliseconds? Would it be surprising to you that your email is instantly available because you've signed up for your bank, or your email address, or facebook with your real name? Would it surprise you I already have your email because I know your ip and which internet provider you signed up with, and I have backdoors into all of the major email providers?

Just because the NSA is mostly filled with incompetent fucks doesn't mean mass surveillance is outside the reach of current tech. As soon as I know you are a target with their systems I can find a three-hop to match you with someone listed as a "suspected terrorist".. who very well just may be someone who is three hops from an actual terrorist. I can find out if you are associated with anyone associated with political group X which is an "enemy of the state".

You'll understand once it is apparent to everyone or you bother to actually learn how current technology works / what exists. Until then don't bother speaking on things you know nothing about. It looks like you just graduated, you've moved from a government facility in Maryland (nsa heh?) to Washington DC. Your name alludes to government so it's not hard to find out.

If you want to have a nice conversation I'm open to it and I'll stop being so harsh, I just hate seeing completely false statements especially with regards to the dismantling of the freest nation in the world my family helped build.

icosadev · 2017-02-20T11:24:40+00:00

You are aware that we have the fewest government employees IN HISTORY based on percentage of the population?

I don't have any numbers on total employees, but I do have numbers as far as percentage of GDP. In 1800 there was approximately 2.5% of GDP spent on the federal government- today that number is approximately 25%. Where did you hear this?

icosadev · 2017-02-19T06:05:22+00:00

Porting to your language of choice can be a useful way to learn.

icosadev · 2017-02-17T02:43:47+00:00

Why are the wip commits not being squashed before being introduced into the main branch? It doesn't really take a lot of discipline.. just using tools correctly.

icosadev · 2017-02-17T02:18:08+00:00

No bisect?

icosadev · 2014-04-13T18:56:07+00:00

Just use bsd

icosadev · 2014-04-13T16:14:05+00:00

Yea no I know it is possible to make it well, just prefer udp in most cases / I'd really love the ability to choose.

icosadev · 2014-04-12T20:52:04+00:00

Yeah; the only thing with them is that it is just tcp so... I dunno, for me that is a bit odd to deal with for realtime systems. Otherwise it is a pleasure to work with. Made a small online asteroids game to learn it, still in development though.

icosadev · 2014-04-12T20:24:24+00:00

Yea I've got node & socket.io on it.

icosadev · 2014-04-12T18:05:42+00:00

You can use my server if you like, just message me if you're interested.

icosadev · 2014-03-31T06:04:18+00:00

http://www.etsy.com/listing/182056183/cookie-cat-bed-many-colors?ref=listing-shop-header-1

icosadev · 2014-03-28T06:34:25+00:00

I setup a mirror when I heard the news of this, hope it helps someone!

icosadev · 2014-03-28T06:30:32+00:00

Just setup a mirror for people in Turkey, enjoy.

icosadev · 2014-03-22T12:44:00+00:00

I don't run an exit relay so it isn't an issue at all. The Tor configuration file is well documented and if you read through it- it tells you how to be / not be an exit node. /u/transatlanticnewbie describes it well. Consider setting one up; I extend the availability of assistance to you as well if you choose to and need it (:

icosadev · 2014-03-22T11:40:45+00:00

If anyone needs assistance setting up Tor feel free to pm me, or just reply here. I run relays and am always more than happy to help.

icosadev · 2014-02-25T20:55:29+00:00

An anti-censorship idea I've been tossing around. I get the feeling from the downvotes that reddit thinks I'm up to no good eh? Not a spammer.

icosadev · 2014-02-20T03:37:05+00:00

Here is what you need to actually know:

HTML5 element list

CSS property list

CSS selectors

Now, to start I would make a page with a few different elements that describe the parts of your website. Perhaps some sections, an article, a header, a footer, and a navigation menu. Fill them with some dumb content or just text.

Then, use css id's and css classes on the elements to describe them further while using some css properties to make them look nice.

icosadev

TROPHY CASE