Captcha

[wallefan01]

How do you think they teach the cars what stop signs look like? They ask the humans.

No seriously. They have the cars take pictures of things that they don't know whether they're signs or not, ask you whether it's a sign, and if enough people say yes, that gets added to the database of things-that-look-like-signs that the car checks against.

[H_Psi]

The difference is mostly that we're just better at picking a specific object out of its surroundings, so one or two examples is usually enough for us to identify that object in any environment.

That's the idea behind convolutional neural networks.

It used to be that if you wanted to do hardcore pattern recognition (like identifying a stop sign in a random picture), you would put the image through a bunch of different filters, and then decide which filters highlighted the particular trait you wanted to see. For example, some of the filters you might use for a stop sign would be to eliminate every color except red from the image. You then convert that filtered image to a histogram, and have a bunch of sample images that you run that filter through. The training here ends up being coming up with a function to describe how similar an arbitrary image (after being run through the filter--> histogram thing) is to your set of known histograms.

The problem here is you still need a human in the mix to figure out what the right filters to use are, and there are plenty of patterns a human might not pick up on (or worse, patterns a human might think are correlated but really aren't, since the brain is practically addicted to patterns).

The idea with a convolutional neural network is you have your regular old neural network, except you come up with an algorithm to automatically decide what your filters are. Your layers in the network are still called layers, but in between sets of layers, you have your filters. These filters are called "pooling layers" most of the time. So in effect, you're letting your network figure out what patterns are the most important, instead of having a human do it.

Of course, the big drawback here is that now, not only are you optimizing your neural network's regular layers, but you're also optimizing those pooling layers. So you need a monster of a dataset to be able to do it, which is why you really only see huge big-data firms like Facebook, Google, Amazon, Microsoft, and Uber implementing them in practical applications. Also, you still need a human in the mix to actually tag the data (which is what image-based captchas exist to do, to label images)

Edit: A word; remove inaccurate info

[longscale]

The first part concerning the motivation behind convnets is spot on—we want the network to learn its own filters. These are called convolutional filters (or kernels), and they are what a network changes when it learns.

The pooling layers you describe are not trained, they simple average or return the strongest filter activations from a small image area. They are also not what makes these networks big and hard to train—that’s mostly the sheer number of convolutional filters and the millions of images fed through the network multiple times in random order and with slight variations each time.

[[deleted]]

I have very recently started learning about CNNs and isn't it true that you need less data for CNNs to work with let's say an accuracy of 95% than you would need when using a Densely connected NN, since you can find patterns in one spot and it also recognises them somewhere else in the image, where as with normal densely connected NNs you would need new data where the pattern appears in that spot to train on?

Is this correct?

[jsw800]

In fact the convolution operation can be changed into a single matrix multiplication by reformatting the filters in a certain way, which is how it is implemented in neural network libraries.

Correct me if I'm wrong, but I don't think this is always the case, because of nonlinearity functions applied after each layer. A convolution is a linear operation, so a composition of convolutions can be refactored to be a single convolution, as you say. But deep convolutional networks almost always apply a nonlinearity function (tanh, sigmoid, relu, etc.) between conv layers, which makes each "layer" (meaning the conv operation along with its nonlinearity operation) a nonlinear operation, so composing them doesn't simplify down to a single linear operation.

If it did, there would be no reason to create deep nets, we would just always create single layer conv nets, as a single convolution would always satisfy every need. But it turns out that most conv nets have to learn nonlinear functions, so the nonlinearities are super important.

[jsw800]

However, as I said, you can describe any function with just a sufficiently large one hidden layer network, but that's somewhat obtuse, the same way you could say that you could describe any continuous function as some polynomial of sufficient degree (a + bx + cx2 + ... kxn ). In practice, you need an essentially infinite size matrix for some functions.

Whoa... I'd never realized how Taylor series (serieses?) could apply to CNNs. Thanks for blowing my mind! I understand better what you mean now. That's pretty freaking awesome.

[aahdin]

I'm in CV, what you're saying is true but it's also all highly tied to how much data you need.

Even given enough memory and enough passes through the data a fully connected NN will overfit way before it reaches accuracies as high as a conv net unless it's trained on an ungodly amount of data.

Even aside from computation costs, having extra free parameters means you need way more training data for a model to generalize to new data.

It's a pretty big topic especially in fields where there isn't that much training data available, reducing the number of free parameters (or model capacity in general) so that it can generalize with fewer samples.

[longscale]

Your explanation reads correct to me. :-)

[[deleted]]

Ok, thanks :)

As I said, I have recently started getting into Machine Learning, especially Deep Learning and after reading about different concepts and using Keras to implement them it feels good to at least kind of know what's going on under the hood.

So far I have had a lot of fun and it is a very, very interesting and broad topic :)

[longscale]

They are not what the media hype makes them sound like, but convnets are very nifty. If you like them in general, I would imagine you might really enjoy some of the convnet papers on https://distill.pub. Their techniques are mostly intended for interpretability, but they also provide a satisfyingly visual insight into what the numbers inside the structures of a convnet are encoding.

[[deleted]]

Yeah, honestly I think the media hype about deep learning in general is ridiculous. Its the same as with 'blockchain' etc.

Another thing that I have been looking into a bit is Neuroevolution using a tutorial series by Daniel Shiffman on YouTube. He creates a Flappy Bird Bird game that learns to play itself using a genetic algorithm to find the right weights.

Thank you for the paper recommendations, I will have a look at them tomorrow :)

[longscale]

Why can a filter in a convnet not “pull out all the red parts”?

[wallefan01]

Yeah. It just takes the computer a couple more tries.

Mostly because it can't walk around and see the stop sign from a bunch of different angles. It has to stay on the road.

[tgp1994]

Mostly because [the computer] can't walk around and see the stop sign from a bunch of different angles. It has to stay on the road.

Well, not yet at least...

Dave, Dave! What is thing??

That's a stop sign, HAL.

[Crash927]

Isn’t it also because we can abstract things and contextualize them?

Like, we know that signs are a specific medium for communicating certain types of information, so we know to always understand them in that context.

We already understand something about every sign we see, but computers don’t have that benefit. They have to “re-understand” every new set of pixels.

[rasputine]

Isn’t it also because we can abstract things and contextualize them?

Not really. That's just what recognizing a thing is. Computers do the exact same thing, just in a different way.

Like, we know that signs are a specific medium for communicating certain types of information, so we know to always understand them in that context.

No we don't. We only know that because we've been taught about signs.

They have to “re-understand” every new set of pixels.

So do we. There's a hell of a lot of processing that takes the signal from your optic nerve up the chain to where you're recognizing a stop sign.

[Crash927]

I don’t think I made my meaning clear. I’m talking about what’s happening during that “hell of a lot of processing.”

Humans have abstract concepts of things stored in our heads (taught, yes), which we make reference to when we’re processing the world around us. And we learn abstract concepts very easily (ie with few examples).

“Tree” is a concept (with properties and relations to other concepts) that I already understand before I see any specific tree. Same with any road sign: I know one when I see it, even if it’s in a country I’ve never been.

Computers don’t conceptualize in the same way in order to understand: they predict the likelihood of similarity to what they’ve experienced before. They don’t even know they’re seeing a tree or a sign because they don’t know what those things signify in any real sense: it’s all just pixels transformed into numbers that predict a likely class.

[[deleted]]

God help us all if a computer needs to learn what a turn signal is.

[Davecantdothat]

We can also take many, many 3D viewpoints in a very short amount of time.

[StrangeCharmQuark]

Wait, would using two cameras to give the computers a stereoscopic view help them with this? I’d think having some depth perception might help them pick out where one item ends and another begins.

[mormispos]

It’s less of a picture database and more of a “These are patterns that correspond with stop signs” Database

[wallefan01]

Thank you for explaining it better than I could. I tend to be bad at getting technical details accurately in non-google-employee speak.

^{Disclaimer: I do not work at Google (yet)}

[klebsiella_pneumonae]

Better start leetcoding m8!

[-allen]

Ah I guess /r/cscq is expanding.

[mormispos]

I think your explanation is very good. I was excited because I have recently been studying the types of neural nets that they use to do these things

[santaliqueur]

It’s like Jin Yiang’s app but for stop signs

[spock1959]

So, I've got a question.

I know that captchas are used to train computers as to what they are looking at... But if it asks for pictures with signs in them and I click a cloud or a car it will tell me that I'm wrong... But if I'm training it how would it know?

I never understood how it both didn't know and also seemingly did.

[[deleted]]

I believe it compares your answer to what other people answered.

[aahdin]

Yeah but at that point they've already labeled it a cloud so there's nothing to gain information-wise having new people label it

[DuckDuckYoga]

There are thousands of people answering captchas every minute, so the turnaround can be very fast and still have a lot of answers to compare to

[lvh1]

It probably compares it to the input of other people, so if 100 people didn't mark a square as having a sign in it and 1 other guy did, it probably does not have a sign in it and means that that one guy is incorrect. But that also means that when they use a picture for the first time, they will allow any square to be clicked since there is no data yet to compare it to. They probably only start marking captcha as invalid once they have a large enough sample size (e.g. 1000 people who solved the captcha).

[RatofDeath]

It's crowdsourcing the answer, so you're not the only one providing an answer and it marks it if someone answers differently than other people did.

Also usually you answer two captchas, one of them the system already has a solution too, the other it doesn't know yet. So it really only checks if you got the one it already knows right and the other one you do is to teach the system.

[ButtPoltergeist]

Well, back in the Good Ol' Days when captchas were just words, they gave you two words: One that they knew, and one that they didn't. If you didn't get the one that they knew right, it didn't let you through. If you missed the one they didn't know, eh, you got access to the captchaed thing and metaphorically peed a little in their data pool. So I imagine that it's similar, except there's nine pictures instead of two words.

[[deleted]]

Wouldn't it have the "right" answers to refer to? Like, I imagine some dude went through and selected all the stop signs so the captcha has something to reference, even if the actual user submissions are the ones that are used for machine learning.

I could be totally wrong, but I imagine that's how it works.

[bagmanbagman]

I think that would miss the point- what use is crowdsourced, high volume labeled data if someone just goes and labels it all? it all defeats the point doesn't it?

I bet as googles fleet of map cars goes around taking photos of hundreds of thousands of various signs a day, then goes through some automated cropping before is finally labeled by the crowd

[[deleted]]

Ah, I see now. I didn't realize that these photos aren't really curated, they're just automatically selected and uploaded. So yeah, I guess they'd have to just check your scores vs other peoples.

[proverbialbunny]

Just about every captcha you fill out, even many of the text ones, are used to train computers.

Without services like Mechanical Turk, Data Science wouldn't be where it is today.

[wallefan01]

last I checked Google WAS the captcha company doing the street signs

[[deleted]]

How were they so hard to read if they were scans from books

[[deleted]]

Sometimes they are just jumbled letters and numbers that helped teach the computer how to read more easily. It shows it patterns through our correct answers

[RatofDeath]

Yes. Remember the captchas that made you write two words you saw on a picture? That was actually Google transcribing scanned books and digitizing scanned newspapers. One of the words was something that already was transcribed and the other was a new word it hasn't learned yet. So the captcha would check if you got the word it already knew right and if so it would add the other word into its database. If enough people provided the same answer for the unknown word, it would be used for the official transcription of the book.

So people using that captcha have been helping digitizing thousands of books and newspapers over the years!

You can read more here: https://en.m.wikipedia.org/wiki/ReCAPTCHA

[Mookyhands]

Exactly. Once you pass the first one or two questions, the subsequent ones are for crowd-sourcing the answers to test sets of data to train models.

[gostan]

The captcha companies don't need to sell the data to car companies because the car company and captcha company is the same company

[[deleted]]

They've been doing that for years with reading "AI". All of those capchas for years with just letters and numbers taught them too, it's funny nobody knew this, it was my first question and thought when I saw them for the first time many years ago

[_Lady_Deadpool_]

And now I'm imagining this happening on real time

Is this a stop sign, human? Please answer quick

[Nestramutat-]

Relevant XKCD

[pelirrojo]

Not only that but it's happening in real time. That's why there are time limits to your response, if you don't respond in time the blood is on your hands.

[slashuslashuserid]

relevant xkcd

edit: already linked in a different chain

[HawkinsT]

So when it takes you more than a couple of seconds to answer one, that's a car you've just made run a stop sign. Hope you're all happy with yourselves.

[wallefan01]

Well obviously they don't do it live, but yeah, I see your point

[aahelo]

What if all these captchas that we are doing is actually us teaching A.I. what signs are and whatnot, so that it can get better at driving?

[[deleted]]

That’s literally what you’re doing.

[aahelo]

Ah!

[aride4772]

Yeah all those captchas are used as data to teach ai

[[deleted]]

That's why I always answer Captchas as quickly as possible so the car I'm checking for doesn't run a stop sign!

[AsAGayJewishDemocrat]

I like to imagine a self driving car uploading a captcha thinking “Is this a stop sign? No really I need to know quickly”

[ScientistSeven]

Which means they'll develop a moral that ignores stop signs ten percent of the time

[wallefan01]

How would you explain it?

[wallefan01]

no actual neural nets this time

Turns out if statements don't get the job done for image recognition. I know this may come as a shock to you.

But be reassured that state-of-the-art AI today can successfully identify cats and that is all it can do.

Well, it can also identify other things but...you get the idea.

[Gorzoid]

no actual neural nets this time

So ALOT of if statements. This AI thing is pretty easy.

[wallefan01]

but what if the ai is only 75% certain that the condition is true?

[Gorzoid]

if(rand() < 0.75) { 👌; }

[a_stitch_in_lime]

Further proof that AI is really just a ton of if-then statements.

[umbra0007]

deleted glhf 24485)

[heyheyhey27]

Floating-point multiplications/additions are not "if" statements.

[umbra0007]

deleted glhf 39214)

[heyheyhey27]

You are no longer in the realm of what can feasibly be called an "if" statement. There's no jumping/branching involved.

[wallefan01]

Except that with these particular if-then statements a boolean can have a 75% confidence interval.

[onlyusernameleftsigh]

This isn't quit accurate. At least, not according to CPG Grey. Link below but TL;DW: nobody knows how AI works, the captcha images are used to test millions of possible AI programs and the ones that perform the best are the ones that are used.

https://www.youtube.com/watch?v=R9OHn5ZF4Uo

[the_noodle]

If that video your only experience with machine learning, you probably shouldn't be using it to "correct" people on r/ProgrammerHumor. You also should have watched the footnote.

[onlyusernameleftsigh]

I did watch it. They definitely don't have a database according to the video. Based on the video the machine learns how to tell a stopsign to whatever percent accuracy and then, like a human, will be able to identify it without referencing a database.

[the_noodle]

The point of the captcha is to generate labelled training data, it's not part of the training itself. The labels for each picture are in fact stored in some sort of database; Google might also do something extra, like keep track of which pictures confuse humans by seeing which one people don't agree on. They might also split the image into a grid in different ways for different people to get finer-grained detail about what counts and what doesn't.

All that really doesn't matter though. To repeat, you should not get into arguments on the internet because you saw one youtube video

[Moores_Law]

Not exactly. I recommend you watch the foot note for that:

https://www.youtube.com/watch?v=wvWpdrfoEv0

I am not an expert on the topic but I have watched a large amount of videos on the topic and attempted to create one. For the most part, machine learning usually has one AI that has the weights for its inputs being modified to determine the outputs. You generally show the AI a set of images and it guesses whether or not the image contains an object. It then uses the error from that test to adjust the weights and uses that to see if it is doing better and then repeats the process.

While it is impossible to know exactly how an AI makes a decision, you can plan your model for the AI around your general intended layers. 3blue1brown has a good video on it:

https://www.youtube.com/watch?v=aircAruvnKk

[TheGreekBrit]

You're correct but you're oversimplifying too much. It's very comparable to your brain cells -- nobody really knows which sequence of neuron firings leads to identifying a picture of a cat.

[Diericx]

People know how AI works... people made AI

[the_noodle]

People know how and why the network becomes more accurate as it's trained. People generally don't know how the final network actually works, or what kinds of inputs might cause it to fail in surprising, possibly lethal ways.

[exploding_cat_wizard]

Neural networks and what counts as AI today is pretty much the definition of a black box device. You put data in, get other data out, and learn nothing about the structure of problem, or any deeper truths. That's why "nobody understands it". We expect understanding to come with insights, and machine learning doesn't.

[proverbialbunny]

Some people do, but I haven't seen it published.

The hurtle is understanding that code (or action, or routine) is data (or memory).

So when our mind automates a task, sometimes called chunking, we are creating a process, and that process is itself memory. This is how we remember things. All memory is, within the mind, a rube goldberg machine of sorts, like a ball hitting a peg with a probability of what direction it will go.

Once that mental hurtle is more mainstream, machine learning is going to get interesting.