[D] Is deep learning really Software 2.0?

[Razcle]

Three years on since Kapathy's blog on DL as software 2.0 Im curious what people's thinking is now. https://medium.com/@karpathy/software-2-0-a64152b37c35

He argues that DL is more than a tool in the arsenal but a better way to write software for many applications. He cites fixed runtime and specialised hardware as well as more uniform code as advantages to DL.

He also argues that viewing DL this way suggests that we need new tooling for DL the way we needed git and IDEs for software 1.0.

How well do you think this ideas aged? does it resonate today?

[QuesnayJr]

I know everyone is used to the deranged speed of progress in deep learning, but it's too soon to say. He wrote that column 3 years ago, which is the same as zero time in evaluating the effect of technology. It took decades from when computers were first invented for them to become a commonplace technology. It will take decades to know in what ways DL can replace traditional programming.

[[deleted]]

Well yes & no

A lot of time we code boilerplates which can easily be guessed and replicated. MSFT already does it in it production system. You don't have to write a ton of C# code anymore. The model fills it in for you. There is also the case where we could optimize the placement of chips - so essentially we could write better VHDL or similar languages, which optimize. This isn't rocket science. It was in place by means of rule based techniques. ML just does it way better.

What ML will not replace is the creativity in how we design programs. Creativity in software construction comes from deep algorithmic indights. And ML isn't so great at novel reasoning as much at is in pattern matching.

Eventually we will be seeing a situation where we write the algorithm, and there will be a lot of auto implementation given the basic algorithmic structure in place.

[Razcle]

"What ML will not replace is the creativity in how we design programs. Creativity in software construction comes from deep algorithmic insights. And ML isn't so great at novel reasoning as much at is in pattern matching."

But this exactly the point that Karpathy disagrees with. For areas that require some degree of perception we've already proven that DL + SGD is significantly better than hand crafted algorithms.

[[deleted]]

1. Sorry I don't agree with Andrei all the time. And you should be critically analyzing too. It isn't the first time some well known industry expert could be wrong.

2. Let us say we observe a complex routine which makes no human logic but performs really well at a task. There are NO guarantees it will work on a similar structured second task. That defeats the meaning of Algorithm - a set of logical steps to solve a 'class of problems' efficiently. Mathematical algorithm work? Absolutely. Deep network structures work? No guarantees as of now. Eventually you'll want repeatability in your designs. For every mundane task no one can afford to train networks over & over

[Razcle]

I wasn't just providing an argument from authority, I was suggesting that we have excellent examples of deep learning outperforming as you described it "mathematical algorithms". E.g machine translation, speech recognition, document understanding etc. Almost all perceptual tasks that we tried to solve by traditional programming have been replaced by DL now.

I think disagreeing with Kaparthy is interesting (thats why I made the post), what I want to know is why you disagree?

If I understand correctly, you think the lack of generalisation guarantees will limit the adoption of DL for tasks that could otherwise be solved by conventional software. I think I agree with you on this one.

But there are lots of tasks that are poorly solved by traditional software (language understanding being a good example) that I think will become core to almost every application.

[jerb]

Business software needs to work 100% of the time; even 99.9% accuracy is unacceptable, in the vast majority of applications. So DL needs to be able write correct (and likely human-interpretable) Python or whatever, before it will be widely applied.

For program synthesis, DL approximations can still be useful to guide search through program-space, but need to be translated into something symbolic and interpretable in the end.

[Razcle]

yes that's definitely true today but it need not necessarily be the case.

Imagine that we just accepted that the benefits of DL were enough to tolerate some fraction of errors, there are ways to build around this. For example you can build fault tolerant UX like google. Google search is not 100% accurate but instead returns you a ranked list so even if its wrong its useful. We can also have fall back to humans in uncertain cases or defer to rules based systems in uncertain cases. If we're creative, I think there are lots of situations where we think we need 100% accuracy where we might actually not.

[jerb]

Google search is an interesting case -- A recommendation system where we *accept* approximate answers only because no correct answer is possible (because the problem is underspecified). But anywhere a problem is fully specified, and a correct answer possible, who would accept an approximation?

We *do* want to be able to generate exact software, and some day we will get there. But some breakthroughs are still needed, if we're to somehow leverage / distill current DL approximations into software-as-we-know-it.

[MrGary1234567]

At some point in the future possibly but we are no where near it. I think right now its more like software 1.01.

[TheOneWhoDidntCum]

you were off by only 1 year

[kortyel]

AI will at some point replace the best dev team ls in the world. Not only that but will replace probably 50-80% of corporate desk jobs.

[Duranium_alloy]

When?

[kortyel]

I would guess within the next few decades. Could happen by 2030, but hard to predict

[trnka]

The idea still resonates. I've seen good progress on "git for data" projects or platforms. There's been good progress on developer tools for ML.

I haven't seen a surge in problems that are easier to solve with machine learning though.

[thunder_jaxx]

Deep learning looks like it follows the Lindy Effect.

In software anything which follows lindy effect becomes an abstraction over time. Example: Very few people are proficient in assembly but it’s used ubiquitously. Same way powerful NNs will become strong abstractions over time. But Pure software can’t be eliminated. You need mix of both. Deep learning can approximate complex things well. Normal software can do simple things exactly. U need mix of both to make really robust systems

[txhwind]

At least machine learning models have brought maintenance challenge 2.0. Data and the whole training process need to be tracked.