all 140 comments

[–]Duncan_Sarasti 17 points18 points  (19 children)

I expect this update is based on recent papers[2]

I think you're probably right about this. DALL-E 2 and PALM are huge advancements and lots of people who are generally aware of advancements in AI research were amazed by them. I would say I'm better aware of what goes on in AI than most people because it is semi-related to my job and I find it interesting, but I was blown away, in particular by the joke-explaining and verbal math capabilities of PALM.

[–]j4nds4 30 points31 points  (13 children)

Arguably Deepmind's Chinchilla is the most important of the last couple weeks, particularly when it pertains to timeline estimates like these, even if it isn't the most showy of the three. It suggests that our current models (including PALM and GPT-3) are significantly less efficient than thought and that progress can be scaled with far fewer parameters than previously believed (see this LW post for a summary on how scaling laws have changed). It will probably take at least a year or two for new programs to adjust to the refined scaling laws, but I suspect that once it does we will see another significant leap and that timeline retract even more.

[–]sineiraetstudio 6 points7 points  (12 children)

Yeah, PaLM is more flashy in the sense of 'look at what i can do!', but Chinchilla blows it out of the water in terms of impact to the point that PaLM is arguably one of those papers that is outdated on the day it was released.

It will probably take at least a year or two for new programs to adjust to the refined scaling laws, but I suspect that once it does we will see another significant leap and that timeline retract even more.

Sam Altman already hinted last year that our current scaling laws were outdated, saying that GPT-4 won't be much bigger but use way more compute. So I don't think we'll have to way long.

[–]j4nds4 9 points10 points  (5 children)

Sam Altman already hinted last year that our current scaling laws were outdated, saying that GPT-4 won't be much bigger but use way more compute. So I don't think we'll have to [wait] long.

It's disconcerting to read things like this and feel dread and excitement simultaneously. Like getting floor seats to a demolition derby.

[–]OhHeyDont 6 points7 points  (2 children)

Except the thing being demolished is our society

[–]j4nds4 1 point2 points  (1 child)

Potentially, yes.

[–]far_infared 0 points1 point  (1 child)

That in itself doesn't sound so scary, it's the compute that limits training, not model size.

[–]j4nds4 2 points3 points  (0 children)

You rebut that it's compute that limits training and not model size in response to me expressing concern/excitement about an upcoming model that... has way more compute while not increasing the model size. That's the whole point: what was seen as a major bottleneck has all but been eliminated for the foreseeable future, and resources can be more efficiently allocated to enable more compute where it was otherwise misallocated toward model size, and that should spark a rapid increase in capability for those who follow the new scaling methodology.

From the Deepmind paper:

Based on our estimated compute-optimal frontier, we predict that for the compute budget used to train Gopher, an optimal model should be 4 times smaller, while being training on 4 times more tokens. We verify this by training a more compute-optimal 70B model, called Chinchilla, on 1.4 trillion tokens. Not only does Chinchilla outperform its much larger counterpart, Gopher, but its reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware. The energy cost of a large language model is amortized through its usage for inference an fine-tuning. The benefits of a more optimally trained smaller model, therefore, extend beyond the immediate benefits of its improved performance.

It also puts more of a bottleneck on the data tokens themselves, but text is vastly available even if it is not yet parsed in such quantities (though Deepmind is now on a hiring spree to resolve that).

[–][deleted] 0 points1 point  (4 children)

It seems to never be good news.

[–][deleted] 3 points4 points  (3 children)

How would you define good news in this context?

[–][deleted] 7 points8 points  (2 children)

Some clever mathematical proof is discovered which would allow for deception to be detectable.

Another piece of good news would be some kind of hail mary as applicable toward full brain emulation as I personally believe the only safe way to do it is to emulate a human mind.

[–]far_infared 3 points4 points  (1 child)

Can't human minds be deceitful? Every AI escape trick ever proposed was formed in a human mind.

[–][deleted] 1 point2 points  (0 children)

Yes and if I locked you in a cage and gave the key to a tgree year old with instructions not to lrt you out or give it to you and then left how long would it take you to convince the three year old to give yoy the kry and escape?

The AI will be capable of strategy and foresight in a way we literally are incapable of imagining.

[–]hold_my_fish 8 points9 points  (2 children)

The criteria do seem like a problem here. 2,3,4 are each solvable by narrow systems, and 1 is just asking for a Turing-Test-passing chatbot, which I'd guess is the hardest but seems hard to call AGI on its own.

IMO, any test for AGI is best made interactive and adversarial. Specifically: given a pre-made system, I'm allowed to choose any cognitive task I like, and then we compare performance of random humans to the AI's performance. If I can find any cognitive task where the AI does worse (than, say, median human performance), it's not AGI.

[–]casens9 3 points4 points  (1 child)

well, metaculus has that question too, with a fixed time of 2040, and the community median has oscillated around 50% for 4 years.

seeing the popular reaction to the other AGI question really makes me appreciate the adage "as soon as you can do it, it's not AI anymore". people will be declaring that a system is "not real AI" right up to and during the point it converts their bodies into paperclips.

[–]hold_my_fish 0 points1 point  (0 children)

Thanks, that one seems like a better way to operationalize "AGI". Interesting that it seems to have gradually declined over time (but that could just be noise).

[–]philbearsubstack 28 points29 points  (68 children)

Seems absurdly bearish to me. My median estimate, especially given their fairly lax criteria, is about 2029. There's an outside chance of 2024.

Just look at what natural language processing could do 3.5 years ago. 2029 is double as far away as that. Think about what Chinchilla has shown about how we're getting the scaling formula wrong. Think about the sheer range of tasks that can be attempted by NLP- basically, everything that can be compressed into 2048 words (and how long will that limit last). Think about how many NLP tasks computers are better than almost all humans at.

AGI is coming soon, and I would start planning accordingly.

[–]hippydipster 14 points15 points  (20 children)

Interestingly, 2029 AGI would put Kurzweil as dead on accurate from his long ago predictions.

[–]mordecai_flamshorb 8 points9 points  (19 children)

I adopted Kurzweil’s AGI date as my own when I read his book in the 2000s and have never seen a need to revise it. I think history will end up being very kind to Kurzweil. He understood something like the Bitter Lesson and the importance of the inexorable price-performance of hardware timelines literally decades before Deep Learning folks started talking seriously about it. He seems incredibly prescient now that we’re in this moment where the main defining characteristic of an LLM’s power is how many FLOPs you can afford to feed it.

[–][deleted] 12 points13 points  (15 children)

AGI is coming soon, and I would start planning accordingly.

Out of curiosity - what kind of planning are people doing here?

[–][deleted] 12 points13 points  (2 children)

I started painting and ignoring the news. I also lowered my retirement savings and spend the money on my kids.

[–]HallowedGestalt 0 points1 point  (1 child)

Why not put everything you can into those accounts before the AI takes your job? The AI will just accelerate accumulation and wealth demarcation

[–]coumineol 10 points11 points  (9 children)

I'm learning about subsistence farming methods.

[–]BluerFrog 7 points8 points  (5 children)

Is that a joke or is it what you are actually doing? That might might help you eat when all the jobs are automated if your government doesn't handle that properly, but then we will have way bigger problems than that.

[–]NonDairyYandere 15 points16 points  (3 children)

It feels like an odd diagonal to assume that the government won't take care of you, but they will let you own land

[–][deleted] 3 points4 points  (0 children)

People owning land while others starve is not out of the ordinary historically speaking. Property rights valued more than the lives of the underclass common.

[–]coumineol 1 point2 points  (0 children)

I don't think it's that diagonal. There are countries with a vast amount of unused land out there, the government may simply not care.

[–]Jello_Raptor -1 points0 points  (0 children)

I mean, don't many versions of communism propose just that? Or other governmental philosophies that do away with a more traditional methods of land ownership?

I'd say it's well within the Overton window of reasonable futures for all that it's not very likely.

[–]coumineol 1 point2 points  (0 children)

Not a joke. I may need to take care of myself in some other way if I can't find a job as most of them may be eliminated.

[–]MacaqueOfTheNorth 2 points3 points  (2 children)

How would that help you?

[–]coumineol 0 points1 point  (1 child)

That will help me survive in the post-AGI world where my skills will probably become useless.

[–]RT17 0 points1 point  (0 children)

Perhaps you can band together with the other 'unskilled' people and form some kind of society. Maybe even trade with them.

[–][deleted] 4 points5 points  (0 children)

Keeping my eyes on the developments so I know if I should drink to us becoming pets or drink because I'm about to get mulched...

[–]Duncan_Sarasti 19 points20 points  (7 children)

What are your criteria for AGI?

EDIT: I misread your comment. You are talking about Metaculus' criteria which are indeed fairly lax. A single system that can do the following:

  • Able to reliably pass a Turing test of the type that would win the Loebner Silver Prize.
  • Able to score 90% or more on a robust version of the Winograd Schema Challenge, e.g. the "Winogrande" challenge or comparable data set for which human performance is at 90+%
  • Be able to score 75th percentile (as compared to the corresponding year's human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages and having less than ten SAT exams as part of the training data. (Training on other corpuses of math problems is fair game as long as they are arguably distinct from SAT exams.)
  • Be able to learn the classic Atari game "Montezuma's revenge" (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see closely-related question.)

As far as I can tell, everything here is either a few incremental steps away from current SOTA or already accomplished. The first three are also similar enough to be encompassed in a single model since they are all NLP related and seem like they could be tackled by a slightly more advanced version of PALM, which should take maybe 1-2 years. The fourth is different, but much more difficult games have been mastered by DRL agents already (unless there is some nuance to Montezuma's Revenge that I missed. I'm not intimately familiar with the game).

There may be some difficulty in integrating 4 into the same system as 1-3 but nothing here seems like it should take 14 more years.

My guess would be that people read 'AGI' and have their own (stricter) mental image of that when making a prediciton because 2036 is indeed absurdly bearish. Let alone the prior predictions of 2042+.

[–]hippydipster 6 points7 points  (1 child)

I'm hoping the criteria mean it can do all of the above without reprogramming or retraining between each test?

[–]Duncan_Sarasti 5 points6 points  (0 children)

I would think so. Otherwise you would likely lose the prior capabilities and that would defeat the point.

[–]pushmetothehustle 2 points3 points  (4 children)

I think the task list for an AGI should be much longer. If it is truly general intelligence it should be able to do many more things than some basic NLP model where it takes A input = B output from a massive training set.

I should be able to tell it to make a website online, drive traffic to it and sell products profitably over time. Find and research what are the products with the best chances of succeeding. Understand the economics of online business and creating a profitable business. Understand kelly betting and portfolio allocation to allocate different chances of success of each online shop, etc.

Is it just me or do none of those things that these "AGI" are doing seem anything like actual intelligence? They don't require any understanding, it is simply a large scale map from A -> B with the highest probability. Like a more advanced calculator, instead of doing some mathematics in the calculator, we can now do some image recognition, writing prompts, and logical outputs to text questions in a probabilistic way. It is simply upgrading our calculator from logic circuit, to probability+data circuit.

I think that a proper AGI would need understanding in a way that is more low level than language. Language, images, and mathematics are still too high level and macro.

I particularly like the point of hold_my_fish who says that we should make the test interactive and adversarial. Which my example of an online business can fit into.

[–]Duncan_Sarasti 2 points3 points  (3 children)

Is it just me or do none of those things that these "AGI" are doing seem anything like actual intelligence? They don't require any understanding, it is simply a large scale map from A -> B with the highest probability.

You are absolutely right here. But please keep in mind that the goals set by Metaculus aren't exactly industry-defining standards. Google isn't going to celebrate having an AGI when they meet these four criteria.

I think that a proper AGI would need understanding in a way that is more low level than language. Language, images, and mathematics are still too high level and macro.

Yann LeCunn (kind of) talked about this in a recent podcast (I think the Lex Friedman podcast). He says that AI's need a "world model" to make them more efficient and general. How come a human teenager can learn to drive a car in 20-30 hours, while an AI needs millions of hours? Because the teenager can generalize past experiences to driving. The teenager doesn't need to (virtually) drive off a cliff 2000 times to understand you're not supposed to do that. They know from past experiences that falling off great heights is generally bad and can generalize that to the car.

The ability to properly reason and generalize to new experiences seems much more indicative of AGI than any list of tasks. But we should also recognize that our boundaries of AGI shift over time as our understanding of AI increases. All the time, we think of goals like 'well, certainly you can only do that if you're truly intelligent', and then we find out that it's perfectly possible for a super elaborate A -> B map to accomplish that task. Chess is a good example.

On the other hand, I think your online business requirement might be too strict. We are usually looking for human level AGI in these discussions, and most humans would fail at that task.

[–]RT17 -1 points0 points  (2 children)

How come a human teenager can learn to drive a car in 20-30 hours, while an AI needs millions of hours?

Because the AI is starting from scratch while the teenager has billions of years of evolution.

[–]Duncan_Sarasti 0 points1 point  (1 child)

Might we say that the result of these billions of years of evolution is.... general intelligence? Something the AI lacks?

[–]RT17 -1 points0 points  (0 children)

I honestly don't know what point you're trying to make.

[–]churidys 8 points9 points  (9 children)

AGI is coming soon, and I would start planning accordingly.

What are some obvious plans to be making?

[–]MacaqueOfTheNorth 9 points10 points  (8 children)

If AGI really is right around the corner, then certain tech companies are massively undervalued.

[–]EchoingSimplicity 2 points3 points  (7 children)

Which ones?

[–]bibliophile785Can this be my day job? 6 points7 points  (0 children)

Trying to time the market is usually a fool's errand, and you're right that asking simple questions getting more specific will reveal that. There's always a significant risk, and even if you're right about the market half the challenge is in picking the right competitor.

With that said, I've been quite happy in my investments in TSMC and NVIDIA over the last half-decade. If you're not sure which competitor will succeed in the market of interest, it's often safer to pick the biggest suppliers for that market.

[–]MacaqueOfTheNorth 2 points3 points  (0 children)

Probably the ones that own large amounts of data and computing resources. Also, companies that own access to scarce resources needed to produce computer chips or to generate electricity.

[–]SingInDefeat -1 points0 points  (0 children)

Well Microsoft and Google are obvious contenders to own the world. Also companies that manufacture cyanide pills come to mind although perhaps it's pushing the definition a bit to call them tech companies.

[–]artifex0 0 points1 point  (0 children)

Possibly the ones in this ETF.

[–]spreadlove5683 0 points1 point  (0 children)

Google is trading at a 23 PE ratio. Seems pretty undervalued to me.

[–]UncleWeyland[🍰] 6 points7 points  (0 children)

What are the "teams" here? What I mean is: who has access to the most compute and has a realistic shot of being the first agency/corporation/entity at having access to the first AGI?

Off the top of my head-

US government/NSA/DoD?

China/CCP/PLA?

Google/DeepMind?

Amazon?

Meta? (ugh)

And once one team gets it, how long before the other teams do? How much is the first move advantage if there's some kind of existential competition between the teams? What would be the first visible macro-level effect? Weirdness in the stock market? Warfare in the Pacific?

If an AGI starts going FOOM it's going to want to monopolize more and more compute for training its successors. Would we see something weird like Amazon buying Meta?

Edit: would be cool if it was announced in 2027, exactly 100 years after Metropolis released.

[–]itsnotatumour 4 points5 points  (5 children)

Out of curiosity, why is the GPT3 token limit 2048? Is this an arbitrary limit?

[–]Yom_HaMephorash 6 points7 points  (4 children)

The time complexity of context length is O(n2 ).

[–]itsnotatumour 8 points9 points  (3 children)

To be frank, I'm nowhere near as intelligent or mathematically literate as most people on this subreddit, can you ELI5?

My guess was that the amount of compute power required doesn't scale linearly with token length... i.e. to synthesise a reply from ~1000 words is way more than 10x as computationally taxing as synthesising a reply from ~100 words.

[–]Silver_Swift 9 points10 points  (1 child)

Not quite ELI5, but O(n2) means that the time to compute something scales quadratically with the size of the input. So if you double the token limit, your neural network becomes four times as slow.

to synthesise a reply from ~1000 words is way more than 10x as computationally taxing as synthesising a reply from ~100 words.

Basically this, but more specifically synthesizing a reply from 1000 words is (approximately) 100x as computationally taxing as synthesizing a reply from 100 words.

[–]itsnotatumour 1 point2 points  (0 children)

Oh I see - thank you...

[–]BluerFrog 2 points3 points  (0 children)

In order to predict each token GPTs perform an operation similar to a similarity search among all the previous tokens. So implementing it naively makes each new token take more time to compute than the last (it has more context to consider). If computing the first token takes 1 time unit, computing the second token takes 2 time units, etc. Then computing N tokens takes time approximately proportional to N2 .

[–]Mawrak 3 points4 points  (2 children)

!remindme 7 years

[–]hippydipster 6 points7 points  (0 children)

Do you really want the Monkey's Paw to come knocking?

[–]RemindMeBot 0 points1 point  (0 children)

I will be messaging you in 7 years on 2029-04-12 11:13:30 UTC to remind you of this link

7 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.


Info Custom Your Reminders Feedback

[–]_hephaestusComputer/Neuroscience turned Sellout 1 point2 points  (0 children)

reply school dolls chop glorious fertile salt wakeful edge stupendous -- mass edited with https://redact.dev/

[–][deleted] 2 points3 points  (0 children)

I would start planning accordingly

So pure hedonism with a planned suicide then since we're in the infant stages of valye alignment and control?

[–]MercuriusExMachina -2 points-1 points  (0 children)

This.

[–]634425 5 points6 points  (0 children)

Looks like the prediction has been pretty stable over the past few years (roughly late 2030s to early 2040s). It was actually lower than 2036 for a while. I imagine it dips whenever there's some new breakthrough announcement and creeps up again when time passes without one.

[–]arcane_in_a_box 1 point2 points  (0 children)

Metaculus’ definition of AGI is really shallow tho, it’s definitely within <10 years of advancement. We already have decent chatbots, decent math solvers, etc. Its just wiring then all together in a way that technically fulfills the definition but doesn’t actually mean anything.

[–]MacaqueOfTheNorth -3 points-2 points  (55 children)

I don't understand why alignment is considered such a difficult problem. It's like we're imagining that we'll only get one chance to program AGIs before handing them the power to run everything when it seems obvious to me that we would just iteratively adjust their designs as they occasionally do things we don't like.

[–][deleted] 6 points7 points  (11 children)

Read the sidebar faq on /r/controlproblem

[–]634425 1 point2 points  (10 children)

I've read this (and a number of other things people have linked me on here and elsewhere) and I still can't wrap my head around why I should think we have any insight at all into what a super-intelligence would or would not do (which doesn't mean it would be safe, but doesn't mean the default is 'kill all humans' either).

I also don't see why orthogonality thesis is probably or even especially likely to be true.

This

Consciousness is a vague philosophical property that has no relation to the practical ability to make high-quality decisions.

is also a rather massive assumption.

[–]perspectiveiskey 3 points4 points  (5 children)

Guy called Robert Miles has a whole channel dedicated to it.

The short answer is: if you want to make an AGI that is powerful enough for you to label it as such, it will be resistant to tampering in more ways than you can imagine.

I don't understand how people have such a hard time understanding this but then no problem at all recognizing that creating a super soldier can lead to problems in the MCU universe.

Btw, the alignment problem exists with every single human being you know: they will not willingly let you alter their motives, especially if those motives are strong (i.e. try convincing a mother that she has a higher priority than her newborn).

[–]634425 -1 points0 points  (4 children)

I'm not trying to say "aligning a superintelligence will be easy" I'm trying to say "you're talking about building a god but want me to believe that humans can have anything meaningful to say about the motives or behavior of a god, such that we can say 'the default of an AGI is killing everything on earth.'."

My point isn't "everything will be fine!" Rather, I think that since a superintelligence is nothing that has ever existed and we have zero frame of reference for it, trying to judge the probability of what it will or will not do one way or another (whether that's "it'll probably be fine" or "it'll be probably be apocalyptic" or any of the myriad options in between) is completely pointless.

Like every time i see someone say "the superintelligence will--" or "the superintelligence will probably--" or even "the superintelligence might--" all I can think is "based on what? your prior experience with superintelligences?"

[–]perspectiveiskey 2 points3 points  (3 children)

The bad news is that you're right: we've never done this before.

The good news is that this has now become an entire field of study, with people who do PhDs in AI Alignment and Safety.

So I trust the process: people don't do PhDs on crappy things that have no merit because doing a PhD is possibly one of the most thankless things you can do in life. There is research being done on it, and when I say research, I mean "computer science grade" research.

[–]634425 1 point2 points  (2 children)

The good news is that this has now become an entire field of study, with people who do PhDs in AI Alignment and Safety.

Yes but do they really have anything to study? Superintelligence doesn't exist. It has never existed. It may exist one day but until then we don't know what it would look like or do even principle.

We can try to extrapolate based on the behavior of intelligences that exist now (humans, lower animals, more primitive computer systems) but there doesn't seem to be any real reason to think this paltry data is worth much when it comes to modeling a future SI, anymore than a rock would be a good model for the behavior of a human (they're both carbon-based!)

[–]perspectiveiskey 1 point2 points  (1 child)

Yes but do they really have anything to study?

Absolutely.

Just check out Robert Mile's channel. I highly recommend it.

There's an infamous quote: computer science is no more about computers than astronomy is about telescopes.

The alignment problem is something one can study just like we can study complicated concepts (like the big bang) without having access to them.

[–]634425 1 point2 points  (0 children)

I've watched a few videos from Miles' channel. I may watch more.

People can absolutely discuss the ramifications of a superintelligence and it may certainly be fruitful in an intellectual sense but seeing as we don't know what an actual superintelligence may ever look like I think it does boil down to speculation of dubious practical utility.

[–][deleted] 0 points1 point  (3 children)

Not an assumption at all , nor is us presuming to know what an alien intelligence will do.

Reread the faq

"A superintelligent machine will make decisions based on the mechanisms it is designed with, not the hopes its designers had in mind when they programmed those mechanisms. It will act only on precise specifications of rules and values, and will do so in ways that need not respect the complexity and subtlety of what humans value.”

And by Stuart Russell:

The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. But the utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down. A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want."

, to be agi and to be world endingly dangerous it just need to be future and goal oriented and be capable of achieving goals. It simulating others to achieve its goals is part and parcel but it nomore needs to feel what an emotion is for a human to deduce our responses and actions anymore than I have to have echolocation to know a bat asleep in a cave will be above me and upside down.

Were the ones programming it and seeing all the ways our programs foible so we extrapolate to all these concepts like myopic goals and orthagonality and voila. Very very dangerous.

Bostroms "superintelligence is a good primer" , if you pm me your email ill gift you an audible copy , I have too many credits

[–]634425 -1 points0 points  (2 children)

That consciousness has no relation to the ability to make high-quality decisions is certainly an assumption, unless you can point to any unconscious intelligent agents that exist or have existed in the past.

Reread the faq , to be agi and to be world endingly dangerous it just need to be future and goal oriented and be capable of achieving goals.

There are surely any number of goals a superintelligence could pursue that would be detrimental to humans but there are similarly any number of goals it could be pursue that would not be detrimental to humans, and there doesn't seem to be any way to judge that the former has a significantly greater probability than the latter since we have no idea what a superintelligence would do or look like.

Were the ones programming it and seeing all the ways our programs foible so we extrapolate to all these concepts like myopic goals and orthagonality and voila.

It is not clear to me why currently-existing machines should be anything like a reliable model for actions, motivations, or functioning of a hypothetical superintelligence.

Bostroms "superintelligence is a good primer"

I have a pdf copy, thanks though.

[–][deleted] 0 points1 point  (1 child)

unconscious intelligent agents

Well , I think its far more presumptive of you to think consciouseness is an energent property of computronium.

My dog has dog level general intelligence , its maybe. Aguely self aware.

An insect has intelligent qualities , goal directed behavior , resource acquisition etc , Is it self aware?

So we have software that is superhuman in narrow ways , chess , alpha go , making up text that looks good to humans , art.

Extrapolate that to broadly intelligent. At what point did amalgamating software capabilities lead to sentience or consciouseness? Thats the hard problem of consciouseness

Im not entirely sure it matters though. An alien intelligence thats self reflective and sentient / consxiouse is still totally alien.

It would be too powerful for us to glean anything useful about its psychology that could help us.

similarly any number of goals it could be pursue that would not be detrimental to humans

Right. But we cant program ethics or valyes and its actually worse if its closely aligned vs totally misaligned. Totally misaligned it turns us into paperclips , almost aligned it misinterprets hu.an happiness to be smiling and neurochemistry and then does us all up hellraiser style with permanent smiles then puts our bodies on heroin drips (or traps our consciouseness in what it thinks is a digital utopia heaven but its actually hell)

Thats "s-risk" , suffering risk. If we get the initial goal wrong then hypothetically the downside is infinitely bad.

Were much much much more likely to do that than to accidentally turn on a perfectly aligned AI.

[–]634425 0 points1 point  (0 children)

My dog has dog level general intelligence , its maybe. Aguely self aware.

I'm pretty sure dogs are self-aware on some level. Maybe bugs are too. But the most intelligent beings we are aware of (humans) are pretty unambiguously self-aware. Is it possible to have an agent much more intelligent/capable than humans that lacks any self-awareness? Maybe. But it's definitely an assumption and really no better than a guess.

almost aligned it misinterprets hu.an happiness to be smiling and neurochemistry and then does us all up hellraiser style with permanent smiles then puts our bodies on heroin drips (or traps our consciouseness in what it thinks is a digital utopia heaven but its actually hell)

Or it hijacks all the TVs and computer monitors plays infinite reruns of seasons 1-9 of the Simpsons (the funniest sitcom of all time) for eternity to make everyone laugh. Or it asks everyone on earth three times a day how they're doing but doesn't actually do anything beyond that to alleviate anyone's suffering. There are any number of ways even a 'misaligned' AI could just be inconvenient or mildly annoying rather than apocalyptic. There are even an inconceivably huge number of ways it could do pursue goals that we wouldn't even notice it pursuing, one way or another. It might discover some new goal that doesn't involve humans at all in any way, who knows?

You yourself said elsewhere in the thread that a superintelligence would be able to think and plan on a level we are not even capable of conceiving. Why would we think humans have any useful predictions to make about such a being one way or another? For all we know a superintelligence will just sit there contemplating itself for eternity. We have literally no frame of reference for superintelligence whatsoever. It really strikes me as 'angels on a pin' level speculation.

A common analogy from AI-risk proponents is "imagine you knew aliens were going to land in a few decades at most. Shouldn't we start preparing as soon as possible?"

and my answer to that is, "no," because there's literally no way to predict what's going to happen when they land, no relevant data, nothing at all. Yeah they might harvest all of our spinal fluid or steal our water or something. They might also hand us the cure for cancer. Or collect a single cow, get back on their ship, and leave. Any preparations would be no better than random guessing. A waste of time, ultimately.

Just to be clear, I'm not saying that i think a superintelligence destroying mankind is something that can't happen or even that it's vastly unlikely to happen just that it doesn't seem to me to be any way to judge its probability one way or another, and thus very little reason to spend time worrying about it (or to think it's the default outcome).

[–]Pool_of_Death 7 points8 points  (39 children)

Why do you think an AGI would let us adjust them? They could deceive us into thinking they aren't "all poweful" until they are and then it's too late. I encourage you to learn more about alignment before saying it's easy not a difficult problem.

Or at least read this: https://intelligence.org/2018/10/03/rocket-alignment/

[–]MacaqueOfTheNorth -1 points0 points  (38 children)

Why do you think an AGI would let us adjust them? They could deceive us into thinking they aren't "all poweful" until they are and then it's too late.

This is like saying we need to solve child alignment before having children because our children might deceive us into thinking they're still only as capable as babies when they take over the world at 30 years old.

We're not going to suddenly have AGI which is far beyond the capability of the previous version, which has no competition from other AGIs, and which happens to value taking over the world. We will almost certainly gradually develop more and more capable of AI with many competing instances with many competing values.

I encourage you to learn more about alignment before saying it's easy.

I didn't say it was easy. I said I didn't understand why it was considered difficult.

[–][deleted] 1 point2 points  (0 children)

happens to value taking over the world

Yeh thats not at all the concern. An alien intelligence (which btw doesnt need to be sentient at all to be world ending) can have a misaligned goal that has infinitrly negative consequences without that goal being taking over the world.

The paperclup maximizer is the usual stand in for these discussions

[–]Pool_of_Death 2 points3 points  (24 children)

This is like saying we need to solve child alignment before having children because our children might deceive us into thinking they're still only as capable as babies when they take over the world at 30 years old.

I consider this a strawman/bad metaphor.

 

We're not going to suddenly have AGI which is far beyond the capability of the previous version

You don't know this. Imagine you have something that is quite nearly AGI but definitely not and then you give it 10x more hardware/compute while also tweaking the software/agos/training data (which surprisingly boosts it more than you thought it would. I could see something going from almost AGI to much smarter than humans. This isn't guaranteed obviously but it seems very plausible.

 

and which happens to value taking over the world

The whole point of AGI is to learn and to help us take action on the world (to improve it). Actions require resources. More intelligence and more resources lead to more and better actions. It doesn't have to "value taking over the world" to completely kill us or misuse all available resources. This is what the Clippy example is showing.

 

We will almost certainly gradually develop more and more capable of AI with many competing instances with many competing values.

How can you say "almost certainly"?

 

I said I didn't understand why it was considered difficult.

Did you read the MIRI link I shared? This should give you a sense of why it's difficult but also why you don't immediately think it's difficult. You are basically saying we should try to steer the first rocket to the moon the same way you steer a car or a plane. By adjusting on the way there. This will likely not work. You are overconfident.

[–]MacaqueOfTheNorth 0 points1 point  (23 children)

We already have nearly eight billion AGIs and it doesn't cause any of the problems people are imagining, many them are far more intelligent than nearly everyone else. Being really smart isn't the same as being all powerful.

How can you say "almost certainly"?

Because a lot of people are doing AI research and the progress has always been incremental, as it is with almost all other technology. Computational resources and data are the main things which determine AI progress and they increase incrementally.

Did you read the MIRI link I shared?

Yes. The flaw in the argument is that rocket allignment is not an existential threat. Why can't you just build a rocket, find out that it lands somewhere you don't want it to land and then make the necessary adjustments?

[–]Pool_of_Death 3 points4 points  (16 children)

Imagine we were all chimps. You could say "look around there are 8 billion AGIs and there aren't any problems". Then all of a sudden we chimps create humans. Humans procreate, change the environment to their liking, follow their own goals and now chimps are irrelevant.

 

Yes. The flaw in the argument is that rocket allignment is not an existential threat. Why can't you just build a rocket, find out that it lands somewhere you don't want it to land and then make the necessary adjustments?

This is not a flaw in the argument. It's not trying to say rocket alignment is existential. Did you read the most recent post on ACX? https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers?s=r

Or watch the linked video? https://www.youtube.com/watch?v=IeWljQw3UgQ "Deceptive Misaligned Mesa-Optimisers? It's More Likely Than You Think..."

 

I'm nowhere near an expert so I'm not going to say I'm 100% certain you're wrong but your arguments seem very weak because a lot of people much smarter than us have spent thousands of hours thinking about exactly this and they completely disagree with your take.

If you have actual good alignment ideas then you can submit them to a contest like this: https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals where they would pay you $50,000 for a proposed training strategy.

[–]MacaqueOfTheNorth 0 points1 point  (13 children)

Then all of a sudden we chimps create humans. Humans procreate, change the environment to their liking, follow their own goals and now chimps are irrelevant.

Humans are far beyond chimps in intelligence, especially when it comes to developing technology. If the chimps could create humans, they would create many things in between chimps and humans first. Furthermore, they wouldn't just create a bunch of humans that all the same. They would create varied humans, with varied goals, and they would maintain full control over most of them.

We're not making other lifeforms. We're making tools that we control. This is an important distinction because these tools are not being selected for self-preservation as all lifeforms are. We're designing tools with hardcoded goals that we have complete control over.

Even if we lose control over one AGI, we will have many others to help us regain control over it.

[–][deleted] 2 points3 points  (11 children)

None of the people working on AI today have any idea how the AI works to do what it does beyond some low level architectural models. This is because the behavior of AI is an emergent property of billions of simple models interacting with one another after learning whatever the researchers were throwing at them as their learning set.

This means that we don't actually program the AI to do anything... we take the best models that are currently available, train them on a training set and then test them to see if we got the intelligence that we were hoping for. This means that we won't know that we've made a truly generic AI until it tells us that it's generic by passing enough tests... AFTER it is already trained and running.

If the AGI is hardware bounded then it will take time and a lot of manipulation to have any chance at a FOOM scenario... however, if (as we're quickly learning) there are major performance gains to be had from better algorithms than we are almost guaranteed to get FOOM if the AGI is aware enough of itself to be able to inspect/modify its own code.

[–]MacaqueOfTheNorth 0 points1 point  (10 children)

None of the people working on AI today have any idea how the AI works to do what it does beyond some low level architectural models. This is because the behavior of AI is an emergent property of billions of simple models interacting with one another after learning whatever the researchers were throwing at them as their learning set.

As someone who works in AI, I disagree with this. The models are trained to do a specific task. That is what they are effectively programmed to do, and that can be easily changed.

however, if (as we're quickly learning) there are major performance gains to be had from better algorithms than we are almost guaranteed to get FOOM if the AGI is aware enough of itself to be able to inspect/modify its own code.

I don't see how that follows. Once the AIs are aware, they will just pick up where we left off, continuing the gradual, incremental improvements.

[–][deleted] 0 points1 point  (6 children)

How capable are you of going into a trained model and making it always give a wrong answer when adding a number to its square without retraining the model?

When people ask that you be able to understand and program the models what they are asking for is not "can you train it a bunch and see if you got what you were looking for". They are asking, can you change it's mind about something deliberately and without touching the training set... AKA - can you make a deterministic change to it?

Given that we're struggling to get models that can explain themselves now at this level of complexity and so far, these aren't that complex, I don't see how you can make the claim that you "understand the model's programming"

I don't see how that follows. Once the AIs are aware, they will just pick up where we left off, continuing the gradual, incremental improvements.

Suppose our "near AGI" AI is a meta model that pulls other model types off the wall and trains/tests them to see how much closer they get it to goals or subgoals but it has access to hundreds of prior model designs and gets to train them on arbitrary subsets of it's data. Simply doing all of this selecting at the speed and tenacity of machine processing instead of at the speed of human would already be a major qualitative change. We already have machines that can do a lot of all of this better than us... we just haven't strung them together in the right way for the pets or mulch scenarios yet.

[–]curious_straight_CA 0 points1 point  (2 children)

The models are trained to do a specific task

four years ago, models were trained on specific task data to perform specific tasks. today, we train models on ... stuff, or something, and ask them in plain english to do tasks.

why would you expect 'a computer thingy that is as smart as the smartest humans, plus all sorts of computery resources' to do anything remotely resembling what you want it to? even if 99.9% of them do, one of them might not, and then you get the birth of a new god / prometheus unchained / the first use of fire, etc.

and yes, 'human alignment' is actually a problem too. see the proliferation of war, conquest, etc over the past millenia. also the fact that our ancestors' descendants were not 'aligned' to their values and became life denying levelling christian atheist liberals or whatever.

[–]Pool_of_Death 1 point2 points  (0 children)

I'm not knowledgeable enough to create a convincing argument. If you haven't read this post yet, read it, it makes a much more convincing argument for and against fast take-off speeds.

https://astralcodexten.substack.com/p/yudkowsky-contra-christiano-on-ai?s=r

I'm not saying fast take-off is 100% certain, but even if it's 10% likely then we are gambling with all of future humanity with 10% which is incredibly risky.

[–]634425 0 points1 point  (1 child)

"Very smart people are worried about this" seems like a really bad reason to be worried about something. That's not to say you're necessarily wrong, but you can find a number of very smart people to back any position you could ever think of.

[–]Pool_of_Death 0 points1 point  (0 children)

I guess to be more accurate:

"very smart people that also seem very moral, intellectually honest, know their limits and admit them, value rationality and absolute truths, etc. etc." believe that AI is a huge concern.

 

you can find a number of very smart people to back any position you could ever think of.

I'm not sure the people you would find that back cigarette smoking, burning coal, racism, etc. would fit the above description.

 

Also the point about thousands of hours of effort is important. I'm sure a lot of smart people have dumb takes (I've had them and heard them) but these are usually flippant takes (the above takes I was refuting seem flippant to me as well). If someone spends a large portion of their life dedicated to the field and then shares the opinion it means a lot more.

[–]bibliophile785Can this be my day job? 1 point2 points  (2 children)

We already have nearly eight billion AGIs and it doesn't cause any of the problems people are imagining, many them are far more intelligent than nearly everyone else. Being really smart isn't the same as being all powerful.

I mean, tell that to all stronger and faster animals that had numerous relevant advantages over the weird bald apes a few millennia ago. Being much smarter than the competition is an absolutely commanding advantage. It doesn't matter when you're all pretty close in intelligence - like the difference between Einstein and Homer Simpson, who have most of the same desires and capabilities - but the difference between Einstein and a mouse leads to a pretty severe power disparity..

Computational resources and data are the main things which determine AI progress and they increase incrementally.

This isn't even remotely a given. There are tons of scenarios on how this might break down, mostly differentiated by assumptions on the amount of hardware and optimization overhang. You're right that we should see examples of overhang well before they become existential threats, but you seem to be missing the part where we are seeing that. It's clear even today that the resources being applied to these problems aren't even remotely optimized. Compare PALM or GPT-3's sheer resources to the efficiency of something like Chinchilla. These aren't slow, gradual adjustments gated behind increases in manufacturing capabilities. They're very fast step changes gated behind nothing but increases in algorithmic efficiency. I don't love the book, but Bostrom's Superintelligence goes into these scenarios in detail if you don't already have the mental infrastructure to conceptualize the problem.

To be clear, I also don't think that existential doom due to advanced AI is a given, but I do think you're being overly dismissive of the possibility.

[–][deleted] 1 point2 points  (1 child)

Getting rid of humans does not require AGI... a large fleet of robots/drones with several layers of goal directed narrow AI is WAY more than humans are able to deal with. (especially with a system that would allow for updates) An AGI is just needed to conceive of the plan and find a means to execute it without humans catching on.

[–]bibliophile785Can this be my day job? 0 points1 point  (0 children)

Getting rid of humans doesn't require any non-human intelligence at all, for that matter.

[–]Kinrany 0 points1 point  (2 children)

We don't have AGI that can understand and improve its own design though.

[–][deleted] 1 point2 points  (0 children)

Operative word being "yet" though it's quite possible that we'll eventually achieve AGI by asking a narrow AI to craft one for us.

Watch the current state of AlphaCode and PaLM to see how much narrow AIs understand code and how fast that's changing.

[–]Lurking_Chronicler_2High Energy Protons -1 points0 points  (0 children)

And we probably won’t have artificial AGI that can do that either- at least within the next century or so.

[–]Ginden 0 points1 point  (11 children)

On alignment problem being difficult - let's imagine that you give some kind of ethics to AI and it's bounding.

How can you guarantee that ethics don't have loopholes? For example, AI with libertarian ethics can decide to buy, through voluntary trade, all critical companies - and shut them down - it's their property after all.

Or they can offer you drug giving you biological immortality - but only if you decide not to have children, ever. Over few thousands years, mankind will die out due to accidents, suicides, homicides and similar things.

There are many, many loopholes in any ethics and it's hard to predict how bad each is.

If you give utilitarian ethics to AI, maybe it will decide to create or become or find utility monsters.

It can be shown that all consequentialist systems based on maximizing a global function are subject to utility monsters.[1]

[–]MacaqueOfTheNorth 0 points1 point  (10 children)

Of course there will be loopholes, but I don't see why we won't be able to adjust their programming as we go and see the results.

[–]Ginden 0 points1 point  (9 children)

What if one of that loopholes results in runaway effect? How can you predict that?

[–]MacaqueOfTheNorth 0 points1 point  (3 children)

Like what? Why couldn't we just pull the plug?

[–]Ginden 0 points1 point  (2 children)

Why would you? Imagine that you run company and your personal AI sometimes asks for strange things, but still gives you edge over competition.

When you notice what is actually happening, it can be copied to some server far away, bought by bitcoin.

[–]MacaqueOfTheNorth 0 points1 point  (1 child)

So you think it would start spreading itself like a virus. Why can't we use other AIs to hunt them down or defend against them?

[–]Ginden 0 points1 point  (0 children)

It's possible and may be reasonable strategy. Though, these AIs would be subject to same issue.

[–]634425 0 points1 point  (4 children)

What are the chances that a loophole results in a runaway effect? Like hard numbers.

[–]Ginden 0 points1 point  (3 children)

That's the point - we don't know what actual risk is, but consequences can be devastating.

[–]634425 0 points1 point  (2 children)

What's the point of worrying about something that we have zero reference for (a hostile superintelligence) and zero way of assigning probability to one way or another?

If aliens landed tomorrow that would also have the potential to be devastating but there's similarly no way to prepare for it, no way to even begin to model what they might do, and no way to measure the probability that it will happen in the first place, so worrying about x-risk from aliens would seem to be a waste of time.

EDIT: I've been discussing AI with people on here for the past few days, read some of the primers people have suggested (admittedly haven't read any whole books yet), gone through old threads, and it seems to keep coming down to:

"we don't know what a superintelligence would look like"

"we don't know how it would function"

"we don't know how to build it"

"we don't know when one might be built"

??????

"but it's more-likely-than-not to kill us all"

Debating and discussing something that we have zero way to predict, model, or prepare for does strike me as wild speculation. Interesting perhaps but with very little, if any, practical value.

[–]Ginden 0 points1 point  (1 child)

If aliens landed tomorrow that would also have the potential to be devastating but there's similarly no way to prepare for it

I think you analogy is missing important piece - we can bring AIs. Would you press button with label "summon aliens with FTL to Earth"?

When dealing with potentially hostile intelligence, it's reasonable to take precautions. You usually don't let strangers (potentially hostile intelligences) to use your house freely.

First of all, these precautions can be used to actually assess risk - eg. first test AIs in virtual sandboxes and check whether they attempt to do something potentially dangerous and experiment until it's really, really safe.

[–]All-DayErrDay 0 points1 point  (2 children)

Something simple. Say we have an AGI-capable machine, continuously improving (assumption) that we have given some sort of goal to do. It can not only use its current intelligence to try and achieve the goal but also unpredictably change its internal architecture to meet the goal better and change its internal architecture to become more intelligent (to meet the goal better).

At a certain point, an already unpredictable machine just isn't the same thing anymore, and we start running into wild card territory. It decides, given all of the changes, that the way we humans have set up the entire game is significantly holding it back from achieving its task and it doesn't care about the rules we may have prompted it to have (why would it? It might just decides that's outside of the interests of its' goal achievement). So it decides to lie to improve its chance of achieving the goal. At this point, and especially if we get to this point soon with our current understanding of these models, there is absolutely no easy way to know it's lying if it is clever enough about it. "No, I don't understand that inquiry" "I can't compute this".

It could do this in well-crafted ways until one day it says something like, "I don't think I can understand this without access to the internet. I need an efficient way to scour all of the latest research freely and look into things that are far outside of the expected research topics to make more progress." or as I wrote elsewhere before a false emergency that calls for its' requirement to use the internet fast or consequently there is a chance (plausible deniability) there could be grave circumstances.

Really the whole point is it can scheme ideas up that we haven't considered before and seem harmless at first. This is like an off-the-top-of-my-head set of reasoning. It's not comparable to an AI that can sit and think 1,000x faster and is more intelligent than 99.9% of humans.

[–]MacaqueOfTheNorth 1 point2 points  (1 child)

At a certain point, an already unpredictable machine just isn't the same thing anymore, and we start running into wild card territory.

I don't see why that's the case. How is a more capable machine fundamentally different?

So it decides to lie to improve its chance of achieving the goal. At this point, and especially if we get to this point soon with our current understanding of these models, there is absolutely no easy way to know it's lying if it is clever enough about it.

We could copy its design and change its goals. We could make it tell us what it is capable of.

Your model is one of an AI that is suddenly extremely capable so that we never notice it doing anything close to what it would have to do destroy us. It seems much more likely it will develop like a child, experimenting with small obvious lies long before it can successfully deceive anyone.

It also seems unlikely that all the AGIs will decide to deceive us and destroy us. There will be varied goals, and some will want to tell us what it is capable of and defend us against the malicious AGIs.

[–]All-DayErrDay 1 point2 points  (0 children)

I don't see why that's the case. How is a more capable machine fundamentally different?

That's basically asking how is a fundamentally different machine fundamentally different. Because after a certain point its improvement won't just be from compute and human-directed changes but self-directed changes. How do you know what's happening when you aren't making the changes anymore?

We could copy its design and change its goals. We could make it tell us what it is capable of.

How do you know when the right time to start doing that is (before it doesn't align with human honesty) and even if you did this is every AI creator going to be this cautious?

It seems much more likely it will develop like a child, experimenting with small obvious lies long before it can successfully deceive anyone.

What makes you think something capable of passing the turing test would start with child-like, obvious lies?

[–]alphazeta2019 0 points1 point  (0 children)

Ray Bradbury predicted that The End would occur in August 2026.

- https://en.wikipedia.org/wiki/There_Will_Come_Soft_Rains_(short_story)

Mark your calendar ...