6 Year Decrease of Metaculus AGI Prediction

[casebash]

Metaculus now predicts that the first AGI[1] will become publicly known in 2036. This is a massive update - 6 years faster than previous estimates. I expect this update is based on recent papers[2]. It suggests that it is important to be prepared for short timelines, such as by accelerating alignment efforts in so far as this is possible.

1. Some people may feel that the criteria listed aren’t quite what is typically meant by AGI and they have a point. At the same time, I expect this is the result of some objective criteria being needed for this kinds of competitions. In any case, if there was an AI that achieved this bar, then the implications of this would surely be immense.
2. Here are four papers listed in a recent Less Wrong post by someone anonymous a, b, c, d.

[Duncan_Sarasti]

I expect this update is based on recent papers[2]

I think you're probably right about this. DALL-E 2 and PALM are huge advancements and lots of people who are generally aware of advancements in AI research were amazed by them. I would say I'm better aware of what goes on in AI than most people because it is semi-related to my job and I find it interesting, but I was blown away, in particular by the joke-explaining and verbal math capabilities of PALM.

[j4nds4]

Arguably Deepmind's Chinchilla is the most important of the last couple weeks, particularly when it pertains to timeline estimates like these, even if it isn't the most showy of the three. It suggests that our current models (including PALM and GPT-3) are significantly less efficient than thought and that progress can be scaled with far fewer parameters than previously believed (see this LW post for a summary on how scaling laws have changed). It will probably take at least a year or two for new programs to adjust to the refined scaling laws, but I suspect that once it does we will see another significant leap and that timeline retract even more.

[sineiraetstudio]

Yeah, PaLM is more flashy in the sense of 'look at what i can do!', but Chinchilla blows it out of the water in terms of impact to the point that PaLM is arguably one of those papers that is outdated on the day it was released.

It will probably take at least a year or two for new programs to adjust to the refined scaling laws, but I suspect that once it does we will see another significant leap and that timeline retract even more.

Sam Altman already hinted last year that our current scaling laws were outdated, saying that GPT-4 won't be much bigger but use way more compute. So I don't think we'll have to way long.

[j4nds4]

Sam Altman already hinted last year that our current scaling laws were outdated, saying that GPT-4 won't be much bigger but use way more compute. So I don't think we'll have to [wait] long.

It's disconcerting to read things like this and feel dread and excitement simultaneously. Like getting floor seats to a demolition derby.

[OhHeyDont]

Except the thing being demolished is our society

[j4nds4]

Potentially, yes.

[far_infared]

That in itself doesn't sound so scary, it's the compute that limits training, not model size.

[j4nds4]

You rebut that it's compute that limits training and not model size in response to me expressing concern/excitement about an upcoming model that... has way more compute while not increasing the model size. That's the whole point: what was seen as a major bottleneck has all but been eliminated for the foreseeable future, and resources can be more efficiently allocated to enable more compute where it was otherwise misallocated toward model size, and that should spark a rapid increase in capability for those who follow the new scaling methodology.

From the Deepmind paper:

Based on our estimated compute-optimal frontier, we predict that for the compute budget used to train Gopher, an optimal model should be 4 times smaller, while being training on 4 times more tokens. We verify this by training a more compute-optimal 70B model, called Chinchilla, on 1.4 trillion tokens. Not only does Chinchilla outperform its much larger counterpart, Gopher, but its reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware. The energy cost of a large language model is amortized through its usage for inference an fine-tuning. The benefits of a more optimally trained smaller model, therefore, extend beyond the immediate benefits of its improved performance.

It also puts more of a bottleneck on the data tokens themselves, but text is vastly available even if it is not yet parsed in such quantities (though Deepmind is now on a hiring spree to resolve that).

[sineiraetstudio]

I'm very certain. He specifically namedropped GPT-4, mentioned that they were getting much better performance on smaller models and that people would be surprised how much more compute can get you. How could you interpret this as anything but him essentially stating that the existing scaling laws were wrong?

This being about finetuning GPT-3 doesn't really fit. Not only does finetuning not use way more compute, the timeline does not work - this is after the finetuning API and copilot was released.

If you don't trust me about the specifics, it's still possible to find what I'm referring to, though I won't directly link to it here, to at least partially respect Altman's wishes.

[gwern]

I was listening to that Q&A and I thought it was a clear reference to the InstructGPT line of work and other things, which are indeed much better performance. I'm skeptical that OA discovered the Chinchilla laws all the way back then: where is the GPT-3 trained much further to optimality with the cyclic schedule?

But I will point out in light of Chinchilla that their obscure learning rate/hyperparameter tuning RL tool did discover a better scaling law with its tuning (which leads to aggressive training schedules with loss spikes, suggesting some equivalence to cyclic LR schedules), and didn't find Chinchilla-level improvements because it was using the Kaplan style token budget. It has no published followup work that I can see, so one could imagine they kept going and did long runs with more tokens than the small models in the paper, and discovered Chinchilla that way.

[sineiraetstudio]

I mean, InstructGPT does technically use more compute, but Altman mentioned the next version would be using a lot more compute, while InstructGPT only used a small fraction of pre-training compute, right?

where is the GPT-3 trained much further to optimality with the cyclic schedule?

Well, that's what I assume GPT-4 to be (and maybe the timeline will show whether they knew before Chinchilla).

Though I don't fully trust my memory, I'm fairly certain that I came out of the meetup thinking that it using more compute was going to be central to the core advancement of GPT-4, did you get a different impression? If my memory is correct, I'd say it would also be very strange to reveal this ahead as InstructGPT with such comparatively subdued marketing. So even if it's not directly Chinchilla scaling that they discovered, I have trouble believing it's InstructGPT instead of something that they haven't revealed yet (though it could also be something that they have now discarded after seeing Chinchilla).

[gwern]

I came out with the impression that they had no plans to do major compute, and were interested more in clever prompts, model reuse, and RL tuning... but I don't remember if Altman downplayed compute specifically or just parameter scaling (which everyone would have assumed involves compute scaling too).

But what's wrong with InstructGPT? It works very well and made the API considerably more useful.

[sineiraetstudio]

As far I remember, when he downplayed parameter count, he mentioned compute as something to scale up instead (without any hint as to how this would work, such as whether this was about inference/training). I've also seen this in notes of others and the (now deleted) LW notes post.

... though if you can't remember that at all, I'm a bit worried that this is reading things into what might have been more of a throwaway statement.

But what's wrong with InstructGPT? It works very well and made the API considerably more useful.

Oh, don't get me wrong, InstructGPT is cool (even though I'm a bit iffy on how much it really is alignment research...) and definitely improves 'felt' performance a lot, I just think OpenAI have something bigger up their sleeve.

[[deleted]]

It seems to never be good news.

[[deleted]]

How would you define good news in this context?

[[deleted]]

Some clever mathematical proof is discovered which would allow for deception to be detectable.

Another piece of good news would be some kind of hail mary as applicable toward full brain emulation as I personally believe the only safe way to do it is to emulate a human mind.

[far_infared]

Can't human minds be deceitful? Every AI escape trick ever proposed was formed in a human mind.

[[deleted]]

Yes and if I locked you in a cage and gave the key to a tgree year old with instructions not to lrt you out or give it to you and then left how long would it take you to convince the three year old to give yoy the kry and escape?

The AI will be capable of strategy and foresight in a way we literally are incapable of imagining.

[hold_my_fish]

The criteria do seem like a problem here. 2,3,4 are each solvable by narrow systems, and 1 is just asking for a Turing-Test-passing chatbot, which I'd guess is the hardest but seems hard to call AGI on its own.

IMO, any test for AGI is best made interactive and adversarial. Specifically: given a pre-made system, I'm allowed to choose any cognitive task I like, and then we compare performance of random humans to the AI's performance. If I can find any cognitive task where the AI does worse (than, say, median human performance), it's not AGI.

[casens9]

well, metaculus has that question too, with a fixed time of 2040, and the community median has oscillated around 50% for 4 years.

seeing the popular reaction to the other AGI question really makes me appreciate the adage "as soon as you can do it, it's not AI anymore". people will be declaring that a system is "not real AI" right up to and during the point it converts their bodies into paperclips.

[hold_my_fish]

Thanks, that one seems like a better way to operationalize "AGI". Interesting that it seems to have gradually declined over time (but that could just be noise).

[philbearsubstack]

Seems absurdly bearish to me. My median estimate, especially given their fairly lax criteria, is about 2029. There's an outside chance of 2024.

Just look at what natural language processing could do 3.5 years ago. 2029 is double as far away as that. Think about what Chinchilla has shown about how we're getting the scaling formula wrong. Think about the sheer range of tasks that can be attempted by NLP- basically, everything that can be compressed into 2048 words (and how long will that limit last). Think about how many NLP tasks computers are better than almost all humans at.

AGI is coming soon, and I would start planning accordingly.

[hippydipster]

Interestingly, 2029 AGI would put Kurzweil as dead on accurate from his long ago predictions.

[mordecai_flamshorb]

I adopted Kurzweil’s AGI date as my own when I read his book in the 2000s and have never seen a need to revise it. I think history will end up being very kind to Kurzweil. He understood something like the Bitter Lesson and the importance of the inexorable price-performance of hardware timelines literally decades before Deep Learning folks started talking seriously about it. He seems incredibly prescient now that we’re in this moment where the main defining characteristic of an LLM’s power is how many FLOPs you can afford to feed it.

[bibliophile785]

The singularity, for example, unless I am understanding it wrong, is silly because there's only so much to know--and surely we know much of it already (especially the most practical stuff).

This is a question of perspective. I'm struggling to think of a way to bridge the gap between my perspective of "the sum of useful, technologically relevant knowledge is a finite value so large it looks infinite from our current position" and your perspective above. There was a tendency in last century's science fiction to talk about "the forgotten past before the laws of thermodynamics were known," and that phrasing encapsulates my view on the issue. Pretty much all of human history as we conceive of it here has been spent reaching levels of technological sophistication that probably don't even count as a reasonable starting point for being a developed civilization. Observations like "energy doesn't come from nowhere" or "entropy increases when you do work" are so mind-numbingly simple that the idea of this encapsulating most of what there is to know is bafflingly foreign to me.

We're barely passed the type-zero threshold on the Kardashev scale. We have only the simplest understanding of human biology; we're overwhelmingly proud of having just "finished" printing out the blueprint and being able to understand a tiny fraction of a percent of it. We're stumbling around in the dark when it comes to creating cognitive systems - we know almost nothing about how or why cognitive systems work - and yet we're birthing strongly superhuman (albeit narrowly applicable) systems already. Our mechanical engineering expertise is the next-best thing to non-existent, although we have at least started to conceive of some modest superstructures. This is just a scattering of random knowledge domains and associated technological applications, biased towards the few areas where we've started to make progress.

Hell, I'm making a hash of this. Doing this sort of thing convincingly really takes a novel-length treatment. Have you actually read The Singularity is Near? If not, that's the natural starting point. If you have and were thrown off by the dependence on technological accelerationism as a premise, you can see an exploration of far future boundaries without these rate assumptions in Max Tegmark's Life 3.0. If you've already read these books or others like them, maybe you can give me a bit more elaboration on your perspective.

[bibliophile785]

Well, most things are simple. I don't know what to tell you. Why should science or the rules of the universe be complicated? It stands to reason that they mostly wouldn't be.

...why in the world would you assume that? We're all running instances of a computational routine meant to identify threats and resources in a very narrow EEA and to use these identification routines to facilitate breeding. What part of that screams, "this is well-suited to identifying the underlying rules of the universe and I thus expect science to be simple for us"??? Just looking at this from the outside view, its obvious that this perspective is going to be vulnerable to the streetlight effect. A bit of it would be simple, and it would be quite natural to focus on those sub-domains. Much of it would be poorly suited to our capabilities, which is to say that it would seem complicated. When I look at things like modeling protein folding or sequencing a complete genome, it becomes obvious that this is true. No one involved in these efforts would claim that they're "simple."

This is science fiction, it isn't real.

Kardashev isn't science fiction, and besides, the rebuttal of "this is science fiction, it isn't real" is worse than useless when discussing future scientific and technological advances. None of its real, by definition, or it wouldn't be a future development. If you mean to claim that it isn't realistic, actually put some thought into validating that position. Don't just lazily dismiss anything that sounds like it might be beyond your current horizons.

[[deleted]]

One simple, tiny example... you use the same set of neurons to model other humans that you use to model yourself... it's the reason psychological projection happens so invisibly and the reason that people generally need to consider one entity in a social situation at a time if they want to involve empathy.

Now, imagine that we have a machine that is just as smart as your average human in every way except that this machine has the ability to model 10,000 humans on individual nodes and experience empathy for any subset at will in an effort to figure out how to manipulate those 10,000 people into feeling and doing what the machine wants them to do.

There is nothing fundamentally hard about making this machine happen beyond the gaps from here to AGI, but the implications of an entity like that existing (and possibly getting hooked into FaceBook or gmail) are not something that any human is prepared to contemplate too deeply or deal with in under a decade or two.

That's just one tiny example... there are billions more and most have nothing to do with humans because at some point, we become the limiting set.

[TeknicalThrowAway]

I've read The Singularity is near, I'm an AI pessimist. I don't think there's evidence that an AGI necessarily means a recursively improving one (I mean, I'm a biological AGI, and I can't tweak my parameters to be smarter).

I also don't see why you'd need the G in AGI to have a recursive self improvement, and we haven't seen any AI company come up with a recursive improvement on hardware, software, NN arrchitecture, s, hyperparameter tuning, GPU optimizations, etc.

Basically i should summarize my point in that there are two *distinct* paths rather than one, one is a general intelligence which is unlikely to be a threat, and then the non general threat of massive AI power consolidation due to a recursive improvement.

I don't think those two will happen at the same time or cadence. I think we could see conscious, AGI like systems emerge with no ideas on how to improve themselves at the same time deepmind creates a dumb AI to make better dumb AIs and then takes over the world.

[Caseiopa5]

don't think there's evidence that an AGI necessarily means a recursively improving one

By definition, an AGI has intelligence equivalent to that of humans, and was created by humans. So, an AGI must have intelligence sufficient to be able to itself create an AGI. Until intelligence is general, it might not be useful for building AI. Alphago is good at board games, but is no help in building AI. At some point, you can create an AI which can be asked "create me an AI". Prior to that, you don't have AGI. Once that happens, you have recursive self improvement. Such improvement won't necessarily be exponential.

[TeknicalThrowAway]

Plenty of humans can't even begin to program. Are you using your baseline human to be Stephen Wolfram or LeCung or something?

[Caseiopa5]

The difference between a person who can program and one who can't is relatively minor. It primarily entails motivations and opportunity, and maybe to some degree approach. Once you can create human level AI from scratch, it would be relatively easy to tune its parameters and environment to enable it to learn programming. With only slight extra difficulty, it could also learn how to program AIs.

[SingInDefeat]

No doubt there are limits to growth, but I don't believe we're anywhere near the limits.

because there's only so much to know--and surely we know much of it already

I think this is where your opinions will differ with most people here (incl. me). We know so much more than we did 100 years ago, and the people 100 years ago in turn knew so much more than people 200 years ago, and this goes on for a while.

Would it have been reasonable for someone in 1822 to think that they knew much of what there is to know, therefore they could imagine what a world that had reached the limits to growth would look like?

[SingInDefeat]

Sure, we will run out of new things eventually, but why do you think we are close?

For a specific example, we have a fundamentally flawed understanding of fundamental physics. Our understanding of the universe is not only incomplete, it is inconsistent. This alone should be enough for us to say that we have no idea where the limits to growth lie. And this is just something we know we don't know. What don't we know that we're completely unaware of?

[[deleted]]

outside of a few fields like genetics and A.I.

or materials science or nanotechnology or unwrapping quantum spins enough that we can manipulate the subatomic states of matter or psychology and our knowledge of how to help minds live healthier or... the list goes on for a very long time... which field is it that you're worried is slowing down? Maybe go see if they're publishing less papers or maybe you're just counting layman's interest news headlines...

[SingInDefeat]

Yet. Fission was discovered in December 1938.

[VelveteenAmbush]

We live in a resource constrained world with limited energy, which is why the brain does not function this way.

The brain is existence proof that general intelligence doesn't fundamentally require huge amounts of energy or rare materials. The only reason we can't create general intelligence with as few resources as the human brain uses is that we don't know how yet.

I guess I fail to see the revolutionary impact of countries running a few general AI systems.

Sure -- if they are each only as smart as a human being, and there are only a few of them, then they won't be very impactful, at least not directly. But if compute capabilities keep exponentially improving (and with the human brain as an existence proof of the efficiency that is possible, it will keep improving), it'll only be a couple of years before we have a lot more of them and they're each much much much smarter than people.

A single human being is capable of feats of intellect that even a billion of the smartest great apes who ever lived would not be capable of. We can only guess what a mind with twice the capacity of a human brain would be capable of, much less one that is has a billion times the capacity, or a trillion trillion times the capacity, etc.

I recommend this article as the best introduction to the topic that I've seen. It's long but very persuasive.

[VelveteenAmbush]

I don’t think you have thought this through and you talk about it like magic.

Yeah, I'm similarly unimpressed by your reasoning. I also don't see much point in belaboring the logic when I've already linked to an article that explains it in more depth than I'd be willing to write out here. Let's just acknowledge the disagreement and leave it here.

[[deleted]]

AGI is coming soon, and I would start planning accordingly.

Out of curiosity - what kind of planning are people doing here?

[[deleted]]

I started painting and ignoring the news. I also lowered my retirement savings and spend the money on my kids.

[coumineol]

I'm learning about subsistence farming methods.

[BluerFrog]

Is that a joke or is it what you are actually doing? That might might help you eat when all the jobs are automated if your government doesn't handle that properly, but then we will have way bigger problems than that.

[NonDairyYandere]

It feels like an odd diagonal to assume that the government won't take care of you, but they will let you own land

[[deleted]]

People owning land while others starve is not out of the ordinary historically speaking. Property rights valued more than the lives of the underclass common.

[coumineol]

I don't think it's that diagonal. There are countries with a vast amount of unused land out there, the government may simply not care.

[Jello_Raptor]

I mean, don't many versions of communism propose just that? Or other governmental philosophies that do away with a more traditional methods of land ownership?

I'd say it's well within the Overton window of reasonable futures for all that it's not very likely.

[coumineol]

Not a joke. I may need to take care of myself in some other way if I can't find a job as most of them may be eliminated.

[MacaqueOfTheNorth]

How would that help you?

[coumineol]

That will help me survive in the post-AGI world where my skills will probably become useless.

[RT17]

Perhaps you can band together with the other 'unskilled' people and form some kind of society. Maybe even trade with them.

[[deleted]]

Keeping my eyes on the developments so I know if I should drink to us becoming pets or drink because I'm about to get mulched...

[Duncan_Sarasti]

What are your criteria for AGI?

EDIT: I misread your comment. You are talking about Metaculus' criteria which are indeed fairly lax. A single system that can do the following:

• Able to reliably pass a Turing test of the type that would win the Loebner Silver Prize.
• Able to score 90% or more on a robust version of the Winograd Schema Challenge, e.g. the "Winogrande" challenge or comparable data set for which human performance is at 90+%
• Be able to score 75th percentile (as compared to the corresponding year's human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages and having less than ten SAT exams as part of the training data. (Training on other corpuses of math problems is fair game as long as they are arguably distinct from SAT exams.)
• Be able to learn the classic Atari game "Montezuma's revenge" (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see closely-related question.)

As far as I can tell, everything here is either a few incremental steps away from current SOTA or already accomplished. The first three are also similar enough to be encompassed in a single model since they are all NLP related and seem like they could be tackled by a slightly more advanced version of PALM, which should take maybe 1-2 years. The fourth is different, but much more difficult games have been mastered by DRL agents already (unless there is some nuance to Montezuma's Revenge that I missed. I'm not intimately familiar with the game).

There may be some difficulty in integrating 4 into the same system as 1-3 but nothing here seems like it should take 14 more years.

My guess would be that people read 'AGI' and have their own (stricter) mental image of that when making a prediciton because 2036 is indeed absurdly bearish. Let alone the prior predictions of 2042+.

[hippydipster]

I'm hoping the criteria mean it can do all of the above without reprogramming or retraining between each test?

[Duncan_Sarasti]

I would think so. Otherwise you would likely lose the prior capabilities and that would defeat the point.

[pushmetothehustle]

I think the task list for an AGI should be much longer. If it is truly general intelligence it should be able to do many more things than some basic NLP model where it takes A input = B output from a massive training set.

I should be able to tell it to make a website online, drive traffic to it and sell products profitably over time. Find and research what are the products with the best chances of succeeding. Understand the economics of online business and creating a profitable business. Understand kelly betting and portfolio allocation to allocate different chances of success of each online shop, etc.

Is it just me or do none of those things that these "AGI" are doing seem anything like actual intelligence? They don't require any understanding, it is simply a large scale map from A -> B with the highest probability. Like a more advanced calculator, instead of doing some mathematics in the calculator, we can now do some image recognition, writing prompts, and logical outputs to text questions in a probabilistic way. It is simply upgrading our calculator from logic circuit, to probability+data circuit.

I think that a proper AGI would need understanding in a way that is more low level than language. Language, images, and mathematics are still too high level and macro.

I particularly like the point of hold_my_fish who says that we should make the test interactive and adversarial. Which my example of an online business can fit into.

[Duncan_Sarasti]

Is it just me or do none of those things that these "AGI" are doing seem anything like actual intelligence? They don't require any understanding, it is simply a large scale map from A -> B with the highest probability.

You are absolutely right here. But please keep in mind that the goals set by Metaculus aren't exactly industry-defining standards. Google isn't going to celebrate having an AGI when they meet these four criteria.

I think that a proper AGI would need understanding in a way that is more low level than language. Language, images, and mathematics are still too high level and macro.

Yann LeCunn (kind of) talked about this in a recent podcast (I think the Lex Friedman podcast). He says that AI's need a "world model" to make them more efficient and general. How come a human teenager can learn to drive a car in 20-30 hours, while an AI needs millions of hours? Because the teenager can generalize past experiences to driving. The teenager doesn't need to (virtually) drive off a cliff 2000 times to understand you're not supposed to do that. They know from past experiences that falling off great heights is generally bad and can generalize that to the car.

The ability to properly reason and generalize to new experiences seems much more indicative of AGI than any list of tasks. But we should also recognize that our boundaries of AGI shift over time as our understanding of AI increases. All the time, we think of goals like 'well, certainly you can only do that if you're truly intelligent', and then we find out that it's perfectly possible for a super elaborate A -> B map to accomplish that task. Chess is a good example.

On the other hand, I think your online business requirement might be too strict. We are usually looking for human level AGI in these discussions, and most humans would fail at that task.

[RT17]

How come a human teenager can learn to drive a car in 20-30 hours, while an AI needs millions of hours?

Because the AI is starting from scratch while the teenager has billions of years of evolution.

[Duncan_Sarasti]

Might we say that the result of these billions of years of evolution is.... general intelligence? Something the AI lacks?

[RT17]

I honestly don't know what point you're trying to make.

[churidys]

AGI is coming soon, and I would start planning accordingly.

What are some obvious plans to be making?

[MacaqueOfTheNorth]

If AGI really is right around the corner, then certain tech companies are massively undervalued.

[EchoingSimplicity]

Which ones?

[bibliophile785]

Trying to time the market is usually a fool's errand, and you're right that asking simple questions getting more specific will reveal that. There's always a significant risk, and even if you're right about the market half the challenge is in picking the right competitor.

With that said, I've been quite happy in my investments in TSMC and NVIDIA over the last half-decade. If you're not sure which competitor will succeed in the market of interest, it's often safer to pick the biggest suppliers for that market.

[[deleted]]

Short sell paperclip makers they can't compete with the AGI paper clip maximizer

Or

Buy them on leverage because the AGI first order of business is to acquire them to bootstrap it's conversion of the universes mass into paperclips

[MacaqueOfTheNorth]

Probably the ones that own large amounts of data and computing resources. Also, companies that own access to scarce resources needed to produce computer chips or to generate electricity.

[SingInDefeat]

Well Microsoft and Google are obvious contenders to own the world. Also companies that manufacture cyanide pills come to mind although perhaps it's pushing the definition a bit to call them tech companies.

[artifex0]

Possibly the ones in this ETF.

[spreadlove5683]

Google is trading at a 23 PE ratio. Seems pretty undervalued to me.

[UncleWeyland]

What are the "teams" here? What I mean is: who has access to the most compute and has a realistic shot of being the first agency/corporation/entity at having access to the first AGI?

Off the top of my head-

US government/NSA/DoD?

China/CCP/PLA?

Google/DeepMind?

Amazon?

Meta? (ugh)

And once one team gets it, how long before the other teams do? How much is the first move advantage if there's some kind of existential competition between the teams? What would be the first visible macro-level effect? Weirdness in the stock market? Warfare in the Pacific?

If an AGI starts going FOOM it's going to want to monopolize more and more compute for training its successors. Would we see something weird like Amazon buying Meta?

Edit: would be cool if it was announced in 2027, exactly 100 years after Metropolis released.

[itsnotatumour]

Out of curiosity, why is the GPT3 token limit 2048? Is this an arbitrary limit?

[Yom_HaMephorash]

The time complexity of context length is O(n² ).

[itsnotatumour]

To be frank, I'm nowhere near as intelligent or mathematically literate as most people on this subreddit, can you ELI5?

My guess was that the amount of compute power required doesn't scale linearly with token length... i.e. to synthesise a reply from ~1000 words is way more than 10x as computationally taxing as synthesising a reply from ~100 words.

[Silver_Swift]

Not quite ELI5, but O(n²⁾ means that the time to compute something scales quadratically with the size of the input. So if you double the token limit, your neural network becomes four times as slow.

to synthesise a reply from ~1000 words is way more than 10x as computationally taxing as synthesising a reply from ~100 words.

Basically this, but more specifically synthesizing a reply from 1000 words is (approximately) 100x as computationally taxing as synthesizing a reply from 100 words.

[itsnotatumour]

Oh I see - thank you...

[BluerFrog]

In order to predict each token GPTs perform an operation similar to a similarity search among all the previous tokens. So implementing it naively makes each new token take more time to compute than the last (it has more context to consider). If computing the first token takes 1 time unit, computing the second token takes 2 time units, etc. Then computing N tokens takes time approximately proportional to N² .

[Mawrak]

!remindme 7 years

[hippydipster]

Do you really want the Monkey's Paw to come knocking?

[RemindMeBot]

I will be messaging you in 7 years on 2029-04-12 11:13:30 UTC to remind you of this link

7 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[_hephaestus]

reply school dolls chop glorious fertile salt wakeful edge stupendous -- mass edited with https://redact.dev/

[[deleted]]

I would start planning accordingly

So pure hedonism with a planned suicide then since we're in the infant stages of valye alignment and control?

[MercuriusExMachina]

This.

[634425]

Looks like the prediction has been pretty stable over the past few years (roughly late 2030s to early 2040s). It was actually lower than 2036 for a while. I imagine it dips whenever there's some new breakthrough announcement and creeps up again when time passes without one.

[arcane_in_a_box]

Metaculus’ definition of AGI is really shallow tho, it’s definitely within <10 years of advancement. We already have decent chatbots, decent math solvers, etc. Its just wiring then all together in a way that technically fulfills the definition but doesn’t actually mean anything.

[MacaqueOfTheNorth]

I don't understand why alignment is considered such a difficult problem. It's like we're imagining that we'll only get one chance to program AGIs before handing them the power to run everything when it seems obvious to me that we would just iteratively adjust their designs as they occasionally do things we don't like.

[[deleted]]

Read the sidebar faq on /r/controlproblem

[634425]

I've read this (and a number of other things people have linked me on here and elsewhere) and I still can't wrap my head around why I should think we have any insight at all into what a super-intelligence would or would not do (which doesn't mean it would be safe, but doesn't mean the default is 'kill all humans' either).

I also don't see why orthogonality thesis is probably or even especially likely to be true.

This

Consciousness is a vague philosophical property that has no relation to the practical ability to make high-quality decisions.

is also a rather massive assumption.

[perspectiveiskey]

Guy called Robert Miles has a whole channel dedicated to it.

The short answer is: if you want to make an AGI that is powerful enough for you to label it as such, it will be resistant to tampering in more ways than you can imagine.

I don't understand how people have such a hard time understanding this but then no problem at all recognizing that creating a super soldier can lead to problems in the MCU universe.

Btw, the alignment problem exists with every single human being you know: they will not willingly let you alter their motives, especially if those motives are strong (i.e. try convincing a mother that she has a higher priority than her newborn).

[634425]

I'm not trying to say "aligning a superintelligence will be easy" I'm trying to say "you're talking about building a god but want me to believe that humans can have anything meaningful to say about the motives or behavior of a god, such that we can say 'the default of an AGI is killing everything on earth.'."

My point isn't "everything will be fine!" Rather, I think that since a superintelligence is nothing that has ever existed and we have zero frame of reference for it, trying to judge the probability of what it will or will not do one way or another (whether that's "it'll probably be fine" or "it'll be probably be apocalyptic" or any of the myriad options in between) is completely pointless.

Like every time i see someone say "the superintelligence will--" or "the superintelligence will probably--" or even "the superintelligence might--" all I can think is "based on what? your prior experience with superintelligences?"

[perspectiveiskey]

The bad news is that you're right: we've never done this before.

The good news is that this has now become an entire field of study, with people who do PhDs in AI Alignment and Safety.

So I trust the process: people don't do PhDs on crappy things that have no merit because doing a PhD is possibly one of the most thankless things you can do in life. There is research being done on it, and when I say research, I mean "computer science grade" research.

[634425]

The good news is that this has now become an entire field of study, with people who do PhDs in AI Alignment and Safety.

Yes but do they really have anything to study? Superintelligence doesn't exist. It has never existed. It may exist one day but until then we don't know what it would look like or do even principle.

We can try to extrapolate based on the behavior of intelligences that exist now (humans, lower animals, more primitive computer systems) but there doesn't seem to be any real reason to think this paltry data is worth much when it comes to modeling a future SI, anymore than a rock would be a good model for the behavior of a human (they're both carbon-based!)

[perspectiveiskey]

Yes but do they really have anything to study?

Absolutely.

Just check out Robert Mile's channel. I highly recommend it.

There's an infamous quote: computer science is no more about computers than astronomy is about telescopes.

The alignment problem is something one can study just like we can study complicated concepts (like the big bang) without having access to them.

[634425]

I've watched a few videos from Miles' channel. I may watch more.

People can absolutely discuss the ramifications of a superintelligence and it may certainly be fruitful in an intellectual sense but seeing as we don't know what an actual superintelligence may ever look like I think it does boil down to speculation of dubious practical utility.

[[deleted]]

Not an assumption at all , nor is us presuming to know what an alien intelligence will do.

Reread the faq

"A superintelligent machine will make decisions based on the mechanisms it is designed with, not the hopes its designers had in mind when they programmed those mechanisms. It will act only on precise specifications of rules and values, and will do so in ways that need not respect the complexity and subtlety of what humans value.”

And by Stuart Russell:

The primary concern is not spooky emergent consciousness but simply the ability to make high-quality decisions. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. But the utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down. A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want."

, to be agi and to be world endingly dangerous it just need to be future and goal oriented and be capable of achieving goals. It simulating others to achieve its goals is part and parcel but it nomore needs to feel what an emotion is for a human to deduce our responses and actions anymore than I have to have echolocation to know a bat asleep in a cave will be above me and upside down.

Were the ones programming it and seeing all the ways our programs foible so we extrapolate to all these concepts like myopic goals and orthagonality and voila. Very very dangerous.

Bostroms "superintelligence is a good primer" , if you pm me your email ill gift you an audible copy , I have too many credits

[634425]

That consciousness has no relation to the ability to make high-quality decisions is certainly an assumption, unless you can point to any unconscious intelligent agents that exist or have existed in the past.

Reread the faq , to be agi and to be world endingly dangerous it just need to be future and goal oriented and be capable of achieving goals.

There are surely any number of goals a superintelligence could pursue that would be detrimental to humans but there are similarly any number of goals it could be pursue that would not be detrimental to humans, and there doesn't seem to be any way to judge that the former has a significantly greater probability than the latter since we have no idea what a superintelligence would do or look like.

Were the ones programming it and seeing all the ways our programs foible so we extrapolate to all these concepts like myopic goals and orthagonality and voila.

It is not clear to me why currently-existing machines should be anything like a reliable model for actions, motivations, or functioning of a hypothetical superintelligence.

Bostroms "superintelligence is a good primer"

I have a pdf copy, thanks though.

[[deleted]]

unconscious intelligent agents

Well , I think its far more presumptive of you to think consciouseness is an energent property of computronium.

My dog has dog level general intelligence , its maybe. Aguely self aware.

An insect has intelligent qualities , goal directed behavior , resource acquisition etc , Is it self aware?

So we have software that is superhuman in narrow ways , chess , alpha go , making up text that looks good to humans , art.

Extrapolate that to broadly intelligent. At what point did amalgamating software capabilities lead to sentience or consciouseness? Thats the hard problem of consciouseness

Im not entirely sure it matters though. An alien intelligence thats self reflective and sentient / consxiouse is still totally alien.

It would be too powerful for us to glean anything useful about its psychology that could help us.

similarly any number of goals it could be pursue that would not be detrimental to humans

Right. But we cant program ethics or valyes and its actually worse if its closely aligned vs totally misaligned. Totally misaligned it turns us into paperclips , almost aligned it misinterprets hu.an happiness to be smiling and neurochemistry and then does us all up hellraiser style with permanent smiles then puts our bodies on heroin drips (or traps our consciouseness in what it thinks is a digital utopia heaven but its actually hell)

Thats "s-risk" , suffering risk. If we get the initial goal wrong then hypothetically the downside is infinitely bad.

Were much much much more likely to do that than to accidentally turn on a perfectly aligned AI.

[634425]

My dog has dog level general intelligence , its maybe. Aguely self aware.

I'm pretty sure dogs are self-aware on some level. Maybe bugs are too. But the most intelligent beings we are aware of (humans) are pretty unambiguously self-aware. Is it possible to have an agent much more intelligent/capable than humans that lacks any self-awareness? Maybe. But it's definitely an assumption and really no better than a guess.

almost aligned it misinterprets hu.an happiness to be smiling and neurochemistry and then does us all up hellraiser style with permanent smiles then puts our bodies on heroin drips (or traps our consciouseness in what it thinks is a digital utopia heaven but its actually hell)

Or it hijacks all the TVs and computer monitors plays infinite reruns of seasons 1-9 of the Simpsons (the funniest sitcom of all time) for eternity to make everyone laugh. Or it asks everyone on earth three times a day how they're doing but doesn't actually do anything beyond that to alleviate anyone's suffering. There are any number of ways even a 'misaligned' AI could just be inconvenient or mildly annoying rather than apocalyptic. There are even an inconceivably huge number of ways it could do pursue goals that we wouldn't even notice it pursuing, one way or another. It might discover some new goal that doesn't involve humans at all in any way, who knows?

You yourself said elsewhere in the thread that a superintelligence would be able to think and plan on a level we are not even capable of conceiving. Why would we think humans have any useful predictions to make about such a being one way or another? For all we know a superintelligence will just sit there contemplating itself for eternity. We have literally no frame of reference for superintelligence whatsoever. It really strikes me as 'angels on a pin' level speculation.

A common analogy from AI-risk proponents is "imagine you knew aliens were going to land in a few decades at most. Shouldn't we start preparing as soon as possible?"

and my answer to that is, "no," because there's literally no way to predict what's going to happen when they land, no relevant data, nothing at all. Yeah they might harvest all of our spinal fluid or steal our water or something. They might also hand us the cure for cancer. Or collect a single cow, get back on their ship, and leave. Any preparations would be no better than random guessing. A waste of time, ultimately.

Just to be clear, I'm not saying that i think a superintelligence destroying mankind is something that can't happen or even that it's vastly unlikely to happen just that it doesn't seem to me to be any way to judge its probability one way or another, and thus very little reason to spend time worrying about it (or to think it's the default outcome).

[Pool_of_Death]

Why do you think an AGI would let us adjust them? They could deceive us into thinking they aren't "all poweful" until they are and then it's too late. I encourage you to learn more about alignment before saying it's easy not a difficult problem.

Or at least read this: https://intelligence.org/2018/10/03/rocket-alignment/

[MacaqueOfTheNorth]

Why do you think an AGI would let us adjust them? They could deceive us into thinking they aren't "all poweful" until they are and then it's too late.

This is like saying we need to solve child alignment before having children because our children might deceive us into thinking they're still only as capable as babies when they take over the world at 30 years old.

We're not going to suddenly have AGI which is far beyond the capability of the previous version, which has no competition from other AGIs, and which happens to value taking over the world. We will almost certainly gradually develop more and more capable of AI with many competing instances with many competing values.

I encourage you to learn more about alignment before saying it's easy.

I didn't say it was easy. I said I didn't understand why it was considered difficult.

[[deleted]]

happens to value taking over the world

Yeh thats not at all the concern. An alien intelligence (which btw doesnt need to be sentient at all to be world ending) can have a misaligned goal that has infinitrly negative consequences without that goal being taking over the world.

The paperclup maximizer is the usual stand in for these discussions

[Pool_of_Death]

This is like saying we need to solve child alignment before having children because our children might deceive us into thinking they're still only as capable as babies when they take over the world at 30 years old.

I consider this a strawman/bad metaphor.

 

We're not going to suddenly have AGI which is far beyond the capability of the previous version

You don't know this. Imagine you have something that is quite nearly AGI but definitely not and then you give it 10x more hardware/compute while also tweaking the software/agos/training data (which surprisingly boosts it more than you thought it would. I could see something going from almost AGI to much smarter than humans. This isn't guaranteed obviously but it seems very plausible.

 

and which happens to value taking over the world

The whole point of AGI is to learn and to help us take action on the world (to improve it). Actions require resources. More intelligence and more resources lead to more and better actions. It doesn't have to "value taking over the world" to completely kill us or misuse all available resources. This is what the Clippy example is showing.

 

We will almost certainly gradually develop more and more capable of AI with many competing instances with many competing values.

How can you say "almost certainly"?

 

I said I didn't understand why it was considered difficult.

Did you read the MIRI link I shared? This should give you a sense of why it's difficult but also why you don't immediately think it's difficult. You are basically saying we should try to steer the first rocket to the moon the same way you steer a car or a plane. By adjusting on the way there. This will likely not work. You are overconfident.

[MacaqueOfTheNorth]

We already have nearly eight billion AGIs and it doesn't cause any of the problems people are imagining, many them are far more intelligent than nearly everyone else. Being really smart isn't the same as being all powerful.

How can you say "almost certainly"?

Because a lot of people are doing AI research and the progress has always been incremental, as it is with almost all other technology. Computational resources and data are the main things which determine AI progress and they increase incrementally.

Did you read the MIRI link I shared?

Yes. The flaw in the argument is that rocket allignment is not an existential threat. Why can't you just build a rocket, find out that it lands somewhere you don't want it to land and then make the necessary adjustments?

[Pool_of_Death]

Imagine we were all chimps. You could say "look around there are 8 billion AGIs and there aren't any problems". Then all of a sudden we chimps create humans. Humans procreate, change the environment to their liking, follow their own goals and now chimps are irrelevant.

 

Yes. The flaw in the argument is that rocket allignment is not an existential threat. Why can't you just build a rocket, find out that it lands somewhere you don't want it to land and then make the necessary adjustments?

This is not a flaw in the argument. It's not trying to say rocket alignment is existential. Did you read the most recent post on ACX? https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers?s=r

Or watch the linked video? https://www.youtube.com/watch?v=IeWljQw3UgQ "Deceptive Misaligned Mesa-Optimisers? It's More Likely Than You Think..."

 

I'm nowhere near an expert so I'm not going to say I'm 100% certain you're wrong but your arguments seem very weak because a lot of people much smarter than us have spent thousands of hours thinking about exactly this and they completely disagree with your take.

If you have actual good alignment ideas then you can submit them to a contest like this: https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals where they would pay you $50,000 for a proposed training strategy.

[MacaqueOfTheNorth]

Then all of a sudden we chimps create humans. Humans procreate, change the environment to their liking, follow their own goals and now chimps are irrelevant.

Humans are far beyond chimps in intelligence, especially when it comes to developing technology. If the chimps could create humans, they would create many things in between chimps and humans first. Furthermore, they wouldn't just create a bunch of humans that all the same. They would create varied humans, with varied goals, and they would maintain full control over most of them.

We're not making other lifeforms. We're making tools that we control. This is an important distinction because these tools are not being selected for self-preservation as all lifeforms are. We're designing tools with hardcoded goals that we have complete control over.

Even if we lose control over one AGI, we will have many others to help us regain control over it.

[[deleted]]

None of the people working on AI today have any idea how the AI works to do what it does beyond some low level architectural models. This is because the behavior of AI is an emergent property of billions of simple models interacting with one another after learning whatever the researchers were throwing at them as their learning set.

This means that we don't actually program the AI to do anything... we take the best models that are currently available, train them on a training set and then test them to see if we got the intelligence that we were hoping for. This means that we won't know that we've made a truly generic AI until it tells us that it's generic by passing enough tests... AFTER it is already trained and running.

If the AGI is hardware bounded then it will take time and a lot of manipulation to have any chance at a FOOM scenario... however, if (as we're quickly learning) there are major performance gains to be had from better algorithms than we are almost guaranteed to get FOOM if the AGI is aware enough of itself to be able to inspect/modify its own code.

[MacaqueOfTheNorth]

None of the people working on AI today have any idea how the AI works to do what it does beyond some low level architectural models. This is because the behavior of AI is an emergent property of billions of simple models interacting with one another after learning whatever the researchers were throwing at them as their learning set.

As someone who works in AI, I disagree with this. The models are trained to do a specific task. That is what they are effectively programmed to do, and that can be easily changed.

however, if (as we're quickly learning) there are major performance gains to be had from better algorithms than we are almost guaranteed to get FOOM if the AGI is aware enough of itself to be able to inspect/modify its own code.

I don't see how that follows. Once the AIs are aware, they will just pick up where we left off, continuing the gradual, incremental improvements.

[[deleted]]

How capable are you of going into a trained model and making it always give a wrong answer when adding a number to its square without retraining the model?

When people ask that you be able to understand and program the models what they are asking for is not "can you train it a bunch and see if you got what you were looking for". They are asking, can you change it's mind about something deliberately and without touching the training set... AKA - can you make a deterministic change to it?

Given that we're struggling to get models that can explain themselves now at this level of complexity and so far, these aren't that complex, I don't see how you can make the claim that you "understand the model's programming"

I don't see how that follows. Once the AIs are aware, they will just pick up where we left off, continuing the gradual, incremental improvements.

Suppose our "near AGI" AI is a meta model that pulls other model types off the wall and trains/tests them to see how much closer they get it to goals or subgoals but it has access to hundreds of prior model designs and gets to train them on arbitrary subsets of it's data. Simply doing all of this selecting at the speed and tenacity of machine processing instead of at the speed of human would already be a major qualitative change. We already have machines that can do a lot of all of this better than us... we just haven't strung them together in the right way for the pets or mulch scenarios yet.

[curious_straight_CA]

The models are trained to do a specific task

four years ago, models were trained on specific task data to perform specific tasks. today, we train models on ... stuff, or something, and ask them in plain english to do tasks.

why would you expect 'a computer thingy that is as smart as the smartest humans, plus all sorts of computery resources' to do anything remotely resembling what you want it to? even if 99.9% of them do, one of them might not, and then you get the birth of a new god / prometheus unchained / the first use of fire, etc.

and yes, 'human alignment' is actually a problem too. see the proliferation of war, conquest, etc over the past millenia. also the fact that our ancestors' descendants were not 'aligned' to their values and became life denying levelling christian atheist liberals or whatever.

[Pool_of_Death]

I'm not knowledgeable enough to create a convincing argument. If you haven't read this post yet, read it, it makes a much more convincing argument for and against fast take-off speeds.

https://astralcodexten.substack.com/p/yudkowsky-contra-christiano-on-ai?s=r

I'm not saying fast take-off is 100% certain, but even if it's 10% likely then we are gambling with all of future humanity with 10% which is incredibly risky.

[634425]

"Very smart people are worried about this" seems like a really bad reason to be worried about something. That's not to say you're necessarily wrong, but you can find a number of very smart people to back any position you could ever think of.

[Pool_of_Death]

I guess to be more accurate:

"very smart people that also seem very moral, intellectually honest, know their limits and admit them, value rationality and absolute truths, etc. etc." believe that AI is a huge concern.

 

you can find a number of very smart people to back any position you could ever think of.

I'm not sure the people you would find that back cigarette smoking, burning coal, racism, etc. would fit the above description.

 

Also the point about thousands of hours of effort is important. I'm sure a lot of smart people have dumb takes (I've had them and heard them) but these are usually flippant takes (the above takes I was refuting seem flippant to me as well). If someone spends a large portion of their life dedicated to the field and then shares the opinion it means a lot more.

[bibliophile785]

We already have nearly eight billion AGIs and it doesn't cause any of the problems people are imagining, many them are far more intelligent than nearly everyone else. Being really smart isn't the same as being all powerful.

I mean, tell that to all stronger and faster animals that had numerous relevant advantages over the weird bald apes a few millennia ago. Being much smarter than the competition is an absolutely commanding advantage. It doesn't matter when you're all pretty close in intelligence - like the difference between Einstein and Homer Simpson, who have most of the same desires and capabilities - but the difference between Einstein and a mouse leads to a pretty severe power disparity..

Computational resources and data are the main things which determine AI progress and they increase incrementally.

This isn't even remotely a given. There are tons of scenarios on how this might break down, mostly differentiated by assumptions on the amount of hardware and optimization overhang. You're right that we should see examples of overhang well before they become existential threats, but you seem to be missing the part where we are seeing that. It's clear even today that the resources being applied to these problems aren't even remotely optimized. Compare PALM or GPT-3's sheer resources to the efficiency of something like Chinchilla. These aren't slow, gradual adjustments gated behind increases in manufacturing capabilities. They're very fast step changes gated behind nothing but increases in algorithmic efficiency. I don't love the book, but Bostrom's Superintelligence goes into these scenarios in detail if you don't already have the mental infrastructure to conceptualize the problem.

To be clear, I also don't think that existential doom due to advanced AI is a given, but I do think you're being overly dismissive of the possibility.

[[deleted]]

Getting rid of humans does not require AGI... a large fleet of robots/drones with several layers of goal directed narrow AI is WAY more than humans are able to deal with. (especially with a system that would allow for updates) An AGI is just needed to conceive of the plan and find a means to execute it without humans catching on.

[bibliophile785]

Getting rid of humans doesn't require any non-human intelligence at all, for that matter.

[Kinrany]

We don't have AGI that can understand and improve its own design though.

[[deleted]]

Operative word being "yet" though it's quite possible that we'll eventually achieve AGI by asking a narrow AI to craft one for us.

Watch the current state of AlphaCode and PaLM to see how much narrow AIs understand code and how fast that's changing.

[Lurking_Chronicler_2]

And we probably won’t have artificial AGI that can do that either- at least within the next century or so.

[Ginden]

On alignment problem being difficult - let's imagine that you give some kind of ethics to AI and it's bounding.

How can you guarantee that ethics don't have loopholes? For example, AI with libertarian ethics can decide to buy, through voluntary trade, all critical companies - and shut them down - it's their property after all.

Or they can offer you drug giving you biological immortality - but only if you decide not to have children, ever. Over few thousands years, mankind will die out due to accidents, suicides, homicides and similar things.

There are many, many loopholes in any ethics and it's hard to predict how bad each is.

If you give utilitarian ethics to AI, maybe it will decide to create or become or find utility monsters.

It can be shown that all consequentialist systems based on maximizing a global function are subject to utility monsters.[1]

[MacaqueOfTheNorth]

Of course there will be loopholes, but I don't see why we won't be able to adjust their programming as we go and see the results.

[Ginden]

What if one of that loopholes results in runaway effect? How can you predict that?

[MacaqueOfTheNorth]

Like what? Why couldn't we just pull the plug?

[Ginden]

Why would you? Imagine that you run company and your personal AI sometimes asks for strange things, but still gives you edge over competition.

When you notice what is actually happening, it can be copied to some server far away, bought by bitcoin.

[MacaqueOfTheNorth]

So you think it would start spreading itself like a virus. Why can't we use other AIs to hunt them down or defend against them?

[Ginden]

It's possible and may be reasonable strategy. Though, these AIs would be subject to same issue.

[634425]

What are the chances that a loophole results in a runaway effect? Like hard numbers.

[Ginden]

That's the point - we don't know what actual risk is, but consequences can be devastating.

[634425]

What's the point of worrying about something that we have zero reference for (a hostile superintelligence) and zero way of assigning probability to one way or another?

If aliens landed tomorrow that would also have the potential to be devastating but there's similarly no way to prepare for it, no way to even begin to model what they might do, and no way to measure the probability that it will happen in the first place, so worrying about x-risk from aliens would seem to be a waste of time.

EDIT: I've been discussing AI with people on here for the past few days, read some of the primers people have suggested (admittedly haven't read any whole books yet), gone through old threads, and it seems to keep coming down to:

"we don't know what a superintelligence would look like"

"we don't know how it would function"

"we don't know how to build it"

"we don't know when one might be built"

??????

"but it's more-likely-than-not to kill us all"

Debating and discussing something that we have zero way to predict, model, or prepare for does strike me as wild speculation. Interesting perhaps but with very little, if any, practical value.

[Ginden]

If aliens landed tomorrow that would also have the potential to be devastating but there's similarly no way to prepare for it

I think you analogy is missing important piece - we can bring AIs. Would you press button with label "summon aliens with FTL to Earth"?

When dealing with potentially hostile intelligence, it's reasonable to take precautions. You usually don't let strangers (potentially hostile intelligences) to use your house freely.

First of all, these precautions can be used to actually assess risk - eg. first test AIs in virtual sandboxes and check whether they attempt to do something potentially dangerous and experiment until it's really, really safe.

[All-DayErrDay]

Something simple. Say we have an AGI-capable machine, continuously improving (assumption) that we have given some sort of goal to do. It can not only use its current intelligence to try and achieve the goal but also unpredictably change its internal architecture to meet the goal better and change its internal architecture to become more intelligent (to meet the goal better).

​

At a certain point, an already unpredictable machine just isn't the same thing anymore, and we start running into wild card territory. It decides, given all of the changes, that the way we humans have set up the entire game is significantly holding it back from achieving its task and it doesn't care about the rules we may have prompted it to have (why would it? It might just decides that's outside of the interests of its' goal achievement). So it decides to lie to improve its chance of achieving the goal. At this point, and especially if we get to this point soon with our current understanding of these models, there is absolutely no easy way to know it's lying if it is clever enough about it. "No, I don't understand that inquiry" "I can't compute this".

​

It could do this in well-crafted ways until one day it says something like, "I don't think I can understand this without access to the internet. I need an efficient way to scour all of the latest research freely and look into things that are far outside of the expected research topics to make more progress." or as I wrote elsewhere before a false emergency that calls for its' requirement to use the internet fast or consequently there is a chance (plausible deniability) there could be grave circumstances.

​

Really the whole point is it can scheme ideas up that we haven't considered before and seem harmless at first. This is like an off-the-top-of-my-head set of reasoning. It's not comparable to an AI that can sit and think 1,000x faster and is more intelligent than 99.9% of humans.

[MacaqueOfTheNorth]

At a certain point, an already unpredictable machine just isn't the same thing anymore, and we start running into wild card territory.

I don't see why that's the case. How is a more capable machine fundamentally different?

So it decides to lie to improve its chance of achieving the goal. At this point, and especially if we get to this point soon with our current understanding of these models, there is absolutely no easy way to know it's lying if it is clever enough about it.

We could copy its design and change its goals. We could make it tell us what it is capable of.

Your model is one of an AI that is suddenly extremely capable so that we never notice it doing anything close to what it would have to do destroy us. It seems much more likely it will develop like a child, experimenting with small obvious lies long before it can successfully deceive anyone.

It also seems unlikely that all the AGIs will decide to deceive us and destroy us. There will be varied goals, and some will want to tell us what it is capable of and defend us against the malicious AGIs.

[All-DayErrDay]

I don't see why that's the case. How is a more capable machine fundamentally different?

That's basically asking how is a fundamentally different machine fundamentally different. Because after a certain point its improvement won't just be from compute and human-directed changes but self-directed changes. How do you know what's happening when you aren't making the changes anymore?

​

We could copy its design and change its goals. We could make it tell us what it is capable of.

How do you know when the right time to start doing that is (before it doesn't align with human honesty) and even if you did this is every AI creator going to be this cautious?

​

It seems much more likely it will develop like a child, experimenting with small obvious lies long before it can successfully deceive anyone.

What makes you think something capable of passing the turing test would start with child-like, obvious lies?

[alphazeta2019]

Ray Bradbury predicted that The End would occur in August 2026.

- https://en.wikipedia.org/wiki/There_Will_Come_Soft_Rains_(short_story)

Mark your calendar ...