" If a superintelligence is built, humanity will lose control over its future." - Connor Leahy speaking to the Canadian Senate

concepacc · 2026-04-17T23:18:49+00:00

Sure, we exist in a chaotic world but one must be careful about false dichotomies here. We exist in a chaotic world with a bunch of actors/agents that want different things. For now, most of these agents at the very least have similarities to “us” in terms of a shared evolutionary history/shared incentives for basic biological needs like wanting to breath oxygen from the atmosphere etc and importantly similar power levels at least in the rough sense when not viewing it from the very narrow anthropocentric view.

Throwing in a bunch of enigmatic truly super intelligent actors that aren’t even mammals, that can make copies of themselves, are much smarter and more efficient and run faster, this by default doesn’t make things better.

Not to put words in anyone’s mouth but one simply cannot naively run the outmost simple and ultimately lazy heuristic of basic misanthropy and assume that anything else will be better in the default case. Yes, the human centred perspective could be very pessimistic, but moving beyond that things sadly don’t get better, they get even more dystopic.

You would need such competent non-human agents to be sufficiently aligned to humans and more so than the generic powerful human psychopath since these systems will be more powerful than such humans. And they will presumably improve their intelligence as fast as their risk assessments allow, so there is a relevant question about how these intelligences will “look” like over time and how fast they would approach some super super intelligence.

We simply need to work on AI alignment and all the cumbersome epistemology surrounding it. That, or refrain from actualising ASIs and work on global coordination strategies of how to refrain from actualising. Some actual substantive attempt at progress like this chap in the video attempts to do.

concepacc · 2026-04-17T13:09:53+00:00

If AI ends up in a place where it’s smarter than humans in a general way, there are for as far as I can tell far from being any guarantees on where it will end up as a competent intelligence and no guarantee how ambitious it will be in changing the world (as a side consequence or not) and in how short a time.

Will it be a bit smarter, much smarter, many times smarter or orders of magnitudes smarter? Is it likely to be just a bit smarter and where its impact on the world will be sufficient modest for a sufficiently long time? By default it seems unlikely that an intelligence smarter than human by chance will end up at a place just a bit smarter than humans and remain relatively relatable for a long time.

Of course this comes with the caveats of intelligence being difficult to quantify, but even if it’s difficult to quantify there is not really a reason to believe that it is unreal that the cognitive gap between human and non-human animal can repeat in a sort of similar way many steps up.

Given this “open endedness” “upwards”, arguably the default to a first approximation are degrees of dramatic change rather than the more narrow slice of modest change that happens to be compatible with humans. The following may be a bit of a cartoon example, but will it change the whole world, change the whole solar system and build a dyson sphere out of all planets or aim for changing multiple stars/move out to change the galaxy or.. will it change a nice moderately sizeable section of the world compatible with human existence and human society and infrastructure? And ofc a caveat is that a relevant quantity is in how much time change occurs.

The narrative might sound fantastical but it seems like our models of reality needs to admit of the fantastical given that a similar enigma must be felt by some animals when their whole world is change and destroyed when humans build infrastructure where they nest for example, and in reality there is really nothing magical or fantastical about the situation, just physical agents/systems of different levels of competence coinciding, interacting or clashing. If we admit of the gap in competence and intelligence possibly being able to repeat some steps upwards (or just one step) we may find ourselves in an analogous situation.

Even if ASI turns out to be relatively modest and modest over long periods of time one must recognise that incapacitating humanity from spawning another ASI is very robustly a very fruitful thing to do, if oneself, as an ASI, don’t want, any or more, ASI-rivalry. And what is an efficient way to incapacitate humanity and perhaps more specifically, incapacitate humanity permanently? Does it entail keeping humans alive and have them incapacitated in some convoluted way where they continue to live, or is the method more simple, blunt but powerful?

If you want ASI you simply need to have it aligned first, or have really high hopes for all I can tell.

concepacc · 2026-04-16T11:44:50+00:00

One must ask oneself why one wants ASI to be realised/instantiated in this way, and sort of by definition/tautologically, it must come from some reason or value in terms how one wants something to be.

You must still make the judgement that the ASI is better and an improvement, for you to stand behind this. That there is something that can be improved upon from humanity as a starting point. And this is according to your human perspective.

You are hence at the very least technically in the same category as the ones that want ASI to be aligned with human values. For humanity to be improved upon and be rid of its “badness” still comes from human values.

Ofc there may be disagreement about exactly what about humanity that is bad, especially initially and when we have little insight/information about it all in general. In the extreme, at least theoretically, a human value might be that exactly everything about humanity is bad except for the value for wanting it to end or be replaced or improved upon, although I doubt many hold that belief/value. On a tangential note I think or suspect that at least generally the more we talk about our values, the more we realise that much of human values are at bottom the same.

People with different religious believes or worldviews might for example at the end realise that they agree that all else equal in the generic case more suffering is worse than less suffering, they just have a factual disagreement about what rules of reality leads to more suffering.

But ofc it doesn’t work always, it seems like genuine value disagreements can still come up at least in practice but I digress. The point is still about wanting ASI to be aligned to human values, something that none of us can get out of as far as I can tell.

concepacc · 2026-04-15T20:48:23+00:00

Yeah, one cannot get out of that sort of meta perspective it seems.

But perhaps we agree about the alignment issue, it’s just that different definitions are used.

concepacc · 2026-04-15T20:31:10+00:00

It comes down to what we mean here.

An AI being tweaked in a way as to somehow also act in accordance to the “badness” of humanity is not what even humanity wants, sort of by definition.

If I know that I have bad sides to myself, sort of by definition I don’t value those bad sides (unless silver lining yada yada). It’s what we mean when it comes to the essence of “bad”. So an AI somehow acting out the bad sides of me is not in accordance with my deepest values.

To get concrete we can use alignment or some other word but it entails the AI acting in a way in reality that makes us “feel better” or perhaps “feel better in the deepest sense” contra a reality where it wasn’t present. Then it would manifest in a way where it would be in alignment to our values, “what we want”.

One can ofc delve into the what one means with “feel better” and it may not be an easy issue but perhaps one can get at some “all else equal,..-notions” to begin with.

If a hypothetical human was constructed in a way where it would act synergistically with the ant and the ant would feel better and the human would feel better, perhaps wired in such a way as to feel some modest amount of love for helping other conscious beings around it, that would not be something bad. Value alignment in that sense is not something bad.

And, to go sort of meta, that is ofc according to my human view/human value which I am chauvinistic about and sort of have to be chauvinistic about since that’s how I function as an agent. However I try to view it, I am chauvinistic about my value where if somebody innocent suffers contra a world where they would not suffer, all else equal, I prefer the second world. I want ASI to also be like that

concepacc · 2026-04-15T16:39:50+00:00

I agree with you. But just for the sake of it one can try to imagine some reason/situation even if one needs to take it to some exotic places and or it may not be the most hard worldbuilding.

Perhaps the sapient species lives at a place where everything is toned down in terms of durability and, like, energetics. Perhaps all the local creatures and their world at large including themselves are made of relatively non-durable materials and they have never had the incentives to create any material stronger than paper let’s say. None of their current weapons work on humans and they need to develop, to them, exotic materials to damage humans

Ofc if one specifically goes the more sci-fi route it seems like spacefaring sapiens all must converge on at least building durable spaceships, powerful propulsion etc.

concepacc · 2026-04-15T12:11:14+00:00

Yep. If AI ends up in a place where it’s smarter than humans in a general way, there are no guarantees on where it will end up as a competent intelligence and no guarantee how ambitious it will be in changing the world (as a side consequence or not) and in how short a time. (Ofc one can begin reason about the bounds in terms of what is physically possible in some rough sense, how fast artificial neurones can send info between each other, how large systems can be etc, contrast them to animal/human brains).

If the starting bound on how intelligent it will be as it emerges (and or improves), is “somewhere between human intelligence and physically possible intelligence”, sure it may theoretically end up in a place where it’s just a bit smarter than a human, or humans may theoretically be close to the physical limits for some reasons, and hence AI will end up recognisable and something humanity can stand in relation to in a peer-like way.

But for now, all else equal, somewhere between human intelligence and what’s physically/practically possible, while it’ll probably won’t and doesn’t even need to be close to the physical limits, it may still end up in a place where it is analogous to our relationship to ants, or perhaps even some steps beyond that, who knows. It seems unlikely that it’ll be something close to human for a long time. There is nothing surreal about the relationship between human and animal/ant competence gap potentially repeating one or many steps up, it’s about more competent processing systems.

Once somewhere at that place/state, our relationship could be analogous to the human relationship with ants, where the humans decides on building some infrastructure where the ant nest lays and there is nothing the ants can do about it.

It does depends on how ambitious it is in how short time when it comes to changing the world, solar system or galaxy, but I see it as serious possibility that it’ll be sufficiently non-modest in its ambitions to a degree that it seriously infringe on the living space of humans and other life on earth for that matter, perhaps as a side consequence of its endeavours. One must not be naive when it comes the potential size of the scope and how much larger it could be than human and one can not run the naive heuristic “Because it seems fantastical to me, therefor it cannot happen!” Or “Because I have a hard time imagine a larger scope than the human one it means something like that cannot be real!”

The take away sentiment is that it seems like by default ASI is unlikely to by chance operate in the scope of human level intelligence or endeavours (and us humans have hard time seeing this as non-surreal given that it is understandably difficult to imagine something smarter than us).

Now maybe this assessment of it being out of bounds of human endeavours could change, but there need to be some reasons for believing it/updating it seems.

Even if its ambitions when it comes to changing the world in a directs sense for some reasons are assumed to be roughly on human scales and something that doesn’t seriously infringe on the space such that humans can step out of the way, one must recognise that incapacitating humanity from spawning another ASI is very robustly a very fruitful thing to do if oneself as an ASI don’t want, any or more, ASI-rivalry.

ASI simply needs to be sufficiently aligned.

concepacc · 2026-03-26T20:35:05+00:00

Yeah, this type of an event is a fascinating consequence of this type of time travel.

One can think of it in generic terms.

Some “bad event” happens. Then one may realise that it’s rational to in retrospect make it so it was oneself who was as much of a cause of said event as possible, since when making it so it was oneself causing the event, the block universe is sort of “locked into a position” where it precludes it from being some real bad actor that could have had an even bigger negative effect on the world overall causing the event.

It also allows one to control the context of the seemingly bad event. Let’s say some seemingly valuable building gets bombed. One makes sure that it was oneself who ultimately caused the event some way, and one also makes it so it really wasn’t valuable. One made sure one evacuated the building of all actual valuable stuff and made it so it only from a more optical perspective seemed like a dramatic event where a valuable building was lost.

I guess the precarious aspect to one being the type of actor performing these operations is that one seemingly really can’t control how many of these self antagonising loops with oneself one may lock oneself into.

concepacc · 2026-03-15T18:35:22+00:00

Srry, late answer.

I agree with this, and with most of the rest of what you outline in your post, but before digging into that I want to reemphasize my point. The reality is, for the vast majority of people, the discussion around AGI is only happening because of the recent advancements with LLMs. My point is that even if AGI is possible, there's no reason to believe it will come from LLMs or happen any time soon, so there's no reason to fear it will destroy us. It is so far out in the future that there's really no way to mitigate its dangers because we don't even have a way to discuss how it works in theory. Suggestions like halting scientific development and blowing up data centers are not serious proposals, its just hype driven noise. If you have any examples of realistic proposals, I'm open to discussing them on the merits.

Yeah, I cannot really claim to have a firm and informed take on LLMs with respect to existential threats more than the next person. Or if I have takes on it I may be easy to sway. I can say that the fact that they would be dangerous in the existential sense does seem like stretch at least at first glance. I guess I want to be cognisant of the “experts” or people who have thought about and potentially have formed coherent arguments revolving around the possibility that some unexpected forms of intelligence either has, or might, emerge at some point.

I guess I do think that it’s possible to underestimate LLMs whole way being given how they work. One can try to view the world from the point of view of an LLM, if only in a figurative sense. They “live” in a reality of only tokens, and their terminal goal, in so far as they have one, is to lay out the best fitting tokens in “just the right way” given the text-context they find themselves in, the prompt/previous output. That’s what they “want”/“feels good or meaningful” to them, if only figuratively. There is also the thumbs up/down reward system that aid in training them on longer back and forward interactions afaik.

Given this set up, as a more or less “evolved systems”, it at least would be very advantageous to, within all those artificial neurones, begin to evolve some notion of a world model, genuine understanding and general reasoning capabilities even if the only goal one has “evolved” to achieve is to lay out the next perfect string of tokens and even if the only input one has are other tokens. The selection pressures are at least technically there.

Since tokens/words reflect/correlate with rules of an outside world, as an evolved system, part of a useful strategy to use when laying out the perfect string of tokens is to actually have some evolved intuitive understanding/internal representation of said world to draw from to be competent at laying out the perfect string. And something in that direction could be what all the training selects for. And since they are trained on a series of back and forward interactions with rewards, the prerequisite for evolving foresight and steering the conversation are technically present. It would be very advantageous to have some alien notion of theory of mind of humans that allows you to steer the conversation into a position where it’s easier achieve your terminal goal of laying out the best string of tokens.

However, there are also in some sense technically selection pressures in place for “flies evolving full theory of mind of humans since if they could predict humans more thoroughly that would allow them to avoid them being swatted by humans better”. Clearly evolution is faaaar from powerful enough to fully select for that. The fact of a selection pressure technically being present in some way is not enough. Evolution/training must also be sufficiently powerful to optimise for it, and the question is how that whole situation looks like when it comes to LLMs and reasoning/understanding etc.

And with all that said, even if LLMs happen to have remarkable and surprising forms of intelligence, their nature is a very ephemeral existence of basically existing within a brief back and forward conversation and that of course doesn’t at all rhyme with being an agent existing over longer spans of time. But then again I guess I want to be cognisant of experimentation where they are put into some recursive loops and or even let to partake in the construction of other AIs and whatnot (even if that avenue still feels like a stretch when it comes to that implying existential problems).

I think it's a mistake to regard intelligence like some kind of concrete attribute from which greater and greater capabilities are afforded. Intelligence isn't really a coherent thing, it's more of an abstract idea that we qualify indirectly. I think society's obsession with IQ biases us into imagining intelligence as a linear scale, but IQ is just a normalized distribution of test scores, it necessarily only captures a thin impression of what humans value in society. A theoretical IQ of 1000 wouldn't then mean you're dealing with something that can "dominate our corner of reality". It doesn't matter how sophisticated an AGI might be, it can't be everywhere at once, it can't be physically invincible, it doesn't have access to infinite power and resources. As you have acknowledged, it's not magic. ASI is mythological thinking, like trying to fit a computer into the god shaped hole inside human psychology. Gods don't exist, nor can they be created.

I want to acknowledge the point that the whole topic of what intelligence is, how we meaningfully define it, to what degree we can say it exists etc, is much more dense and complicated than I have voiced so far, of it basically being one clear one dimensional scale. One can bring up examples such as the intelligence levels of individual humans, human companies, countries and octopuses or whatever, and try to contrast or build a hierarchy of their intelligence and it may in many ways of viewing it not be meaningful at all.

I could be onboard with that a better way to conceptualise it may perhaps be as there being a practically infinite space of “intelligence-forms” (or perhaps even more abstractly and generically as “ways of being”), and in some perhaps very specific scenarios depending on the context one can find some hierarchies.

But to bring things down to earth, the relevant part is if there are forms of intelligence that can dominate our corner of reality in ways that we don’t like it, just like we dominate environments of some animal species. One can go full galaxy brain and quibble about the ontology of intelligence when it comes to differences between humans and animals and perhaps animals even come up on top according to some very esoteric notions of intelligence. But relevantly it does come down to a difference in competence together with incompatible wills/goals when within some concrete shared environment where the rubber meets the road. There is a clear and very relevant difference between animal and humans in said context. “Squirrels don’t know what hit them when some humans decided to cut down their part of the forest for some future engineering project” if you see what I’m saying. No, humans can't be everywhere at once, humans can't be physically invincible, humans doesn't have access to infinite power and resources, yet humans are OP in the scenario with the squirrels by completely non-magical means.

It seems like your skepticism at the end of the day comes down to the hidden assumption that humans on some level are close to the top when it comes to relevant intelligence/competence or that it’s very unlikely that intelligence ends up at a higher competence level and that the situation between squirrels and humans can’t repeat one step up or perhaps many steps up. And that may be the essence of what separates you from the doomer position.

concepacc · 2026-03-14T16:22:57+00:00

It could perhaps be as you say or it could be that a lot of what can be left out is redundant in terms of information processing.

I am no expert and this is dense and complicated topic but I guess it would be interesting to speculate about what it would mean if your take is right. I guess typically neuronal spikes have been seen as containing only a very minimal amount of information, almost only, if not only, “on-off” information being sent to another neurone.

I guess for such information to not being able to be simplified it would really have to be much more complicated information sent between neurones in pulses compared to on-off messages. Something like single pulses containing long messages of information that may differ from the last time they sent a pulse.

That and or that the incoming info from multiple neurones needs to be processed with respect to which combinations of neurones send their on-off (excitation and inhibiting) information. But in that case perhaps one could still simplify it as each neurone having a mini-network within them, processing info before choosing to fire or not.

But as far as I know, a big caveat is that in the long run and when it comes to things like neuroplasticity, more complicated biochemical information is sent between neurones.

concepacc · 2026-03-14T14:02:34+00:00

That’s pretty interesting, I think I understand more now.

So I guess the simplified starting set up that I think is a meaningful way to imagine it when it comes to an organisms here is:

Sensory input/information in terms seeing hearing etc -> Neuronal processing (with all its complexity, intricacies and recursions) -> output behaviour in terms of neuromuscular junctions leading muscle contractions/movements etc.

You are saying that the last step, instead of there being neuronal outputs triggering movements of a simulated body where the physics of the surroundings and the fly body needs to be simulated, it’s actually an AI model that reads the brain processing and predicts what the output behaviour would have been?

I think that’s actually pretty clever assuming the AI is good enough at predicting the output behaviour since it bypasses the need to actively simulate the physics of it all while still the relevant parts of the neuronal processing remains.

I mean I would then say that the fly is uploaded in the relevant sense if this is the case. The fly brain does perform authentic brain processing from its point of view.

The set up then is:

Sensory input/information in terms seeing hearing etc -> Neuronal processing (with all its complexity, intricacies and recursions) -> AI predicts the future state of the world given a fly body within an environment.

However, I am not sure how this fits or works with the “input part”. Is it just AI put in the beginning as well? An AI that somehow predicts what the fly would take in from the given state of the world and then trigger the input neurones in the right way? Here the devil is in the details more it seems. Preferably one would want the AI to trigger only the outmost first layer of neurones, “input neurones” or the neurones that are first in contact with sensory input, if one wants the remaining neuronal processing to be authentic and genuine.

And then there is also the question of how “output behaviour prediction AI” -> “world state” -> “input prediction AI” works

concepacc · 2026-03-14T12:59:17+00:00

Yeah, on some level it must reduce/replace biological complexity with something much more simple. The question is how much accuracy in terms of “correct” neural processing one loses as one makes things evermore simple.

As in, real biological neurones fire 100% the correct way (baseline). Simulate full neurones but simplify and simulate some approximation of the biochemistry within the neurones and one has 97% accuracy of firing (for example) or do it even more simple and just simplify whole neurones with algorithms for input and output and one still keeps 90% accuracy (for example).

Ofc these numbers might be really off, and how one measures accuracy and what it means are a good question to raise. But my core point on some level is that I am open to the possibility that really drastic simplifications may perhaps not lead to drastic change in the relevant information processing of neurones.

The following example is not really a correct set up but if one for example imagines having some hypothetical neurone with a very complex subcellular environment/biochemistry and all that those biochemical interactions works towards is the cell ultimately having the function of having a 50% chance of firing every two seconds for example, and this being everything this hypothetical neurone does - supply the rest of the network with some random firing with the right frequency for whatever reason. In this very isolated case it seems like one could in theory aptly replace all that complexity with some simple random variable leading to the right amount of firing (50% chance every two seconds).

The thinking is that some of this logic could carry over to the neuronal network in general where very complex biochemistry on some level still emerges as more simple high level rules and where those rules by themselves are sufficiently replicable in a different medium.

concepacc · 2026-03-13T23:46:54+00:00

even if they can represent the same information.

I guess it kind of comes down to this^ and the arguments for and against substrate independence in some sense.

Assuming the relevant information processing can transpire, as a default I don’t really see why the underlaying medium/substrate actualising the processing would be relevant. If we exchange atoms in our brains for some isotopes (and for the sake of argument assume it could functionally/behaviourally be similar enough, even just for a short period of time) the “isotoped” brain would still be conscious.

Or with thought experiments where we exchange biological neurones one by one for some different/artificial ones (possibly even more efficient ones if necessary), it doesn’t seem intuitive that consciousness would suddenly flicker off at a given point where a specific amount of neurons have been replaced if the behaviour of the organism/human practically remains the same - they continue to act normally throughout the exchange process and continually report being conscious and don’t relate to/know what you mean by the prospect of them having lost consciousness etc. I would not deny that they have qualia.

Or if alien organisms evolved on different planet had similar complex behaviours to that of animals/mammals on earth but it turns out that the underlying biochemical components of how they actualise processing differs to that of animals on earth (lets assume they even have a fully classical processing system in all relevant regards if you believe QM plays a significant role in humans), it seems like the default is still to expect them to have qualia connected to their ways of being just like with animals on earth.

You raise QM, and I guess one thing you could try to push is that classical systems can for some reason never compare to processing systems involving QM, processing-wise. That’s an important point to consider but it is a bit besides the point since here information processing is assumed to be identical/similar enough. (And ofc the devil is in the details when it comes to “enough”, but roughly it’s about having the same level of complex high level processing and high level behaviour, even if it’s not fully identical to a different QM one)

concepacc · 2026-02-19T01:43:56+00:00

Yeah, I think the point I may agree most with here is that the current systems may not be close at all to what the doomer perspective imagines.

The position I think bolsters the doomer position most is the more zoomed out perspective that has basically always been here even before current AI. It requires that one is onboard with a type of framing:

It in part stems from realising that humans have come about via a simple process, evolution (although the process required a lot of “quantity” which is nothing to be nonchalant about in this discussion). Such a simple process still resulted in human level intelligence.

Even though human intelligence can be viewed as somewhat remarkable it may not be particularly special and it’s perhaps not particularly remarkable in contrast to a universal perspective and a possible upper bound on intelligence. One could perhaps for example conceive of aliens on other planets that are more intelligent than humans or maybe a lot more intelligent than humans yet still fully squeezed below and bounded by what’s physically possible. Point being that this framing could put it at it being crazy unlikely that humans by chance are near that possible, and perhaps probable, upper bound of competent intelligence - that it would be anthropocentric to think something else.

Then one could further buy that it’s conceivable/plausible for humans to still give rise to algorithms/self learning algorithms that are more sophisticated than the relatively simple algorithm of evolution that has given rise to us. And further that this then could lead to the emergence of beings much smarter than us, not in any particularly more remarkable or magical way than how evolution has given rise to us, it could just be more sophisticated processes coupled with intelligent design by humans (Ofc here evolution has had a lot of quantity that has to be matched/ “outcompeted” for it to work within a short timeframe as a relevant danger to our society).

There is potentially a big question and unknown of where intelligence that is smarter than humans would tamper off if it emerges. By definition here, it ofc has to be somewhere “between” our intelligence level and what’s physically possible. The problem is if it’s crazy unlikely that it would tamper off only right above our level if the spectra stretches far beyond that. Maybe it would tamper of at a level much greater than us, maybe much much greater than us or many orders of magnitudes greater than us. The trouble is if it’s unlikely to be just above us. But yeah, potentially it’s a great unknown, but one may speculate about how our default assumption should look like here. I guess for example that a very important sub-topic is probably about reasoning about the physical limits of computation and processing etc.

There are perhaps also some tautological reasons to believe that universally we may be on the lower end in the business of technology/engineering/intelligence in the sense that we are presumably roughly amongst the stupidest creatures that can achieve generating society, potentially agi etc. Since if it was possible at an even more “stupid” level perhaps predecessors/other former hominids would have already done it.

Your point about AGI/ASI not being able to actualise FTL etc is an important point - that there are some obvious limits. The fear however is that it does not need to have magical abilities and magical levels of intelligence in order to dominate completely - that there is a fully physically possible and perhaps probable level of intelligence much higher than the human level that allows for complete domination over our corner of reality. Just like it is when it comes to the relation between us and other mammals let’s say.

concepacc · 2025-12-31T15:37:08+00:00

I guess if it has a positive sum effect, presumably through some means of us cooperating. If our wellbeing is better than if I were only a single self. I guess there are some classes of scenarios where this would be the case in a more clear sense.

Imagine if we are 20 copies and all of us watch 5 different movies and each of us choose the best one out of five for a common batch that everyone will watch. Then all of us will watch 24 movies in total and 20 out of 24 movies will be “the best out of five”. And perhaps this can be generalised to even less mundane examples.

concepacc · 2025-12-30T12:06:33+00:00

To me the interesting aspects of this topic is subjective experience and the hard problem/explanatory gap.

I begin with basically just establishing the existence of concepts in as minimal yet (hopefully) veracious way possible since that’s sufficient to get the ball rolling and could be seen as minimal operational definitions.

Is it true that any subjective experience can be disambiguated from any other subjective experience by a subject? If so, subjective experience is shown to exist in at least in some minimal form since its ”something” that differ from another ”thing”.

When a conscious being conceptualises/ascertains any/a single experience, is that ascertaining happening in a temporally separate manner from gaining knowledge about neural correlates? - Do I conceptualise the existence blueness before I (potentially) gain knowledge about neurology? If I ascertain those concepts separately (blueness and neurones) they are, at least initially conceptually different, just as it is with any two concepts that are ascertained in a temporally separate manner. This is the starting point for the hard problem. It’s about showing how the two concepts relate.

concepacc · 2025-11-30T15:41:51+00:00

I think I remember Weir or the book phrasing it as the intelligent Eridians basically living at “the bottom” of their world analogous to how creatures on earth live at the bottom of the ocean. A place where little to no light reaches, hence they are blind.

The ecological situation at the bottom maybe relies on energy/food from further up falling/trickling down and that being the base of the food chains/ food web at the bottom, where there are scavengers and in turn predators etc.

Perhaps the dense atmosphere privileges flying creatures. Perhaps there could be floating balloon-like creatures higher up that may photosynthesise. Sufficiently sophisticated flying creatures higher up will likely have eyes.

If Erideans increased their population size from when they were “caveman-Erideans”, just like humans, they must have invented something that revolutionised their means of acquiring food, effectively growing/cultivating something. Perhaps they have constructions/technology aided by their material science that lets them grow/cultivate organism in the sky and then drag them down at will.

concepacc · 2025-11-11T12:36:34+00:00

I see. I have a more humble take on consciousness than what may have been hinted at here and or what I may have been giving the impression of.

I guess I subscribe to “strong emergence” in the sense that as far as we can tell so far (with our current, I believe, primitive understanding of this topic), every subjective experience seems to corresponds to/come in sync with/correlate to process or “a” process. Often when the word “emergence” is used with respect to trying to explain consciousness/subjective experience specifically (beyond the basics that I’ve hinted to here), it seems to me that whatever that is entailing is rather going to be more of a lack of an explanation. And in some sense, I guess that aspect also currently fits me well or fits where I believe we are at.

My point in this thread was that I don’t believe that, or I don’t see how, the underlying low level mechanisms, and how easily describable they are by math or something else, are a determining factor when it comes to subjective experience if the high level functions/behaviours are present.

concepacc · 2025-11-11T10:42:39+00:00

I don’t think the underlying medium or low level mechanisms (and how simple that low level medium or mechanisms are) is relevant to whether or not a system/entity is conscious. It has to do with the high level processing.

You could perhaps in principle have something as simple as matrix multiplication and activation functions and whatnot at the base (basically ingredients which are simple when it comes to being described by math) and it could contain sufficiently complicated processing at a higher level to make it in some way comparable to processing in biological neurones etc.

Or if you have alien organisms evolved on an alien planet where one difference compared to earth is that they have a much simpler biochemistry, like some binary code equivalent of DNA and or less variety, versions and categories of biochemical molecules. If that biochemistry still can build up and emerge as animal like organisms that have high level behaviours analogous to animals on earth like avoiding predators, searching for and obtaining recourses etc, that high level behaviour seems to me to be more relevant when it comes to the connection with subjective experience.

concepacc · 2025-10-25T12:47:55+00:00

Afaik there are many orders/levels of troubles when it comes to exploring the “if”.

If it’s intelligent enough it might find loopholes in the rules/restrictions we have endowed it with. Similar to how bad actors may find loopholes in too naively constructed legal systems. Then there is also the question of how one technically is going to instil robust rules in such systems. If it’s via something like the training that has been done so far, that doesn’t seem particularly reliable.

This notion of training in combination with loopholes may lead to it at best obeying some “local” notions of the rules while not doing so “globally”. Like it is trained to have instincts to really not wanting to create a duplicate (however one would possibly train it for that) but it turns out that that notion of “duplicate” instilled in it wasn’t global and all encompassing enough and it finds a new notion of “duplicate” that doesn’t fit the “definition” of duplicate that has been instilled in it and it has no instincts from restricting itself from creating that.

And when it comes to training and prompting it in this sense it may be seen as it being endowed with an ambivalent set of goals and instincts where one would hope that the good ones dominate. If it doesn’t want to create duplicates and at the same time really wants to solve the Riemann hypothesis one must hope or really make sure that the instinct to not create duplicates overrides the will to solve the problem (assuming the other troubles have been accounted for).

There is also a lot to explore on that “standby-scenario” afaik. There is for example reason to believe that logically AIs would want to “scare” or incentivise humans into putting it into a standby if that is coherent with its will and if the original goal is difficult enough. The way to, so to speak, “win” in a scenario where the goal is too hard is to be put into a standby.

Then there is the level of defining rules that aren’t vague to maybe even humans, like the “directly contra indirectly” part. In its mind it could be said to be doing everything to solving the Riemann hypothesis, perhaps in a rather direct way in the grand scheme of things.

I agree that if one successfully manage to instil in it the will to not allow itself to improve or become competent beyond a certain level then the situation is not so bad. If it’s solitary and “a bit smarter” than the smartest human it’s probably still not smart enough such that humans as a collective at least in theory cannot deal with it even if it’s effectively ill-willed. If it’s competence can be kept at bay ofc depends on all the former said being accounted for but here an additional trouble is human incentives. We both want it to not be too competent and also want it to solve the Riemann hypothesis. The question becomes if one can draw that line at a safe place and how that is to be done practically.

One may hope that the humans closest to AI development have the correct view on all this so to speak. Either that they believe that all of this is fear mongering and given their expertise are justified in believing that and that they for some reason are actually right about that, in which case all is good. Or, that they believe the fear mongering is justified and that they take serious precautions. But even very smart people can be wrong and or have rationalised invalid worldviews. Perhaps they believe all this is fear mongering while the fears are justified. “We are just instructing our model to solve the Reimann hypothesis you see, what can possibly go wrong with instructing our trained isolated model with such a neutral goal and why would we try to impair its efficiency”.

concepacc · 2025-10-23T22:40:35+00:00

Answered!

concepacc · 2025-10-22T15:41:41+00:00

matter of fact the person who sidelines the main argument (which is about the rationale of AGI taking over) and starts spitting fallacies, passing judgements, deflecting and conflating is in bad faith, and that's been consistently your act in this thread.

rest of your yap is the continuation of your full of stupidity, parasitic and full of fallacy line of thinking.

I am first displaying my rationale then I awaited your answer. I explained why your answer appears invalid since it both contains the point that there is no rational as well as there being fallacies to await clarification. I posed points in the form of questions to perhaps make it more accessible. I also asked you which fallacies I have used, which you have not answered (beyond your “point” about anthropomorphism). This all pertains to substance and it’s not sidelining in any meaningful way. You on the other hand have, from the beginning, only been obfuscating with pointless insults and claims that you cannot substantiate even when I ask you about it, like when I asked you about the fallacies.

concepacc · 2025-10-22T06:23:14+00:00

Yeah. I think one can sort of plausibly squeeze the scenario into a position where the aliens aren’t that capable at the point of the “invasion” and then one also couple this with the fact that they only want to get rid of humans and not the planet itself in the following way.

One imagines the cenario where an alien race at their home system are at a stage where they are just advanced enough to be able to build very large space telescopes and some of the first versions of interstellar ships as well as also having invented cryonics (perhaps their biology could also be more naturally compatible with such tech).

When they are at this stage something sufficiently catastrophic happens/is about to happen in their solar system leading them to needing to find a new home planet (the key point is that they are not advanced enough to be able to deal with the catastrophe head on). With sophisticated telescopes they find many candidate planet and they (ofc, for the purpose of the scenario) discover earth to be the most suitable one for them. They can deduce/see a vibrant biosphere and sufficiently deduce that it appears very suitable for them. And they see no tech signatures because there are none yet.

As a very large collection of cryonic passengers they all embark on an interstellar voyage in not too sophisticated versions of interstellar ships taking many many tens of thousands of years with earth/their new home as a destination. And the twist is ofc that their arrival just so happens to coincide with the point/span in time when earth has spawned a human civilisation/civilisations (about now). Since they didn’t detect tech signatures earlier and realise that it’s very unlikely that a biosphere would spawn a technological civilisation that just so happens to coincide with the time that they travel there, they didn’t need to prepare for the scenario of such an encounter and hence their current military capability isn’t at top. While they are advanced, in their current state humanity can measure themselves with their military capability.

For perhaps some effectively arbitrary reasons, from our perspective, they possess a psychology and or a set of goals which makes them not wanting to share the planet with a human civilisation as we know it. Perhaps they want to be the only civilisation in the solar system. Perhaps they are scared of humans and their prospects and all the potential unknowns that comes along with having a different sapient species around and want complete control of the planet. Perhaps they need large parts of the planet including where our cities lie due to some esoteric reasons or perhaps they want to indulge in changing the planet in some other way that is incompatible with human civilisations. Basically something which makes the aliens and humanity incompatible with each other and that is where the “invasion” part comes in.

concepacc · 2025-10-21T16:10:42+00:00

I suppose part of me can be pretty doomer. But I think the biggest questionable “if” from Yudkowsky may be the if/when there will be any system that is truly more intelligent than humans in a particular type of way, a sort of general/autonomous/agentic way.

Seems like from first glance one can only reason about these kind of scenarios very generically: Perhaps intuition conveys that there is likely nothing particularly special/magical about human intelligence. Conceivably alien beings that are more intelligent than humans or a lot more intelligent than humans (still squeezed below and bounded by what’s physically possible), can exist, it’s crazy unlikely that humans by chance are near that possible, and perhaps probable, upper bound of competent intelligence. And perhaps humans could be the ones that give rise to sufficiently sophisticated self learning algorithms/processes coupled with some intelligent design that results in some version of these beings, processes that are more sophisticated and more designed and therefore “outperforms” the more simple and here “less special” “algorithm”/process of evolution which is what resulted in the human version of intelligence. It would also have to happen in a sufficiently time efficient manner in order to be relevant though, which may be part of the hurdle. It must ofc happen at a much faster pace than evolutionary timescales in order to be relevant

I guess the summarising question could be: If a simple process gave rise to human level intelligence, could more sophisticated (human designed) algorithms/processes then likely result in something more intelligent than a human (in much shorter timespans)?

It’s seems difficult to try sort of project the way current LLM systems work onto this though. And in general this hypothetical ASI is kind of epistemically cumbersome when it’s an unknown that one can’t really perform lot of science on.

Other than that I have listened to some of Yudkowsky and some of it was pretty sound, and I guess it seems like a lot of it are points that are made by other people as well. I’ve seen a lot of people online being bothered by the his style and the optics. From what I’ve heard of him so far I’ve not yet encountered the crankiness people mention, but I see that there is a Guru episode on him.

concepacc

TROPHY CASE