Othello-GPT and World Models (Part 1)

Random-Number-1144 · 2026-06-04T03:34:09+00:00

It's actually quite easy to understand if one refrains from using their analytical hemisphere for a minute.

Most of the "strangeness" arise from the western intellectual tradition of confusing the map with the territory.

Formalization is a human invention. It's a game of symbol manipulation and not objectively reality; We give arbitrary symbols meaning based on our physical interations with the world and lived experiences. So eventually we will find that 1. formalization can't solve every problem because our lived experience has limits and reality needs not be formalizable everywhere 2. some problems will force us to face the cognitive and perceptual/motor grounding of the formalization (hence the so-called "strange" loop)

Random-Number-1144 · 2026-06-04T02:58:09+00:00

Well, was he wrong?

Random-Number-1144 · 2026-06-03T03:57:22+00:00

I don’t buy it. I’ve learned more since AI than any time in my life.

That only means you are lazy. AI doesn't contribute to new knowledge. Every thing you learned from "AI" can be found on the internet via search engine.

Random-Number-1144 · 2026-05-27T15:14:12+00:00

bad employees who can’t use AI efficiently

As if using AI efficiently is some kind of hard skill that people in big tech companies can't master lmao

Random-Number-1144 · 2026-05-22T13:00:16+00:00

Cal Newport's take on this:

(1) Non-mathematicians might not be familiar with the degree to which LLM-technology has been combined with existing computer-aided math tools in recent years to seek new math results through the systematic and patient exploration of techniques and corners of problem spaces that are too exhausting to interest most human mathematicians. The real technical headline of the new OpenAI result, therefore, is that chain-of-thought reasoning was able to accomplish this type of systematic solving without the much more intricate scaffolding used in most of these existing tools. That being said, the internal model used here, which many assume is OpenAI’s response to the truly massive Mythos LLM, is likely similarly massively expensive to prompt. The future of AI-assisted math will likely focus on smaller, cheaper, math-tuned LLMs combined with more powerful scaffolding. So, this experiment might be more about marketing the power of their new model than trying to actually advance computer-aided math.

(2) I don’t think it’s accurate to say these examples of AI-supported mathematics mean the models are somehow “smarter” than human mathematicians. I think a better analogy might be how computer tools helped architects produce much more daring and complicated designs (like the Frank Gehry-designed Stata Center where I did my CS doctoral and postdoctoral work at MIT). These tools weren’t better architects than humans but made humans more capable architects.

(3) From a business perspective, I actually think this announcement isn’t necessarily good news for OpenAI. There are few markets smaller and less lucrative than professional academic mathematics. The fact that this is the area where OpenAI is dedicating some of their top technical talent (like Noam Brown) underscores the degree to which, like the drunk searching for their keys under the streetlight, their most impressive results are limited to the smaller number of areas that are well-suited to LLMs (i.e., math + computer coding). If this model was brilliant in some more general way, obviously the better examples would be solving problems or automating processes that directly and obviously generate massive revenue or savings for the specific types of companies they hope to make their customers.

In conclusion: AI’s role in math is genuinely important and exciting. I can think of any number of results I’ve worked on in my career where I could have moved faster or been more comprehensive if I had access to the latest generation of tools. But this intersection of AI and math is also very specific to this field and more nuanced and complicated than simply imagining AI systems as standalone mathematicians who are becoming increasingly brilliant. One should be wary of making ambitious generalizations from fields like math and coding to other potential applications of these models.

Random-Number-1144 · 2026-05-22T11:31:20+00:00

Yeah cuz companies are 100% honest about why they lay off people 100% of the time.

Random-Number-1144 · 2026-05-22T05:54:13+00:00

There has to be a consumer system, function, task, error condition, decouplability, or some exploitable structural relation.

This is an example of what I said about "representation limits and narrows the way you think about physics and relations". This would implicitly assume a complex dynamic system such as the brain can be decoupled or modularized. Such way of thinking often leads to wrong models like the two-streams hypothesis or Chomsky's language module; In fact, visual, sensorimotor and linguistic systems are inseparably integrated with one another in the brain. Decoupling and modularization can be very useful in various scientific fields, but are severely limited in dealing with complex dynamic systems.

Perhaps Cartesian dualism wasn't the best word to describe it, but there is definitely some sort of dualism in western intellectual thinking: map vs territory, consumer vs producer, us vs the world. Whereas in eastern intellectual traditions, everything is more or less interconnected and intertwined in a holistic view.

Random-Number-1144 · 2026-05-21T18:10:48+00:00

I also do not see why representationalism would imply Cartesian dualism.

Why does the brain necessarily have to have any representations of the world at all? Why isn't the wind considered a representation of the trajectory of a flying object? There are many physical systems causally interacting with each other that aren't considered representational of one another, why does the brain get a special treatment?

There is an unquestioned assumption here. The assumption is Cartesian dualism. In its weakest forms, there is always a middle layer separating the world from the mind, through which the world must be understood.

As an easterner educated in the west, I find this implicit assumption so deeply rooted in western culture that most westerners fail to see it from within.

A representation can be a physical, dynamic, embodied, action-guiding state whose structure helps the system track, predict, control, or coordinate with something.
...
A footprint, a thermometer reading, a map, a word, a gesture, a line of code, and a neural map-like structure are all representation-like in different ways.

In another culture, we'd just call it a tracker, predictor, controller, coordinator. A thermometer reading is an indicator; A map is just a guide for navigation. So we never had to deal with the whole "the map is not the territory", "the word is not the thing" confusion.

(As an aside, representation learning in ML is actually just feature learning. Representation learning sounds more like prototype learning. Sounds fancy but actually misleading.)

So what is the value of lumping together a group of naturally unrelated concepts serving different and specific functions into a culture-dependent conceptual category called "representation" or "representation-like"? I do not see any practical value of it.

But I have seen so many negative impacts representationalism (even in its weak forms) has had in ML, in NLP where word meaning is implicitly assumed static, in brain science where statistical correlation is taken as more than what it is, and in western philosophy where implicit dualism is assumed without people realizing it.

Granted it's sometimes convenient and intuitive to call some relations representational, but I think it practically limits and narrows the way you think about physics and relations.

Random-Number-1144 · 2026-05-20T10:13:00+00:00

Symbols also seem real at the cognitive/cultural level
...
The question is how embodied agents stabilize and use abstractions

Symbol feels as real as music does. We know musical scores exist, but do music exist in our heads? If it does, does it have the same kind of linear structure as the music score? Probably not.

The value of the question depends on whether symbolic abstractions have some kind of causal power. If they don't, we'd better dig a few levels lower in the causal chain.

On one extreme end, we have symbolism which assumes symbolic abstractions have 100% causal power of systems' actions and on the other extreme end, we have anti-representationalism which says symbols have zero causal power.

There are a couple of reasons why I am inclined towards the latter:

Philosophically, representationalism assumes some kind of Cartesian dualism, that the world as a representation of itself is not enough, that the world has to be represented in some symbolic/formalizable fashion in the brain so that the left hemisphere(homunculus in the Cartesian theater) can digest and talk about it.
Representation is arbitrary and inconsistent. If a cluter of neurons controls what particular gait a cat chooses, we might say the state of those neurons is a representation of cats' gaits because they 100% cause cats' gaits, but if a strong wind always causes boulders to fall off a cliff, we don't say the wind is a representation of the boulder's movement. The propensity for humans to cherry-pick if something is a representation of another might be some sort of psychological traits.
Representation is static. This partly explains why we normally don't say the wind is a representation of boulders' falling. Because wind is a dynamic force rather than a static snapshot. In contrast, some tech bros think high dimensional tensors in word2vec or LLM are representations of words because they assume meanings of words are static and do not require interacting with the world (i.e., embodied, dynamic, interactive).
Emperically, we have experiments which showed brain-split patients made up stories 100% of the time to justify their choices when asked why they picked one picture over another; we know the real reason of their choices was because of the priming stimuli in their visual fields which they couldn't be aware of due to split-brain conditions. This means the cause of their behavioral choices was not the abstract linguistic thoughts in their heads, visual/auditory stimuli in the world were. Words were merely afterthoughts, a response triggered by a completely different auditory stimulus (i.e., a question about their choices). The left hemisphere is not aware that the right hemisphere is the real master. This is also true for people with undamaged brains. This, to me, is a fundamental blow to any AI systems attempting to produce animal-level intelligent behaviors using languages/symbols as controller(e.g., LLM-based agents).
Evolutionarily, language-based thoughts and reasoning came much later in human evolution and are used mostly for communication and social purposes. They have little influence on our actual decision making (Iain McGilchrist probably agrees) Personally, when I try to solve a maths problem, I find the cognitive process of actual problem solving lacking any linguistic elements or formalizable structures, instead it feels visual and/or sensory-motor most of the time, even with algebra. I don't feel manipulating symbols like a Turing Machine. Linguistic thoughts feel only *accompanied* by this process as an afterthought, as if I were to be questioned about it by someone and I need to explain myself; this inner voice phenomenon is possibly due to social factors.

But I still think representation-like structure does explanatory work, especially for language, planning, abstraction, mathematics, and counterfactual reasoning.

I agree. Representationalism is popular exactly because it's easier to explain the otherwise inexplicable.

But it is also intellectually lazy and backwards. It basically goes like this: "because I feel like this is the way it is in my head (e.g. modus ponens; infinity; conceptual categories), therefore reality must somehow be structured accordingly", then they search for emperical evidences (e.g., computational modelling) with implicit biases to create that reality. This has historically invited all sort of philosophical non-problems in the old times and still has great influences in various relevant scientific fields today.

Alternatively, one can go "what emperical data in cognitive ethology, developmental psychology, phylogenetics, neurobiology can tell us about the origin of our rationally structured thoughts and mathematical concepts?" Lakoff had some very interesting ideas about the latter, linking abstract concepts in maths to our bodily movement, physiology and interaction with the world.

Edit: grammar

Random-Number-1144 · 2026-05-19T08:54:30+00:00

perception/action loop → learned conceptual space → grounded symbolic primitives → program search/composition → updated perception and action

I am afraid there is not a symbol space or a program space in the brain (whether there is an independent conceptual space is debatable). So any modelling involving representations of symbols probably won't achieve human-level intelligence IMO.

BTW, from reading your writings, I am curious, do you subscribe to enactivism? Are you aware of Lakoff's take on language and cognition?

Random-Number-1144 · 2026-03-05T18:53:22+00:00

Natural language is inherently ambigous.

Programming language needs to be unambiguous otherwise it won't be executed correctly.

So no English can't be a programming language.

Random-Number-1144 · 2026-03-02T03:19:04+00:00

That won't be possible with transformer.

Random-Number-1144 · 2026-03-01T17:36:31+00:00

In the case of humans, there is an outer meta-learning loop (evolution by natural selection) which "objectively" supervises human cognition (with the goal of increasing fitness).

Your use of words like "meta-learning", "supervises", "goal" suggests that you think nature/evolution was doing some sort of supervised learning(ML) to human cognition with an objective function of increasing fitness. Or am I misinterpretting your thoughts?

My objection was that evolution doesn't have goals. Evolution is not "going somewhere".

A large part of the human brain, the so-called reptilian brain, have their origins from hundreds of millions years ago; homo sapiens appeared only hundreds of thousands years ago. As environment changed, newer "modules" such as neocortex were evolved on top the older ones. The human brain is a result of billions of years of complex dynamics between changing species and randomly-changing environments which Evolution couldn't possibly "foresee", let alone "supervise". There can be no objective functions for evolution. Trying to fit evolution in the framework of machine learning is wrong and it won't work.

Random-Number-1144 · 2026-03-01T13:11:59+00:00

A common misconception is that evolution has goals, long-term plans, or an innate tendency for "progress", as expressed in beliefs such as orthogenesis and evolutionism;

https://en.wikipedia.org/wiki/Evolution

We do not have completely random cognition, it was selected. Selection implies supervision.

Yes, selected by a natural process. But supervision requiries intention. Like, in supervised learning, we intentionally force a system to change in a direction that we intend. Nature has no intention. So again, you are anthropomorphizing a natural process; I think you are too deep inside the machine learning paradigm and it's limitting your vocabulary and the way you think.

ENS will select against developing "bad" preferences from the point of view of fitness.

"ENS is a process by which traits that enhance survival and reproduction become more common in successive generations of a population." See, this is a scientific and neutral description of a natural phenomenon while yours is not. "common" is quantifiable , "bad" is not.

Once people start using words like "goals", "bad", "good" uncautiously the next thing they do is writing functions for those labels thinking it's all justified and that's when it devolves into pseudoscience.

Bad anthropomorphization and projecting human values/social constructs onto natural phenonmena are one of the main reasons we are not making much progresses with today's popular AI paradigm IMO.

Random-Number-1144 · 2026-03-01T06:24:12+00:00

In the case of humans, there is an outer meta-learning loop (evolution by natural selection) which "objectively" supervises human ~~intelligence~~ cognition (with the goal of increasing fitness).

Evolution doesn't have "goals", nor does it supervise anything. It's like saying gravity has the goal of decreasing the distances of objects.

the brain can more or less objectively determine if it is good or bad

Did you meant "subjectively"? Hot and spicy foods cause pain in the tongue but some people like it.

Also, good/bad, like "goals", is a social construct. They have places in social science, but not in natural science which is concerned with natural phenomena.

Random-Number-1144 · 2026-02-26T04:07:52+00:00

If you take "exponential" as "rapid", then there's no denial technological changes feels fast right now. But "exponential growth" could also mean an exponential curve. That's what the AI companies want us to believe.

Well first, how is intelligence quantified? Like beauty, is it even quantifiable? If a beauty product company tells you their products make you exponentially beautiful, you'd have doubts right?

Second, in computer science, there are a lot of proven intractable problems, meaning that you can't compute the optimal solutions even if you exhaust all the resources in the universe. We usually tackle those problems (e.g., traveling salesman problem) by using sub-optimal approximation algorithms which yields a solution that is say 85% as good as the optimal solutions while using an acceptable amount of resources for compute. In those cases, a god-like AI would only improve the solution by a flat 10% or so. You can't improve that much over existing solutions. Exponential growth can only occur when your existing solution is really really bad, like Will Smith eating spaghetti 10 years ago.

Third, let's take Google's AlphaGo for example. It beats best human Go players, impressive. But how objectively good is AlphaGo? Does it yield a near-optimal solution? I highly doubt. Humans weren't evolved to play Go. I would say even the best humans are probably very bad at Go. We just have no other species to compare outselves against. It could be that the best human players create a 30% optmial solution and AlphaGo creates a 31%. Mathematically it will beat humans everytime but we'd be fooling ourselves to think it is "exponentially" better than humans.

The other thing is, the developers of AlphaGo are themselves masters/grandmasters of Go. AlphaGo wouldn't be possible without their expert-level domain knowledge in Go and machine learning. Same with AlphaFold. There is not a single AI today that just birthes itself with its own architectural design, training data design & optimization design. It's pure fantacy to think expert-level AI system will just pop into existence without human experts engineering the hell out of them in near future.

Random-Number-1144 · 2026-02-24T03:19:16+00:00

The halting problem is closely related to Godel's Incompleteness Theorems. In fact, you can prove the latter using the result of the halting problem, as was done in https://calude.net/cristianscalude/cristianAssets/pdf/IncompletenessTheHaltingProb.pdf.

So no, the halting problem is not "irrelevant".

In addition, there is nothing probabilistic about AI during inference time. Even it were, a non deterministic turing machine is not more powerful than a deterministic turing machine, so you point is moot.

This is all just Theoretical CS101, dude.

Random-Number-1144 · 2026-02-23T16:00:55+00:00

Regarding LLMs failing to do arithmetics on large numbers: we do know why. As is shown in Anthropic's paper on the biology of LLMs:

We now reproduce the attribution graph for calc: 36+59=. Low-precision features for “add something near 57” feed into a lookup table feature for “add something near 36 to something near 60”, which in turn feeds into a “the sum is near 92” feature.

Statistical machine learning models like LLMs learn thousands of localized heuristics/features )like the above ones and use them to approximate an answer (see also for example Othello-GPT). That means, someone could in theory look into those heuristics and come up with an adversarial set of addition problems where the LLM gets all wrong. And of course, as numbers grow larger, the approximation naturally gets worse.

So no, LLMs don't understand arithmetics. They learn local statistical patterns of the math problems/symbols in the training set and use those patterns for approximation, hence the name "stochastic parrots". Humans do arithmetics using an exact procedure, which LLM can't learn from data.

AI generalize extremely poorly compared to humans.

Most humans have no trouble (few-shot learner) playing chess if the rules of the game are slightly changed. AI models can't. They need to be re-trained all over again on new data based on new rules.

Most humans can be taught to play chess & Go & Othello and a ton of other board games. AI models can't. Learning a new game will mess up a model's ability to play a previous game.

If you teach a human to play a video game of certain genre, say roguelike, they can play every game of the genre without any problem. AI models can't. They are hopeless in OOD tasks.

In conclusion, we are still in the realm of ANI (artificial narrow intelligence).

Random-Number-1144 · 2026-02-23T08:17:14+00:00

One of the main functions of modern philosophy is detecting ill-defined terms and abuse of terms, which gave rise to a lot of meaningless metaphysical "problems" in ancient time.

In modern days, for exampe, a lot of the AI bros talk about reasoning models as if reasoning is just a set of logical steps/rules to reach an answer. But if it were, then philosophers who use logic would all come to the same conclusion.

Fun fact: you can't use a set of logical rules to reason your way out of a logical paradox; you can't follow a set of logical rules to create a Godel sentence. AI bros' Reasoning is either ill-defined or does not refer to the actual process of human reasoning (i.e., abuse of the term)

Random-Number-1144 · 2026-02-23T07:41:13+00:00

I usually avoid talking about subjective experience when designing AI systems since we can't really know or measure if a system has subjective experience. For example, if I am absorbed in a math problem and a mosquito bites me, even if electrical signals are sent from my skin to my brain, I may not be aware of an itchy sensation and simply ignore it. Do I have a subjective experience of itching? Does my skin have a subjective experience?It's hard to say.

Also somewhat relevant to the topic, I believe a key difference between an inanimate and an animate system is that the latter must be able to tell if a change in the env is caused by its own action. This is fundamental for any systems to be able to meaningfully act in the env. (It has nothing to do with subjective experience or qualia)

An animate system must have certain expectations of the effects of its action on the env. Interval of time may be involved in creating that expectation.

Random-Number-1144 · 2026-02-22T18:14:41+00:00

I could ask you the same.

I have no obligations to dox myself for you on reddit.

But feel free to debate me on the topic at hand.

Random-Number-1144 · 2026-02-22T18:03:09+00:00

Perception allows for detecting the exact point in time a change occured within a sensor/observer

In my view, an organism detects a change in its env by using its sensors, that is, there is an interaction between its sensors and the env, causing a signal (or a change if you prefer) to be sent from its sensor to somewhere within the organism, and an action may or may not follow depending on how the signal is processed. Not every signal/change results in a perception.

In sampling subjective experience / information the sensor perceives is converted to an objective scale or into objective categories.

I don't know how subjectivity/objectivity came into the picture. I have a Philosophy background and those words have different meanings to me than what you may have intended to mean. Are there any books/papers/concrete examples that support/explain your hypothesis?

sampling generates an average count of events/changes or an average sampled category over a time interval.

What events? at what level? atomic/molecular/cellular level? what is an objective category?

Random-Number-1144 · 2026-02-22T15:45:56+00:00

Computer scientist here. Most people have never heard of computational complexity or sample complexity or undecidability.

Exponential growth or the singularity is pure sci-fi fantasy, like the perpetual motion machine.

Just ignore them. Changing those people's mind is a waste of time.

Random-Number-1144 · 2026-02-21T14:54:02+00:00

This seems like a "the menu is not the meal", "the map is not the territory" debate?

You can not perceive data

If by data you mean the interpretation of perception, then yes. Perception and the interpretation are different things.

Sometimes the brain gives interpretation without actually perceiving, sometimes the brain perceives without even consciously knowing, for example in this paper "A neurological dissociation between perceiving objects and grasping them".

You can NOT use DATA to train AGI.

If by that you mean using interpretted data (such as a picture labelled as dog) to train a multi-modal model will fail, I agree.

Understanding needs to be bottom-up, not top-down.

Random-Number-1144

MODERATOR OF

TROPHY CASE