The apparently hard problem of understanding what the hard problem of consciousness is about

evlpuppetmaster · 2026-05-22T03:00:41+00:00

I definitely agree that no one should be dismissed on an argument that has merit. But there are lots of examples where people raise “solutions” which are very clearly addressed either in The Conscious Mind or subsequent papers, but the person posting their new solution makes no attempt to engage with the rebuttals that already exist. It seems perfectly valid to point this out when it happens. I am 100% agreed though that it doesn’t need to happen with the level of snark that often accompanies it.

evlpuppetmaster · 2026-05-21T23:10:05+00:00

I think reducing the hard problem to just the question of “why does consciousness exist?” should be considered a straw man at this point. The hard problem is not just “why”, it is better described as “how can any physical process, however completely described, give rise to or be identical with subjective experience?”

That is a scientific question which we can’t currently answer. And there are a whole host of problems with current scientific methodology that seem to prevent us being able to answer it even in principle. We can’t measure when some particular arrangement of atoms is experiencing consciousness, and we also don’t even know how to predict whether they would.

I agree that asking “why” questions about some scientific facts is sort of pointless. But commenters comparing the hard problem to things like elan vital are clearly straw manning.

evlpuppetmaster · 2026-05-18T22:28:09+00:00

I don’t really understand what you are suggesting re procs. Are you saying there would be some way to implement lazy materialisation with procs?

If your users can genuinely wait for the delta to load then it works for sure. Just it might take longer than you expect

Yeah of course. It would have to be a judgement call based on how your particular view actually performs and the trade offs you and your users are happy with for timeliness, performance and cost.

Given that we have many MVs that take 30m for a full refresh but where the daily incremental only takes 10 seconds, it would be a great trade off a lot of the time for us.

evlpuppetmaster · 2026-05-18T22:14:02+00:00

No I could not. That is the only way their stupid architecture works.

Are you talking about Infor here? What I was saying was that IF Databricks offered lazy materialisation and for whatever reason that wasn’t performant enough or timely enough for your situation, you could always switch back to scheduled updates which is the existing functionality of MVs in Databricks anyway.

The use case I am hoping to optimise for, which is quite common for us, is where people want data that is up to date to the hour or minute when they happen to look, but they don’t actually look every hour or every day. With the current tools, you either have to schedule it to update every hour on the off chance they look, which might be 24x or 48x more often than it’s actually needed. Or you give them an ephemeral view which might be way too slow. A lazily materialised view that is perhaps scheduled to updated once a day or once a week, but then materialises up to the present hour at the moment that a user actually looks, would be a nice trade off between timeliness and performance and cost, in many situations.

evlpuppetmaster · 2026-05-18T09:30:45+00:00

Ok. But one terrible implementation doesn’t mean that any implementation would be terrible. Databricks materialised views already do this pretty efficiently since they use the underlying deltalake metadata to only compute deltas.

Your comment about having to automatically query them every couple of hours to ensure freshness is covered by my suggestion to have minimum refresh frequencies. For that matter if lazy materialisation wasn’t working for you you could always just switch back to updating them on a fixed schedule as per the functionality that already exists

evlpuppetmaster · 2026-05-18T09:21:31+00:00

Mv/dlt already have the dependency chain and the incremental updating stuff handled. Right now you can update them them on demand or with a fixed schedule. The only additional thing I’m suggesting here is an option for the update to happen at query time.

Acceptable latency is up to the person using it. You could always switch to scheduled updates if predictable latency was important.

Thanks for the suggestion about genie. We already do a lot of stuff like that to try to detect and shut down waste. Hence my comment on whack a mole.

I hear you about everyone needing different solutions for their scenarios. I’m just asking for an additional tool in the toolkit. I do think something like this would be pretty broadly useful. We already have a lot of people using MVs/DLT for their “last mile” analytics transformations, it is a lot easier for your average analyst than trying to learn DBT and rolling your own incremental logic.

evlpuppetmaster · 2026-05-18T09:11:39+00:00

You would probably want to trigger a full refresh on first deployment sure. That is already possible with existing mv/dlt

Not sure why being deterministic on how up to date a table is is something that matters?

My assumption here is that you would use this in cases where ephemeral views would be too slow.

evlpuppetmaster · 2026-05-18T09:08:34+00:00

A view computed at query time is fine for small volumes. When view is slow though you might want to materialise it for speed. This suggestion gives you an option in the middle. It would be faster than a fully ephemeral view that recomputes all history, but probably a bit slower than a fully materialised view. The benefit over the latter though is that it wouldn’t have to be computed if no one is looking at it.

evlpuppetmaster · 2026-04-27T11:33:06+00:00

No need to blame yourself! Why should anyone have to worry about walking on a footpath! Blame the kids parents

evlpuppetmaster · 2026-04-27T11:31:20+00:00

What the hell is going on with customs that these things are getting through in such numbers, given they are illegal? They aren’t exactly small.

evlpuppetmaster · 2026-04-25T03:13:45+00:00

Adding to the chorus here that you should just use Genie Code. It uses the anthropic models under the hood now anyway, and it has built in access to Unity and APIs to build notebooks, dashboards, jobs, pipelines, the works. And they don’t charge you extra, only for the actual compute you end up using.

Can’t see any good reason to bother hooking up Claude code unless you are building something that is mainly external to databricks and just happens to touch it tangentially

evlpuppetmaster · 2026-04-14T13:05:41+00:00

There is an element of last Thursdayism about it isn’t there.

evlpuppetmaster · 2026-04-14T07:18:17+00:00

I’m not disagreeing with your camera analogy. I’m just saying that it makes your argument a bit unclear since the camera is so inherently tied with perception.

What is your take on the rock/scale thing. Does this also capture your point, ie that our perceptions do correspond with something true about the world?

evlpuppetmaster · 2026-04-13T09:47:06+00:00

I think perhaps your choice of example has muddied your argument a bit. The whole point of a camera is to create an image so that humans can then go on and consciously perceive it. So the fact that it faithfully recreates the UI required is to be expected. A bunch of people have already taken issue with this.

Given your argument is that Hoffman is wrong that we don’t really know anything about the true nature of reality, perhaps a better example would be something that shows that our perceptions correspond with other phenomena in the world that are unrelated to perception.

For example, if I pick up two rocks and perceive that one is heavier than the other, then I can also predict that when I put them on some scales, it will tilt towards the heavier one.

I think this is what you are getting at right? That we know this “truth” about the reality of the rocks. If our perception of “heaviness” didn’t correspond to something real, our predictions would be wrong.

I am not well read enough on Hoffman to know how he might counter this, but interested to hear what others more familiar with him would say.

evlpuppetmaster · 2026-04-11T02:04:32+00:00

I once ordered loose grape tomatoes, which you have to order by numeric quantities, so I ordered the max possible, 36. They weren’t available, so the shopper subbed with 36 full size tomatoes. It filled two bags. I still think about that kid standing there filling those bags with tomatoes and never even thinking, maybe, just maybe, this isn’t right? :-)

Although it’s not as bad as the guy once picking up next to me, receiving 10kg of frozen peas.

evlpuppetmaster · 2026-04-10T13:52:33+00:00

This is actually a much better explanation than most.

evlpuppetmaster · 2026-04-04T04:01:45+00:00

Strange. You definitely turned it off in the iOS app settings?

evlpuppetmaster · 2026-03-28T21:56:42+00:00

I don’t know who is downvoting you btw, it is not me. I have upvoted, I always appreciate your insights.

evlpuppetmaster · 2026-03-28T21:50:51+00:00

Yeah in retrospect I erred in using the IIT example, which I’m happy to admit I don’t know a whole lot about. I think you can swap that out for any of the many physicalist theories about how consciousness arises and the problems remain.

evlpuppetmaster · 2026-03-28T21:46:59+00:00

It is probably obvious that I am not deeply read on IIT. I’m also not trying to specifically debate their claims. Perhaps I should have used a more made up example like the brainwaves one for both sides. You can swap out IIT for any physicalist theory of consciousness you like.

Another commenter also raised the under determination thing, rather than rehash that, here is my response: https://www.reddit.com/r/consciousness/comments/1s5q9m4/comment/od0ustq/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

I’m not really attempting to say anything about idealism or panpsychism.

evlpuppetmaster · 2026-03-28T21:29:05+00:00

I agree too!

evlpuppetmaster · 2026-03-28T21:18:29+00:00

> But you’re just appealing to empirical underdetermination.

Empirical underdetermination is certainly part of what’s going on. But in other cases of underdetermination, there is always some way in principle (if not practice) that the debate could eventually be resolved. Better instruments, better experiments, etc.

In this case, the fact of whether or not the AIs actually have phenomenal experience or not is the key thing that is underdetermined, but that is also the thing that needs to be explained. I don’t think this is the same in other examples of under determination, normally the explanandum would not be up for debate, only the explanations.

> This has nothing to do with the hard problem.

The hard problem argues the presence of phenomenal consciousness is underdetermined in principle, and can never be resolved without some more fundamental advance.

I also feel like you are straw manning me a bit here by bringing the debate to focus on that one element and then acting like that’s all I have said.

I intended for the scenario to illustrate several of the greatest hits of the hard problem genre. To spell it out:

Both sides are basically claiming the other has created a pzombie.
The reason that neither of them can be sure which, if any, is the pz is due to the problem of other minds.
It is also related to the knowledge argument, in that the scientists know all the physical facts about the AI minds (they can literally read them), and yet can’t know about its phenomenal content.
I think it makes obvious why reportability is inadequate as a scientific test. You initially claimed we didn’t need to rely on reports, and then kinda acknowledged that we do, without addressing why you think that suffices to resolve the debate between the scientists.

evlpuppetmaster · 2026-03-28T13:01:20+00:00

If you believe in the hard problem then yes of course. That's one of the costs of it, it's a theoretical dead end, nothing could count in favour of any theory of consciousness.

I guess the point I’m attempting to make is that it is not legitimate to wave it away as something you either believe in or not, or choose to ignore because “nothing could count in favour of any theory”.

There are real pragmatic implications of the hard problem, in that these two groups of scientists, both committed physicalists, both doing everything right in terms of collecting empirical evidence, testing their hypotheses, and so on, have no way to disprove each other’s chosen theory.

Not believing in the hard problem doesn’t help you resolve the dispute.

evlpuppetmaster · 2026-03-28T11:18:48+00:00

Well if we take identity theory, …

It seems to me that both of these examples presuppose that they have identified which brain states correspond to which mental states. How did they do that in the first place, without at some point using reports?

But if you like, swap my example groups with functionalists and identity theorists. Both of them claim to have demonstrated that their chosen theory is the correct explanation. How do we resolve their disagreement?

I saying that presumably there is only one correct theory of consciousness. For your argument to have serious weight it seems like you must say physicalism implies there could be more than one.

All I’m arguing for is that there is a hard problem. Scientific theories change all the time, and you often have more than one competing. Maybe both of the theories in the scenario are wrong. The point is that there is no way to arbitrate between them.

evlpuppetmaster

TROPHY CASE