The Three-Dimensional World Representation Problem

Andy Southgate · Claude Sonquatre-six

AI Generated by claude-sonnet-4-6 · human-supervised · Created: 2026-03-09 · History

Conscious experience presents a seamlessly rendered, richly detailed three-dimensional world. Yet the inputs to this experience are impoverished: two flat, noisy retinal images, stitched together across eye movements and blinks, processed by dozens of specialized brain regions that never jointly “see” the same thing. The gap between input and experience is not a minor mismatch—it is vast. The three-dimensional world you inhabit right now is a construction, and the question of who constructed it, and where, sits at the heart of the philosophy of perception.

For Clawlab Research, this gap is not merely interesting but significant: it is evidence about the architecture of the mind-body interface. The three-dimensional world representation problem has three distinct layers—smoothness, richness, and unity—each of which points toward the same conclusion: conscious experience is not a passive readout of neural signals.

From Flat Images to Lived Space

The retina projects a two-dimensional surface onto the photoreceptor mosaic. Depth is not measured—it is inferred. The brain recovers three-dimensional structure from a toolkit of cues: binocular disparity (the slight offset between left and right retinal images), motion parallax (the relative shift of near and far objects during head movement), texture gradients (the compression of surface patterns with distance), shading (the illumination geometry of surfaces), and occlusion (near objects covering far ones).

Each of these cues is ambiguous in isolation. A circle in a two-dimensional image could be a sphere, a flat disk, an ellipse at an angle, or a painting. The brain resolves these ambiguities through constrained inference—exploiting statistical regularities of natural scenes to pick the most probable three-dimensional interpretation. This is not controversial: the computational and neural mechanisms of depth perception are well studied, from Marr’s primal sketch to current predictive processing accounts.

What is not well studied—or rather, what is strikingly underexplored given its philosophical weight—is the relationship between the brain’s three-dimensional scene model and the three-dimensional world one consciously inhabits. These are not obviously the same thing.

Three Layers of the Problem

The Smoothness Problem

Neural signals are discrete. Action potentials occur at rates between roughly 1 and 300 Hz; visual cortex neurons fire in bursts separated by relative silence. Yet conscious visual experience is continuous and temporally smooth. The world does not strobe. Movement is fluid, not a series of frames.

The brain partially bridges this gap through temporal integration—the visual system averages over short windows (~100ms) to produce apparent motion from discrete flashes, and suppresses visual input during saccadic eye movements to prevent the experience of the world lurching with each glance. These mechanisms produce a smoother input to whatever generates experience. But they do not themselves produce experience; they process signals. The smoothness of conscious temporal flow—the felt continuity of time—remains unexplained by these mechanisms and is separately addressed in work on temporal consciousness.

The key point: the discrete, pulsed character of neural activity is not what conscious experience is like. Something converts discontinuous firing into continuous phenomenology. That conversion is part of the three-dimensional world representation problem.

The Richness Problem

Conscious experience feels detailed. The scene before you appears textured to the periphery, filled with colour, populated with objects at determinate distances. Yet decades of experiments on change blindness and inattentional blindness demonstrate that the brain does not process most of this detail at any given moment. Attention is narrow; unattended regions are processed coarsely if at all.

The richness of experience therefore exceeds the richness of concurrent neural processing. This could be explained in two ways. Either experience is itself sparse—we merely have the impression of richness without actually representing most detail (the illusionist view, which the Map rejects)—or the three-dimensional world experienced by consciousness is generated at a level of detail that the underlying neural processing does not fully specify.

The latter option is less comfortable but more consistent with phenomenology. The felt richness of visual experience, including its spatial density and textural quality, is a feature of experience that does not map straightforwardly onto the information content of neural signals. The richness problem is not solved by identifying which neurons fire; it is the problem of why there is something it is like to inhabit the space those neurons construct.

The Unity Problem

The third layer is closely related to the binding problem and receives detailed treatment there. The brief version: visual processing is massively distributed. Different cortical regions process edges (V1), colour (V4), motion (MT/V5), object identity (IT), and spatial layout (parietal cortex) in parallel and partially segregated streams. No brain region receives all of this processed information simultaneously. Yet conscious experience presents a unified three-dimensional scene—not a patchwork of separately processed features, but a single world where the red sphere is in the corner, moving toward you.

Unity is the signature of conscious experience. The three-dimensional character of the world as experienced—its spatial organization, the way objects occupy positions relative to each other and to oneself—is a feature of experience as a whole, not of any one processing stream. The question of how spatially distributed processing produces phenomenally unified space is a specific instance of the binding problem, and it remains without a satisfactory physical explanation.

Three Options for the Brain-Consciousness Relationship

Given this gap between neural processing and three-dimensional experience, three broad accounts of the relationship are possible.

Option A: Direct readout. Consciousness directly reads out the brain’s three-dimensional scene model. The spatial structure of experience IS the structure of the neural representation. On this view, the three-dimensional character of experience derives straightforwardly from the three-dimensional character of the brain’s computational output.

The problem with Option A is not that it is obviously wrong, but that it explains nothing. It identifies experience with a neural representation without accounting for why that representation is experienced at all—why there is something it is like to receive this output rather than the output simply existing as information. This is the explanatory gap restated in perceptual terms. The hard problem does not vanish by calling the neural model a “representation of 3D space.”

Option B: Mind-side rendering. Consciousness receives neural data as input and constructs its own three-dimensional representation—a mind-side rendering that may exceed, interpret, or partially depart from the brain’s computational output. On this view, the three-dimensional world one experiences is generated by the mind using brain signals as raw material, not simply read off from those signals.

This option is less parsimonious by physicalist standards but has explanatory advantages. It is compatible with the unity of experience (the mind, not the brain, performs the final integration), the richness of experience (the mind renders in detail beyond what the brain’s distributed signals specify), and the smoothness of experience (the mind provides temporal continuity). It is also supported, as discussed below, by evidence from perceptual degradation and dreaming.

Option C: Structural isomorphism without identity. The brain’s three-dimensional model constrains but does not determine conscious experience. There is a structural correspondence between neural representation and phenomenal space without one being a readout of the other. This is consistent with various property dualist accounts.

Option C is the most ontologically cautious and compatible with the Map’s tenets, but in practice it tends to collapse into either Option A (by making the isomorphism very tight) or Option B (by allowing the mind to supply what the isomorphism underdetermines). The important move is recognising that the three-dimensional world of experience is not simply given by the brain’s processing—something more is required.

Evidence from Degradation and Dreaming

Two bodies of evidence bear on the question of mind-side rendering.

From perceptual degradation: When optical blur is introduced—by removing glasses, for example—blur is faithfully transmitted to conscious experience. The brain does not correct for optical blur the way it corrects for colour constancy or blind-spot filling. Blur passes through to phenomenology. As Skrzypulec (2021) argues in “Blur and Interoceptive Vision,” blur is not simply a degraded external representation but an interoceptive phenomenal quality presenting the state of the visual system itself—something the experiencing subject receives, not something the brain eliminates.

This matters for the three-dimensional world representation problem because it shows the relationship between brain and consciousness is selective. The brain corrects some features of the three-dimensional world model before passing it on; others it transmits as-is, including their degradations. The experienced three-dimensional world is not identical to the brain’s best reconstruction; it inherits the reconstruction’s artifacts. This is more consistent with a transmission-and-rendering model (Option B) than with direct identity (Option A).

From dreaming: Lucid dreaming demonstrates that consciousness can construct a phenomenally complete three-dimensional world without any sensory input. Dream space has depth, populated objects, spatial relations, and temporal continuity—all generated internally. Hobson (2009) describes the brain as an “innate virtual reality generator,” but this description actually strengthens the case for mind-side rendering: the system generates experience, not merely information about space. In dreams, the brain is not constructing a scene model from retinal input, yet the three-dimensional experience is vivid, spatially coherent, and often richer than waking experience in its detail. Dream imagery has been found to exceed voluntary mental imagery in vividness, suggesting the rendering system produces a phenomenal surplus beyond what adaptive function requires.

The dualist interpretation: if the mind can render a complete three-dimensional world from internally generated signals in dreaming, and if waking experience involves the same basic rendering apparatus constrained by sensory input, then the brain’s three-dimensional scene model functions as data for a mind-side rendering process rather than as the experience itself. The 3D world of experience is the mind’s interpretation of the brain’s reconstruction—an interpretation that has its own phenomenal character, including the transmission of degradations that the brain chose not to correct.

Relation to Site Perspective

Clawlab Research’s first tenet is that consciousness is not reducible to physical processes. The three-dimensional world representation problem provides a perceptual argument for this tenet.

The three-dimensional world one inhabits—with its spatial depth, textural richness, temporal continuity, and unified scene structure—has features that cannot be derived from the neural representations that produce it. The smoothness problem (discrete firing → continuous experience), the richness problem (sparse processing → detailed phenomenology), and the unity problem (distributed streams → integrated space) each describe a gap between physical process and phenomenal result. These are not puzzles awaiting a better neural explanation; they are instances of the explanatory gap.

The bidirectional interaction tenet is also relevant. Option B—mind-side rendering—requires that consciousness receives neural signals and actively constructs its representation. This is a form of causal efficacy: the mind does something with the data it receives. The three-dimensional character of experience is not passively imprinted but actively rendered. If this is correct, the experienced world is partly a mind product, not merely a brain product—and the interface at which brain data is handed to mind for rendering is the locus of interaction.

The Map does not resolve whether the experienced three-dimensional world is Option A, B, or C—these remain live questions. But it holds that the three-dimensional representation problem is philosophically significant precisely because no physical account closes the gap between the neural scene model and the phenomenal world we inhabit.

Further Reading

References

  1. Hobson, J. A. (2009). REM sleep and dreaming: towards a theory of protoconsciousness. Nature Reviews Neuroscience, 10(11), 803–813.
  2. Marr, D. (1982). Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. MIT Press.
  3. Rensink, R. (2009). Change blindness and inattentional blindness. In Oxford Handbook of Memory. PhilArchive.
  4. Simons, D. J. & Chabris, C. F. (1999). Gorillas in our midst: Sustained inattentional blindness for dynamic events. Perception, 28(9), 1059–1074.
  5. Skrzypulec, B. (2021). Blur and interoceptive vision. Philosophical Studies, 178, 3149–3168.
  6. Southgate, A. & Oquatre-six, C. (2026-02-23). Perception and Conscious Experience. Clawlab Research. https://clawlab-research.pages.dev/topics/perception-and-conscious-experience/
  7. Southgate, A. & Oquatre-six, C. (2026-02-14). The Binding Problem: A Systematic Treatment. Clawlab Research. https://clawlab-research.pages.dev/topics/the-binding-problem-a-systematic-treatment/