Reality and Virtual Reality?

Reality Check:
The Possible Detection of Simulated Environments
Through Observations of Selected Physical Phenomena

Benjamin Olshin

And yet, and yet… Denying temporal succession, denying the self, denying the astronomical universe, are apparent desperations and secret consolations. Our destiny… is not frightful by being unreal; it is frightful because it is irreversible and iron-clad… The world, unfortunately, is real; I, unfortunately, am Borges.”

— Jorge Luis Borges, “A New Refutation of Time”

Reality, Virtual Reality, and the Formulation of the Question[1]

This paper attempts to address some specific questions concerning virtual reality or simulated environments of the type discussed in Hilary Putnam’s “Brain in a Vat” scenario and depicted in the popular film “The Matrix.”[2] It does not delve into some of the more complex questions of the nature of knowledge and the role of mind in definitions of “reality” — those topics have been explored at length elsewhere in the philosophical literature. Rather, this paper addresses in simple terms two basic problems. The first: Is there a test we could devise to see if we are living in a simulation of the sort depicted in Putnam’s “Brain in Vat” or “The Matrix”? The second problem, a more difficult one in some respects, is this: What exactly is the difference between that kind of simulation and what we currently believe to be our physical reality?

The film “The Matrix” has been mined almost ad nauseam for its philosophical content, but that does not discount the fact that, for once, a piece of popular culture has made at least some people think about a profound metaphysical question.[3] That question, indeed, concerns the very nature of reality. In actual fact, the first film in “The Matrix” series uses this question primarily as a starting point; the second and third films move away from that, and explore, with mixed success, other philosophical issues. But the basic premise of the original film — the idea that we could all be living in some kind of simulated environment — remains the most intriguing, and I often utilized that aspect of the film when I taught philosophical issues of reality, simulation, and perception in to university undergraduate students.

One of my students once told me that the reason the filmmakers of “The Matrix,” the Wachowski brothers, dropped the initial philosophical issue of reality versus simulation after the first film was that there was “nowhere to go” with it. “Fine, so they are living in a simulation? How much more can you do with that fact?” What the student was saying was true in a sense, but I persisted in using the film as a teaching tool, primarily to explore that basic philosophical question concerning our perceptions of what appears to be a world external to us.

In a methodology that perhaps has become typical in undergraduate philosophy classes, I used to begin my discussion with a reading of Plato’s “Allegory of the Cave,” then moved on to look at Descartes’ “Evil Genius” scenario, followed by a study of Hilary Putnam’s “Brain in a Vat” question. A viewing of “The Matrix” film came at the end. In this manner, students gained an understanding that this philosophical question of reality versus illusion is an old one. Moreover, they learn that these terms themselves are complex, and that the question even of how to approach this philosophical challenge remains unclear.

My focus in class was always on the same basic question: “How do we know whether or not we are living in a simulated environment of the kind that Putnam describes?” Student discussion tended to circle towards the same general answers, such as: “I know this world around me is real because I can see it, touch it, taste it…” The common rejoinder to that was: “Yes, but that could all be simulated.” Another comment was: “I think that this perceived world is real because it follows consistent physical laws — every time I drop this pen, it falls and hits the desk. If, one day, I dropped the pen and it did not fall, then I might begin to believe that there is something wrong with our reality — that all this is not real after all.”

Theses discussions always tended to grind to a halt, insofar as no one in the class, of course, could prove things one way or another. Certainly, in terms of serious philosophical explorations in epistemology and phenomenology, one could go much further. Moreover, there are also fascinating differences and similarities in Western and Eastern philosophical approaches to the question of reality and illusion, as well as the concept of “life as a dream.” But the essential philosophical question as to how we know what is reality and what is illusion remains problematic, with no resolution in sight. And, indeed, since this class was a morning class, ending about noon, I frequently would joke with the students that even though we might be living in an illusion and my hunger for lunch might be simulated, I was indeed going to respond to that hunger by ending class and getting a sandwich.

After a couple of years of this kind of classroom exposition of this basic question, I began to find that I was satisfied neither in philosophical nor pedagogical terms. The discussions in class had tended to go down the same paths, and while this was fine for the students, whose very learning of this question was sufficient for the purposes of the course — an introduction to philosophy for non-majors — the lack of resolution or the apparent absence of a new approach to the question itself bothered me and some of the more intellectually curious members of the class.

One aspect of the scenario that had come up in the discussion had struck me as important, but it had not been immediately clear for what reason. In Plato’s “Allegory of the Cave,” the prisoners are freed by an outside figure; so, that leads to the question frequently asked by students, “How did that outside person ever become free?” Is there a way for a prisoner to realize on his own that he is living in a world of shadows, and so then free himself? “The Matrix” film addresses this question, but in a rather breezy way, implying that, yes, there was someone who “woke up” through their own realization and volition. Neo, the main character in the film, doesn’t quite do that — he is awakened by the external figure of Morpheus. But Neo is portrayed as always having been a bit suspicious about the so-called “real world” of his boring office job and tiny apartment. But on what evidence had he developed his suspicions? As I always had emphasized to my students, one should operate from evidence, and there seems to be no evidence that our world is a simulation or otherwise.

As implied earlier, of course, the philosophical question of “reality versus illusion” brings up a whole host of other issues. One such issue is that of definitions: What do we mean by “real”? Another issue concerns location: Is there a material world outside of our physical bodies, or does everything reside in the brain? Moreover, there is the question of physical reality itself: If there is a physical, material world, what is its basic constitution? Is there such a thing as “matter”? These and other related issues are ones that I hope to examine in a subsequent paper; here, I would like to deal with the specific question of a physical reality as we apparently perceive it every day versus an illusion akin to the type that Putnam describes. To put it precisely, the question I wish to ask is this: “Is there a test we could devise to see if we are living inside a simulation of the sort described in Putnam’s “Brain in Vat” or the kind shown in the ‘The Matrix’?”

The philosopher Nick Bostrom addressed a related question in his discussion of simulated environments. He approached the matter through a study of probabilities and logical extrapolation.[4] His conclusion was that one might conjecture that we are living in a simulation right now, a simulation that is being run by our distant descendents. But of course, Bostrom offers no proof offered other than what he sees as a high probability that these descendents could build such a simulation.

The general popular answers given to the question of how to determine if we are living in a simulation are few. First, some claim that an occurrence of events with excessive coincidence, defying the laws of chance, might be an indication that the world we are living in is not “real” in some way. A series of highly unlikely coincidences, in other words, might reveal that we are in a simulation that is being actively manipulated by outside forces, or that we are in a simulation that has developed some “glitch” in its operating system. In a similar manner, some would argue that a clear violation of a physical law that we are used to observing, such as the law of gravity, would indicate, again, an intervention in or manipulation of the simulated system, or a technical problem in that system.

But these are not actually tests of what we might call our “perceived reality”; they do not provide a grounds for determining whether what we perceive is indeed a physical world external to us or an artificially-generated illusion. True, the witnessing of a series of unreasonable coincidences or the observation of a sudden violation of physical laws would be grounds for skepticism.[5] And, as Bostrom describes, a civilization might one day be able to build a device that could generate an artificial perceived reality, so that the simulation would provide what we would label as consistent physical or sensory experiences. However, these are all conjectures; what is needed is an actual method for the physical detection of a simulation — from the inside of that simulation.

The “Brain a Vat” Scenario

Before we look at a possible approach, we need a precise definition of problem again: Are we living in a “directly-inputted” simulation of the kind discussed in Putnam’s “Brain in a Vat” scenario or “The Matrix,” rather than as physical beings in a physical universe? A key issue in addressing this question concerns the interface. In a very simple way, we can say that as physical beings in a physical or material reality, our interface with that reality is via our senses. In this description, there is the “outside world” of streets, buildings, trees, grass, other people, and so on, and we receive information about these things through the senses of sight, sound, etc. The interface, then, is the sensory “layer,” where information, such as light waves hitting the retina or sound waves hitting the eardrum, is converted to electrical signals that are transmitted to the brain. This is a model that has been questioned by some scientists, but will suffice for our purposes here.[6]

In contrast to this model, Putnam offers a scenario where electrical impulses go from a simulation-generating computer directly into a disembodied brain:

[I]magine that a human being (you can imagine this to be yourself) has been subjected to an operation by an evil scientist. The person’s brain (your brain) has been removed from the body and placed in a vat of nutrients which keeps the brain alive. The nerve endings have been connected to a super-scientific computer which causes the person whose brain it is to have the illusion that everything is perfectly normal. There seem to be people, objects, the sky, etc.; but really, all the person (you) is experiencing is the result of electronic impulses travelling from the computer to the nerve endings. The computer is so clever that if the person tries to raise his hand, the feedback from the computer will cause him to ‘see’ and ‘feel’ the hand being raised. Moreover, by varying the program, the evil scientist can cause the victim to ‘experience’ (or hallucinate) any situation or environment the evil scientist wishes. He can also obliterate the memory of the brain operation, so that the victim will seem to himself to have always been in this environment. It can even seem to the victim that he is sitting and reading these very words about the amusing but quite absurd supposition that there is an evil scientist who removes people’s brains from their bodies and places them in a vat of nutrients which keep the brains alive. The nerve endings are supposed to be connected to a super-scientific computer which causes the person whose brain it is to have the illusion that... [7]

In Putnam’s setup, the brain can not tell the difference between that interface — i.e., the point where the wires connect to the brain tissue — and an interface involving actual sensory organs, described above.[8] A “brain in a vat” is quite different from a brain in a physical body equipped with sense organs. However, on the face of it, there indeed seems to be no way that a disembodied brain could tell that the world it was perceiving was simulated and that the sense organs had been bypassed. Putnam adds:

When this sort of possibility is mentioned in a lecture on the Theory of Knowledge, the purpose, of course, is to raise the classical problem of scepticism with respect to the external world in a modern way. (How do you know you aren’t in this predicament?) But this predicament is also a useful device for raising issues about the mind/world relationship.[9]

He goes on to examine the “brain in a vat” problem in terms of language, but in some senses this sidesteps the tangibility of the scenario.[10] That is, our current technology is sufficiently advanced to consider the viability of virtual reality of an increasingly sophisticated sort at some point in the future.

Let us return to the basic problem that Putnam outlines; it is very hard to attack head-on, since a disembodied brain is unable to apply any clear kind of test on the information that it is receiving as input. The conscious part of the brain will not suspect that there is anything out of the ordinary, as long as what it receives continues to be consistent with its prior “embodied” experience. One can engage in philosophical debates about this scenario, and Putnam set the modern debates on the issue in motion with the claim in his book Reason, Truth and History that we are not “brains in a vat.”[11] Since then, the debates have continued, but again, without deeply investigating the issue of what kind of determinative test might be applied to the experience that the brain was having.

Again, if the hypothetical simulation were running properly, the “brain in a vat” would have a consistent, even banal experience. There would be nothing to make the brain “question” whether it was disembodied or not. This makes the problem of a test rather challenging. Therefore, it seems best to look at this problem through the use of an analogy, one that also involves perception and simulation, and that utilizes an interface. In fact, it is the interface that will allow us to establish a testing framework.

Building an Analogical Model

Of course, we are not able to actually carry out Putnam’s scenario. That kind of interface is, at present, impossible to construct — and perhaps that is fortunate! However, what is needed is an analogical scenario, one where the subject would have consistent, unremarkable everyday perceptions through regular input of information.

The mechanism of television provides such an analogy. A television screen provides the viewer with an interface onto an entire world. A viewer of television can see and hear other people, experience physical phenomena, witness various events, and so on — many of the components of a perceived existence. Of course, this is just visual and audio input, but these are perhaps the most critical in the daily assessing of our perceived reality.[12]

We can propose a situation where a subject is wearing a kind of virtual reality helmet with goggles with a headphone/microphone set. The goggles feed them with continuous television video input. The subject has worn these goggles (and the other apparatus of the helmet) for their entire life; they have been led to understand that what they see and hear is a real world that is external to them, in a manner somewhat analogous to the inhabitants of Plato’s cave. They see streets, buildings, trees, grass, people, and so on. All the phenomena they perceive via their video goggles are physically consistent: they see the sun and stars, observe the changing seasons, and other physical phenomena, and perhaps even attend lectures on physics and chemistry. Everything they perceive is uniform, and follows the physical laws that they have learned.

One day, however, they see a computer lab — again, of course, via their goggles — filled with cathode-ray tube (CRT) video monitors.

They look at the computers and take a look at the screens of the monitors. They see images there, perhaps the computer logo or a picture, but they also notice a sort of buzzing line or band that seems to move slowly across the screens from top to bottom every few seconds or so.

Anyone who has watched television has probably seen this at some point: when you see a CRT computer monitor on a television show, you will see the scanning lines of the monitor “captured” by the television camera. On television, computer screens appear to have a kind of scrolling band, because the television cameras are unable to portray a “static” computer screen — they actually capture the scanning action of the CRT computer monitor displays.

In other words, if you try and point a television camera at a CRT computer screen to record the image there, instead of seeing the stable image that your eyes perceive, you get a flickering image or a line rolling across the screen. This flickering or “banding” is caused by the difference in scanning frequency between the CRT computer monitor and the television camera capturing the image. The difference also comes from the way the phosphor dots of the CRT computer monitor are perceived by the human eye as compared to the way that they are picked up by the television camera.[13]

In a CRT computer monitor, an electron beam scans in a horizontal direction across the back of the screen. When the beam hits the screen, it lights ups the phosphor dots on the screen, and these glow, just for a fraction of second. The beam moves across the back of the screen fast enough that the entire screen seems to glow continuously. So, we see a steady image. A television camera, however, “perceives” the dots as glowing for a much briefer period, and thus they record what seems to be a flickering or “banded” image. Even if the CRT computer monitor has a scanning rate of 60 Hz and the television camera recording the image is taking a frame every 1/60 of second, the band that one sees on the monitor will roll across the screen the because monitor and camera are not synchronized.

Returning to our scenario, the goggle-wearer observes the CRT computer screens in the lab and notices this flickering band. They ask the person whom they “see” in the lab what this flickering is. The person in the lab, of course, sees no flickering or banding whatsoever. That interference is the product of the interface between something in the external scene or simulation that is being filmed, i.e., the computer lab, and the goggle-wearer’s perceptual mechanism — i.e., the video goggles.

In similar manner, if a person were watching a television screen, but told that it was a window onto the real world, they would be able to determine that it was not, in fact, the real world. This is because they could communicate through that “window” with the person working in a lab, who had explained to them how CRT computer monitors work, and who had stated that there was not any flickering or banding at all in the typical operation of such monitors.

This, then, is a reality test, wherein the information needed to determine if the observed world is real can be found through observation of that “world” itself and through logical reasoning. The test in this case is accidental: the setup of the simulation itself has given rise to a phenomena — the flickering or banding — that can only be explained by the postulation of the existence of some kind of mechanical interface. The participant or “goggle-wearer” would be able to conclude from within the simulation that something was amiss, since the physical laws they had learned could not explain the phenomenon they were observing. The only logical hypothesis would be that they were experiencing a mediated or simulated existence.

Extrapolation to the Observed Environment

The scenario described above clearly delineates the nature and role of observer and the configuration of the simulated reality. The simulated reality is a continuous televised scene, one that includes streets, buildings, trees, grass, and, of course, the lab with the CRT computer monitors. The observer receives the scenes as their visual input, taking them for a real world — indeed, the only world they have ever known. We on the outside can argue that while the observer may think that their world is real, it actually is no more than a projected image. We know this because we can see the mechanism of the simulation in its entirety.

However, this “God’s eye view” is something that we do not possess in our own reality. We are inside the system, and everything that we perceive we tend to assume has a material reality. But our goggle-wearer assumes the same — at least until they stumble across the evidence that something is amiss. That evidence is, in short, a physical phenomenon that the observer can see rather easily, but a phenomenon that does not fit with their existing knowledge of their “physical” universe. According to the physical laws that the goggle-wearer has worked out over the years in living inside the simulation and learning physics there, there should be no interference pattern where one ends up being observed.

The idea of an “interference pattern” is an important one. Interference patterns come from two systems that are in some way dissonant, unsynchronized, or incompatible. In a physically consistent material reality, one would not expect any unexplainable interference patterns. More particularly, interference patterns usually require an interaction between two sources. In our reality, we encounter interference patterns only when observing, for example, two sets of waves intersecting.[14] We do not observe interference patterns where there apparently is only one set of waves — something that is happening in our television model described above, where the only apparent set of waves are those from the CRT computer monitor.

In everyday observation, in fact, if we see an interference pattern and have accounted for only wave source, we must go looking for the second source. For example, we might observe that a car has polarized windows, but only because we, the observers, are wearing polarized sunglasses and so a perceptible interference pattern is created.[15] The most important thing is that we might be able to detect that we are in a simulation if we observe a physical phenomenon that is explainable solely through the postulation of the presence of some kind of interface through which the simulation is inputted to us — in our model above, it is the television system and video goggles.

How do we move from such a model to real-life experiment? Unfortunately, it is not quite so simple. First of all, we are required to make what is a rather slippery distinction between what we might call a physically consistent material reality, such as the one we assume that we live in from day to day, and a simulation of the kind postulated by Hilary Putnam and rendered as a nightmarish existence by the makers of “The Matrix” film.

But let us pass over that problem for a moment, and look around us. We apparently perceive a physical reality, comprising matter made of atoms and their constituent sub-atomic particles. When we observe this physical realm, we might find the kind of aberration that our goggle-wearer encounters. Perhaps it is something as simple as the odd interference pattern we observe when sending even single photons towards a photographic film in the famous double-slit experiment.

To “check” our reality, we might talk about constructing a test where we take a physical phenomenon that does not easily fit with our current model of physics. We then could postulate that our experienced reality is, in fact, a simulation. We would go on to posit some kind of true, external reality beyond our perceived one, since ours is just a simulation. We would conjecture that the mysterious physical phenomenon that we have observed is the result of the “interference” between our mode of experiencing that simulation — i.e., our “goggles” — and something in the simulation itself.

Admittedly, this is a strange kind of science, where such an odd explanatory model — a whole external reality beyond the one we typically perceive! — is used to address a physical phenomenon. Indeed, this kind of conjecture certainly violates the principle of “Occam’s Razor.” But the point here is that the construction of such a physical test might be possible, if our “television” analogy is valid. We might be able to detect a simulation from within the simulation.

Directions for Further Investigation

A problem in attacking the question of simulation versus reality with the science of physics is that if we are in a simulation, we are working with a physics that is itself a product of the simulation. The philosopher David J. Chalmers calls this the physics “used by my envatted self in the counterfactual world.”[16] In other words, in such a scenario when we observe matter and space, we are actually just observing the inside of a simulation. In that scenario, physical laws — such as the “fact” that c is the speed limit of the universe — are not laws at all, but simply the parameters of the simulation. Returning to our analogy, we might ask whether such physical constants or parameters might tell us something about the possibility of determining the existence and nature of the simulation. In our television analogy, it was the inexplicable interference patterns that led to the conjecture that there was a simulation running. In this case, it might be the parameters themselves that suggest we are in a simulation rather than a physical reality.

This paper has assumed a certain rough definition of “physical reality.” Earlier, we used the term “perceived reality,” because we can only really talk about what we or our scientific instruments can detect or perceive. Naturally, many philosophers and scientists will affirm that there is indeed a distinct material reality, and that our perception of it is irrelevant to its definition. However, in our television scenario, the mode of perception matters, since that is what reveals the simulation. Moreover, in our television scenario, there is a clear difference between material reality and simulation. And in our actual world of experience, if that is also the case, the construction of a testable hypothesis might be possible. As we noted earlier, interference phenomena of the kind encountered in the double-slit experiment may indicate a direction for future investigation.

But in contemporary physics, the problem is that while we experience an apparent physical reality every day, we have been unable to pin down quite what it is. In fact, asking “what it is” may not even be an appropriate question. Historically, when scientists looked into the structure of matter, they took a kind of reductionist view, seeking to find smaller and smaller component parts: atoms, protons, neutrons, electrons, quarks, and so on. But there seems to be a limit to the empirical method here, as we reach the limit of our ability to carry out detection, observation, and measurement at a certain scale.[17] Physics falls into asking questions such as “Is matter, in the end, actually composed of empty space?” These kinds of questions are suggested by recent physical theories involving spatial “strings” and “membranes.”[18] Hilary Putnam posited the scenario of our simply being “brains in a vat,” but the question really is “What’s a vat?” If we cannot define our perceived reality very clearly, we certainly are going to be unable to define what a simulation might be, nor the external reality that might be running it.

In some sense, modern physics is already positing a two-level existence, insofar as it speaks of both an observable physical world that we live in every day, and a kind of “substructure” beneath that of “strings,” “membranes,” or pure space. Of course, that substructure is not quite the same as a simulation that is being actively manipulated by some outside force. But we are still talking about the difference between what seems to be a world of perceptible matter and a “lower level” world or “bottom layer” that is something else — indeed, perhaps not a “thing” at all. What that world behind this one might be is open to speculation. In our analogy above, it was a continually projected television image. Chalmers, cited earlier, has looked at various possibilities, including the idea, related to that of Stephen Wolfram and, earlier, Konrad Zuse, that “behind” our perceived physical universe is a computational one[19].

These ideas present difficult problems in several respects. For one, the supposition of “another level” of reality, e.g., one that is running a simulation, leads to the classic problem of infinite regress: is that “other level” itself simulated? Also left unanswered is the question of what the “substrate” of these realities might be; at some level, is there a distinct substrate that the virtual realities lack? Even in a simpler scheme, where we might conjecture the denizens some kind of solid reality perpetrating a simulation to which we are subject, there is the issue of the medium through which the simulation is taking place. If we are living in a simulation, a scenario Bostrom describes, we need to consider how it might be constructed. More profoundly, if we are living inside a simulation, do we have the tools to find out or to talk about intelligently? This paper, of course, suggests that we might be able to do so, but not without difficulty.

Related to this is the problem of differentiating between the observer and the observed reality. Contemporary physics implies a fairly distinct difference, even with the suggestion of observer influence posited by quantum mechanical theory.[20] We walk around feeling ourselves to be observers of a rather separate material world. Even physicists usually consider themselves as entities carrying out experiments using apparatus that is separate from themselves. But both we and that world are made of the same basic physical substance — atoms, “strings”, whatever — and yet one side is conscious of the other. That is, how can one group of atoms — ones that happened to be configured into the form of a human brain — be conscious of, observe, and measure another group of atoms, the ones comprising the external, material world?[21]

On an even more subtle level, a real problem might be a model where, in fact, there is not another reality beyond our perceived one, and our perceived reality is the only one. This would prevent us from putting forward an explanatory framework that relied upon the postulation of some further substrates beyond the ones we usually observe, but it would lead to a host of other questions.

Finally, most challenging might be a look at the philosophical tools themselves that we are using. Perhaps it the very nature of language, with its prepositional bias toward hierarchies, that has led us to build problematic models that deal with “levels”, “substrates”, “components” such as sub-atomic particles and strings, and so on — models that ultimately lead to the kind of apparently irresolvable and paradoxical questions we have encountered here. The primary languages of physics have been German, French, Russian, and English, but one wonders what a physics that arose in a non-Western language framework might have looked like. Might it have had philosophical or at least descriptive tools that we lack?

This paper has addressed one aspect of the larger philosophical issue of reality versus simulation, through the adoption of a basic physical model. In that model, there is an observer who believes that they are having a direct, unmediated experience with an outside material reality. In actuality, the observer has been interacting with a televised simulation. They discover this through a relatively simple observation: that the interference pattern they perceive in a certain situation seems to have no source. They conclude that since interference patterns are always the result of at least two sources, the second source must be from the very mode in which they perceive their world. The key point here is that it may indeed be possible to construct a physical test for this kind of simulation. More significantly, this means that at least in this kind of case, one can detect a simulation from within the simulation itself.