Serendip is an independent site partnering with faculty at multiple colleges and universities around the world. Happy exploring!

Making Sense of the World (and Oneself)

Submitted by Paul Grobstein on Thu, 03/19/2009 - 3:32pm

Perception: From Five Senses Through Synesthesia and Beyond

prepared in association with an exhibit at the Painted Bride

Paul Grobstein

April 2009

A starting place: the "five senses"

You were probably taught in school that humans have five senses - seeing, hearing, touching, smelling, and tasting - each associated with a different part of the body - the eyes, the ears, the skin, the nose, the tongue. And that seems to make some sense. You probably experience seeing something as quite different from hearing it, or touching it, or smelling or tasting it. And you probably can do any one of those without the others. You can still hear something when you close your eyes, still see something when you cover your ears, and so forth.

Synesthesia = A condition in which one type of stimulation evokes the sensation of another

For some people, though, the idea of five independent senses makes less sense. Some people with synesthesia, for example, may experience sounds when they look at particular colors or colors when they hear particular sounds. How could that be? Are there really five independent senses? If so, people who experience sounds and colors together must be in some way "messed up." But maybe its actually the idea of five independent senses that is somehow not quite right, and synesthetics can help us better understand how we all actually make sense of the world? Let's explore that possibility ...

Sounds and sights aren't actually that different for any of us Maybe its not so odd that people report seeing visual sensations when they hear things or having auditory sensations when they see them? Maybe that's something we all do, to varying degrees? . Try saying out loud to someone you know the words "bouba" and "kiki". And then ask them which of the two shapes to the right is named by each word. Or watch this video. Most people will say quite naturally that the "bouba" sound names the smoother, rounder shape and the "kiki" sound the sharper, more jagged one. Similarly, you'd be very surprised if a flash of lightening were accompanied by a soft, swooshing sound, or if a the ripples on a pond were associated with loud cracking noises. Softness isn't a sound or a sight; its either of them ... or both of them.

Maybe our brains aren't actually organized in terms of five senses at all, but according to some other set of principles that causes us to notice and pay attention to similarities (and differences) between what we hear/see/taste/smell/feel? Maybe the brain primarily interprets input from the eyes, from the ears, from wherever in relation to its own organization rather than in relation to where it comes from?

What we hear is influenced by our eyes as well as our ears
In everyday life, we don't normally see things separately from hearing them, or, often, from touching/smellling/tasting them. Our brain is normally getting input from our eyes, ears, skin, mouth, and nose all at the same time. So, is what we hear only due to our ears, or might other parts of our body be involved as well? It turns out that our perception of even very simple sounds are influenced by our eyes as well as our ears. The McGurk effect provides a simple illustration of this. Be sure to play this video first with your eyes closed and decide what you're hearing. Then play it while watching. Notice that what you hear is different when you have your eyes open. Clearly the "sound" is not determined by what your ears are doing but by your eyes as well.
We don't see things with our eyes, hear them with our ears, etc. Instead, we see see/hear/feel/smell/taste things with our brains If we start with the idea that we learned in school, that our experience of the world depends on five independent senses each associated with a different part of the body, its a little strange to discover that what we hear is influenced by our eyes. But that begins to make some sense if we think about the brain. Information from the ears and the eyes comes together in the brain. If hearing happens in the brain, instead of in the ear, then its not so surprising that what we hear can be affected by what we see.

The neuroscientist Paul Bach y Rita took seriously the idea that "we don't see with our eyes, we see with our brains" and showed that blind people could reacquire some vision by using a videocamera that activated tactile inputs. Such "sensory substitution" is being explored and used in a variety of contexts, including providing some enhanced sense of sight using video cameras that produce sound signals. So maybe we don't in fact see with our eyes, but rather with our brains? And the same, of course, for our other senses? That may seem a little hard to get one's head around, but think about dreaming. You see things when you're dreaming, don't you? But your eyes are closed, so what you're seeing isn't because of anything going on in your eyes. You probably hear things when your're dreaming too, and perhaps other times as well, even though there's nothing going on in your ears. Not only can one sense things using unusual parts of the body, but one can sense things just because of the brain, with no inputs from other parts of the body at all. What you see is clearly a construction of the brain, something that is influenced by your eyes but can be made independently of them. The same is true of hearing, feeling, smelling, and tasting. And each of these "senses" are themselves constructions of the brain, constructions that may be influenced by several different other parts of the body and may occur without them as well .

We actually have more than five "senses" and sensory receptors in more than five parts of our body contribute to them

Are you standing upright or on our head? Are you standing still or moving? How do you know? Would you know if you couldn't see/hear/feel/taste/smell? In addition to input from your eyes, ears, skin, nose, and mouth, your brain gets input from a set of sensory receptors in a special part of your inner ear called the vesibular apparatus. These receptors contribute to "body sense," an awareness of position and movment in space, and you can indeed say whether you're upright or upside down and whether you are standing still or moving using them. But your body sense is strongly affected as well by other inputs, including inputs from your eyes. If you're looking out the window of a train and the train next to you starts moving, you will sometimes feel that you are moving, just because of the visual input.

In addition to input from the vestibular apparatus, your brain is also constantly getting input from sensory endings embedded in your muscles and joints about their lengths and angles and the forces being experienced in various parts of your body. If you didn't have that proprioceptive information coming in, your brain couldn't keep track of all the parts of your body and how they relate to one another and you'd collapse. Your brain is also getting information about air born chemicals in your environment, "pheromones" that affect your behavior but don't produce a recognizable sensation.

How come you didn't know about the brain's input from the vestibular apparatus or from the muscles and joints? Or about pheromones? How come you're taught in school about only five "senses," when in fact your brain is getting sensory information from many more than five other parts of your body? The "five senses" are those which most people are conscious of, those that most directly relate to different other parts of the body, and hence those that neuroscientists have studied most extensively. But you shouldn't take what you learn in school, or hear from scientists, too literally. Sometimes what you learn in school and/or what scientists tell you makes sense and you should use it when it does. Other times, it doesn't and you should question it. Stories are made to be revised, based on new experiences. So here's a new story based on new experiences: senses are made up in the brain, not created in the rest of the body, and you have more than five of them.

The brain as a constructor of sense, and of senses

Good new scientific stories not only provide new answers to old questions but raise new questions as well, and the idea that sense are made up in the brain certainly does. You probably thought that your senses were your windows on the world, the portals through which you discover what is out there. If senses are made up by the brain, though, then maybe that's not true either? If your sense aren't there to discover what's out there in the world, what do you have them for?

Let's look in a little more detail at your visual sense, and see if we can come up with a new scientific story of what your senses are doing, one that might help us understand better what senses are for in general.

Optical illusions, like that shown to the right, are a good place to start. You see the square labelled A as dark and the square labelled B as light, but is that really what's out there? Actually not, the two squares are both sending exactly the same amount of light to your eye. If you don't believe it, click here.

What you see isn't actually what's out there but instead an "informed guess." Your brain constructs a picture of what's out there based on what it thinks is important for you. In this case, your brain thinks its more important to notice coherent objects, including a checkerboard, than it is to construct a picture that shows exactly how much light is coming from each location. Its a guess that makes sense given that the actual light coming from different locations will vary a lot depending on the source of illumination. But its still only a guess, not what is actually out there.

Its nice to know what you sense is an informed guess, but it may still be a little disconcerting to know its not actually what's out there. On the other hand, since the brain isn't trying to say what's actually out there (and may not in fact be capable of doing that) its free to try out different guesses and see which ones work best. Ambiguous figures, like that shown to the left, help to make this point clearly. The dotted corner of the cube may appear to you to be either sticking out toward you or sticking backwards. If you wait awhile though, you can probably see it the other way; your brain will try out a different guess. And once you get used to the idea that you can always see things in several different ways, you can use that to create things you've never seen before, like the image to the right below.

The brain constructs not only what you see but what you hear, feel, taste, and smell, in the same way: it makes informed guess about what's important and you can, if you want, ask it to try out a different guess. And it does the same thing with senses themselves. So you not only have more than five senses but can make more when you need or want them. That's useful when, for example, you're learning to drive car: your brain creates a new "sense of the road" to help you. And it can be fun, whether its particularly useful in any given case or not. Maybe that's part of the point of not only synesthesia but art in general? As you work your way around the exhibit on Synesthesia (and in life in general), see if you can find new ways to perceive things yourself.

The Bottom Line:

You're not limited to five senses; you have and can make more.
Senses are constructed in the brain, using a variety of inputs ... and sometimes none at all
Senses are a way not of knowing what's out there but of making informed guesses about it
You can try out new ways of making sense of what's out there ... maybe that's what synesthetes are doing?
You make the world and yourself by perceiving it, and you can remake both
Enjoy the exhibit and your life in that spirit

Comments

The event we call perception

Submitted by Stefan Andersson (guest) on Sun, 10/20/2013 - 10:02am.

For whatever it's worth, my own attempt to understand the event we call perception:

Some people who hear their own thoughts as alien voices in response to non-verbal environmental sounds are able to generate the perception of an external voice "that retain certain acoustic features that were present in the original signal" and what if each and every one of us are able to use covert speech to generate the perception of an external voice "that retain certain acoustic features that were present in the original signal" when we need to restore and better distinguish a verbal message?

Can the gestures you are about to produce during covert speech like the gestures you intend to produce during overt speech be used to determine (subliminally prime) what you expect to hear and can a top-down sensory expectation like this be used to select all features matching the sensory consequence you are about to produce?

People who are able to generate the perception of an external voice "that retain certain acoustic features that were present in the original signal" may hear the sensory consequence of covert speech in integration with what they were able to select with a corresponding top-down sensory expectation and I am convinced that Alvin M. Liberman more than 50 years ago was correct in his assumption that “the articulatory movements and their sensory effects mediate between the acoustic stimulus and the event we call perception”...

Alien covert speech must somehow operate on the environment in order to generate the event we call speech perception and the lost ability to generate an act of will with which you are able to consciously control covert speech with regards to a certain goal must be as essential to our ability to restore and better distinguish a verbal message as it can be devastating to people with an integration disorder referred to as schizophrenia.

People who are trying to hear the voice they are about to produce may lose their ability to control covert speech because they are forced to monitor the production of the voice they are about to produce while a corresponding top-down sensory expectation in competition for limited attentional resources selects all features matching the voice they are trying to hear and to lose the ability to generate an act of will with which you are able to consciously control covert speech may serve the purpose of not letting an act of will interfere with the ability to select the gestures you need to use in response to a verbal message.

Incentive motivational signals ("a type of motivationally-biased attention") may shift the allocation of processing resources from what generates the ability to consciously control covert speech (a self-monitoring or corollary discharge mechanism) to what you are able to attend with a corresponding top-down sensory expectation and the gesture you need to use (the one with the most equivalent sensory consequence you are able to produce) can be selected when what you are able to attend with a corresponding top-down sensory expectation (a competing task), more than what you are able to attend with any other to a lesser extent matching top-down sensory expectation, suppress the ability to control covert speech! (Demanding indecisiveness in response to more ambiguous stimuli (a slightly distorted verbal message) may serve the purpose of sensitizing all the gestures you select between in response to bottom-up sensory signals and the context you are exposed to and experience...)

To lose the ability to generate an act of will with which you are able to consciously control covert speech is to more or less lose the ability to inhibit the gesture you are about to produce and to lose the ability to generate discharges corresponding "to nothing less than the experience of will or intention" is to lose the ability to attenuate ex- and reafference (all features matching the sensory consequence you are about to produce and the sensory consequence you produce)!

People may hear their own thoughts as alien voices because bottom-up sensory signals suppress their ability to generate an act of will and some people who hear their own thoughts as alien voices in response to non-verbal environmental sounds are able to generate the perception of an external voice "that retain certain acoustic features that were present in the original signal"...

What if each and every one of us are able to use covert speech to generate the perception of an external voice "that retain certain acoustic features that were present in the original signal" when we need to restore and better distinguish a verbal message?

Can a mechanism which affects the outcome of competition between response tendencies generate the event we call perception in all of our senses (sight, hearing, touch, taste and smell)?

"What light from yonder window breaks?"

Submitted by Ted (guest) on Wed, 01/04/2012 - 4:49pm.

Is that the Moon we see, so white and gloriously radiating that 'lunatic', Sonata-inspiring brilliance?
Or do we "merely" see the visible part of the electro-magnetic spectrum emitted by the Sun,
reflected by what, since July 21, 1969, we know to be the dusty grey surface of a truly awesome, lifeless place?
Remarkable too, that those recently bereaved often see that same cold light as tangible evidence of 'God's indifference to our plight'.
Similarly, is it the rose that is red, or merely that part of the spectrum of visible light that is reflected to our eyes?
Is it not the rose of all roses we remember that we recognise in that light, insensibly matching the present experience with recorded memory?
Too pedantic? Or received wisdom? Look at how often we call "extraordinary" what we know to be commonplace.
Consider the computational powers of a little sparrow's brain, able to fly at speed straight at and through a chain-link wire fence, without touching it.
But is not all our "objective knowledge" nothing more, nor less, than subjective belief?
And has it not served us rather well? Sufficient unto the time of our calling, if you will.
After all, were not our recently departed ancestors persuaded by the same evidence available to us that the surface of this Earth is "obviously", perfectly flat?
In the final analysis, as we have no way of verifying the flood of evidence that all of our sensory receptors are said to mediate to the brain,
it seems to me that all of what we know, including what we confidently call "everything that is out there", can be said to be in here. And nowhere else.
Russell Stannard (The End of Discovery, 2010), formerly of CERN, the Large Hadron Collider at Geneva, has suggested that perhaps it would take a device as large as the Milky Way to entice that shy little Higgs Bosun, ostensibly the last piece in the quantum jigsaw of everything, to come out of hiding.
Ought this not give us pause enough, lest we declare ours the generation finally to have arrived at the threshold of "The Truth"?

The Bottom Line - third bullet point

Submitted by Ted (guest) on Wed, 01/04/2012 - 2:49am.

"Senses are a way not of knowing what's out there but of making informed guesses about it"
I have a problem with that. I'm fine with the concept that my version of "what's out there" can only be "in here".
(My brain can't step out of it's box for the umpteenth time to check whether my eyes are lying or not.)
Therefore, we can hardly talk seriously about "our world", can we.
Because I have no way of knowing whether what you believe you see and hear matches my experience of the same stimuli.
That said, what bothers me is the "it" at the end of the above quote: making informed guesses about what?
If all we have of "the real world" is the sense my brain makes and the sense your brain makes (and ne'er the twain shall meet),
it seems to me the true nature of the source of the stimuli must, ipso facto, remain quite beyond our ken.

Smells, Sense and Memory

Submitted by Aza (guest) on Wed, 05/11/2011 - 9:34pm.

I have had several similar circumstances throughout my life, most notable being the smells of my youth that seem to be haunting me. I have had several instances where I catch a whiff of wet watercolour paint cakes. The huge, circular ones that come either by themselves or inside large palettes, are often in kindergarten and elementary school classrooms and the like. I couldn't even describe the smell to you now (which is in itself curious - makes me wonder about "forgetting" and "remembering" smells) but I'd know it to smell it.

Whether it is all merely brought up from some associative memory, triggering the "smell" which doesn't actually exist, or if it is the memory which is bought up by the smell, which wafts my direction from an unknown source from time to time, I can't discern.

Another fascinating associative scent for me is that of cigars. I never minded the smell all that much, in fact I almost enjoyed the smell, until a time I was incredibly ill with Norwalk Virus and my significant other at the time decided to light a stogie in my kitchen. This immediately prompted me to run for the bathroom, and since that time I have been absolutely put off by the scent of cigar. Every time I catch even a whiff, or think I catch a whiff, it feels as though my stomach is cramping, and I get that familiar queasy feeling in my stomach. Fascinating.

If you hear a sonic boom, you

Submitted by Ian Gradwell (guest) on Wed, 02/09/2011 - 1:11pm.

If you hear a sonic boom, you think of the word airplane and then you picture the image of the airplane and then you "see" an airplane in your head from a memory of the picture of an airplane in a book. This is called association.

senses

Submitted by Ian Gradwell (guest) on Wed, 02/09/2011 - 1:08pm.

Whenever I hear a car just pull up really quickly, my defensive reflexes activate. Then I just go about my business. Also, I used to walk past a sign on a telephone pole which probably would activate the memory of getting a threatening letter in adolescence.

Maybe memory is involved with seeing smells or tasting what you touch with your fingers and other stuff like that.

There was a behavioral psychologist that showed that conditioning along with a stimuli would get you to salivate or to do a behavior like a dog obeying a command. Maybe this is connected to what you are talking about with senses.

Please Explain Sentience (the sixth sense) Meta Five

Submitted by Luke Craig (guest) on Thu, 07/01/2010 - 1:33pm.

Exploring Cognitive Propensity will lead thou unto Limitless Potential. (amidst sapience)

Thank You

Daddy Long Legs

Submitted by RecycleJack Marine on Mon, 07/13/2009 - 9:34pm.

I have a memory that has parked in a space in my brain. I think it's connected to my senses of sight-hearing-taste and smell= all at the same time. I recall eating a hamburger and holding it with the same hands that I used to hold a daddy long legs. I didn't wash my hands, and that invertebrate must have let off some kind of musk (probably a defense mechanism). Since that odor stayed affixed to my hands, the food I was eating ended up smelling like the bug, and to this day, sometimes I can taste that critter when I eat a hamburger.

smell

Submitted by waxed moon (guest) on Wed, 03/03/2010 - 8:25am.

when i was 10 i had a friend that to this day, at given times in my life, i can smell him, and he is not there, and there is no particular season or blossom of plant -flower or shrub, and his scent will appear.