Learning, categorizing, creating

Serendip is an independent site partnering with faculty at multiple colleges and universities around the world. Happy exploring!

Remote Ready Biology Learning Activities

Remote Ready Biology Learning Activities has 50 remote-ready activities, which work for either your classroom or remote teaching.

A Serendip experience
Simple Networks, Simple Rules: Learning and Creating - 6/6

The bottom line ... and beyond

We started by wondering whether simple things interacting in simple ways could learn. And the answer is yes. So its certainly not true that computers can do only what they are told to do. Or, at least, its not true that one has to tell them explicity what to do for every example of what you want. You can give them a general set of operating instructions, and a few specific examples, and the computer will not only learn the specific examples but use these to itself create a rule, a categorizing scheme, that it can apply to additional cases.

What's particularly interesting is that the rule the computer creates may or may not be the one you had in mind. You might have had in mind that the shorter and thinner something gets, the more it should be called a rabbit (like one scheme the computer came up with), but the examples experienced are, for the computer, equally consistent with most things, even quite short and thin things, being elephants. This may seem silly, but it actually says something quite important about how many different solutions there are to particular problems, about the extent to which experience can account for observed generalizations, and probably about brains and people as well.

So:

Learning CAN occur in a simple, "mechanical" system

Clearly computers can learn. So too then can anything that consists of interacting parts with properties and rules of interaction that bear close relations to those in the model with which we've been working. More generally, learning, at least in the sense we've been using it (which includes the ability to create new categories), ought probably to be thought of not as a special characteristic of humans, but rather as a quite general characteristic, not only of living things but of at least some non-living things as well. Any system in which the interactions of the parts of a system itself bring about changes in the parts and/or their interactions should be expected to show some form of learning, at least in the sense of exhibiting at any given time some record of past activity. Among other examples, biological evolution clearly exhibits a form of learning (though one dependent on a quite different set of interacting elements and learning rules: "Both the evolutionary process and the immune system collect information about the environment by a process of registering its actions on internally generated variance ..., Grobstein, 1988).

There may be lots of "solutions" to a given "problem"

What is "silly" in one context may be profound in another. The neural network we've been working always creates a category which distinguishes between elephants and rabbits. It always learns to distinguish the two, but may or may not learn to distinguish them in a way that matches the way you distinguish them. It learns a distinction, but maybe not the distinction you had in mind. Depending on exactly why you wanted to teach the network something in the first place, this may be either an exasperation or a charm. If you think there is one "right" way to do things, and want them done that way, then the network is clearly exasperating. On the other hand, it might never have occurred to one that there are a variety of different possible ways to slice a given cake, in which case the ability of the network to generate a variety of different solutions is not only charming, but quite significant. Some solutions might be better under some circumstances, others better under others, and, regardless, one can stretch one's own imagination by seeing what the networks do.

Learning is NOT simply a function of experience

No one doubts, at least in this day and age, that experience can affect behavior ... in fact (in this day and age) there is some tendency to believe that behavior is largely or entirely a function of one's experiences: one is what one has learned to be. There is increasing evidence that things are not quite so simple, that genes, while not determining behavior either, also influence it. In this context, its worth further emphasizing (if not belaboring) the point that there are lots of different solutions to any given problem, and that the solution a network arrives at in not totally determined by the learning experience it has. Two networks, given exactly the same training experiences, will, in general, come up with different solutions (and hence exhibit different behavior) if their initial starting characteristics (the starting synaptic weights) are different. Experience, then, is only a PART of learning; the other important part is the state of the system at the time the relevant educational experiences occur. The point, of course, is not only that a given set of experiences might have different effects on computers and on elephants, or on rabbits and on people, but might in fact have different effects on different people. Why might the starting conditions vary? Perhaps because of previous experiences, perhaps because of genetics. Perhaps because of a combination of the two, mixed with a little genuine serendipity.

Is it a good thing or a bad thing that different individuals may learn different things from the same set of experiences? That depends, of course, on one's perspective. If one wants uniformity out of an educational process, its not so good, in which case one ought to try to make not only the learning experiences for everyone but also the starting conditions for as similar as possible. An alternate perspective is that there are lots of somewhat different ways to do the same task, and people who learn to do it differently from one another can as a consequence subsequently learn additional things from each other (as they might learn from elephants or rabbits or computers).

That learning depends not only on particular experiences but also on pre-existing structure has some significant broader implications as well. In the sense in which it is used here, learning, with its accompanying creation of somewhat arbitrary categories, is not, of course, something which occurs only in a classroom. It is instead the basis of most (all?) human understanding, including scientific. As William James put it, in his 1890 Principles of Psychology: "What we experience, what comes before us, is a chaos of fragmentary impressions interrupting each other; what we think is an abstract system of hypothetical data and laws". The former are the experiences from which learning occurs, the latter the categories one uses to make sense of "reality". The issue is how one gets from one to the other, whether the categories are inherent in the experiences, and hence a reflection of the "real world", or are something else. James answer: "Every scientific conception is, in the first instance, a 'spontaneous variation' in someone's brain. For one that proves useful and applicable there are a thousand that perish through their worthlessness. Their genesis is strictly akin to that of the flashes of poetry and sallies of wit to which the instable brain-paths equally give rise. But whereas the poetry and wit (like the science of the ancients) are their own excuse for being ... the 'scientific' conceptions must prove their worth by being 'verified'. This test, however, is the cause of their preservation, not of their production ..." In short, the validity of science and related forms of advancing human understanding of external "reality" depend fundamentally not on the purity (or "objectivity") of their observations, but rather on the continual testing and retesting of generalizations which have, at their root, an origin in arbitrary categories.

A given system may not be capable of achieving a "perfect" solution to all conceivable problems.

So, our simple network can learn some things ... but not others. What should we make of that? One possible conclusion, of course, is that learning isn't in fact so simple, and hence that while simple things interacting in simple ways can do some kinds of learning, there must be some other explanation entirely for other kinds of learning. That learning isn't simple is true; there are lots of different kinds of learning. But the corollary, that one must need something other than simple interactions of simple things to account for all kinds of learning, doesn't necessarily follow. Our simple network, and the history of research on neural networks, provides a case in point. We've been working with an example of what is called a "perceptron", originally developed by Rosenblatt in 1958. A formal critical analysis of the capabilities and limitations of such "single layer" networks was published by Minsky and Papert in 1969, with a resulting lessening of interest in network learning capabilities. In 1986, Rummelhart and others showed that at least some of these limitations could be overcome by changes in the learning rules which permitted multilayer networks ("back propagation networks"). Interest in the learning capabilities of neural networks has exploded, and research continues in a number of directions, with the clear conclusion that there are a variety of different networks (in terms of both architecture and learning rules), each with its own capabilities and limitations. So the issue of whether (relatively) simple interactions of relatively simple things can account for all of learning remains open (and one wouldn't want to bet against that conclusion).

Is there ONE kind of network, yet to be found, which could itself account for all kinds of learning? No one knows, of course. But some of the conclusions we've reached here may be relevant in thinking about the question. Learning is something which results simply from experience, but instead something that results from interactions between experience and pre-existing structure. It seems likely that different pre-existing structures (in terms of both network architecture and learning rules) would affect the ability to learn things (just as the pre-existing synaptic weights affect the categories created). If so, the bottom line may well be that there is no single, optimal learning network, and that "learning" itself will always need to be understood in terms of a series of different networks optimally structured for different tasks. And this, of course, presumes that one knows in advance all learning tasks. If learning itself generates new sorts of things to be learned, one can certainly confidently expect (and happily so, if one likes open horizons) that no single network will ever "solve" the learning problem.

Going Beyond

Rosenblatt's Perceptron Learning Algorithm, a Java implementation which allows exploration of variations in learning parameters

Neural Nets, on line version of a book by Kevin Gurney, Psychology Department, University of Sheffield, United Kingdom

An Introduction to Neural Networks, by Leslie Smith, Centre for Cognitive and Computational Neuroscience, University of Stirling, United Kingdom

Neural Computing, course notes from Department of Electronics and Computer Science, University of Southampton, United Kingdom

Backpropagator's Review, by Don Tveter

Showcase, from Intelligent Financial Systems Ltd, includes examples of practical neural net use and some Java tutorials illustrating back-propagation networks.

FAQ for comp.ai.neural-nets newsgroup

A Brief Introduction to Genetic Algorithms, by Moshe Sipper, Swiss Federal Institute of Technology

The first few chapter of James' Textbook of Psychology are available on line, together with the texts of several articles by him, at Classics in the History of Psychology, from York University, Canada

William James, an extensive web resource by Frank Pajares, Division of Educational Studies, Emory University

Mind and Body: Rene Descartes to William James, by Robert Wozniak, Department of Psychology, Bryn Mawr College

Science Education, from Serendip

I'm intrigued. Can I go back to the beginning, please?

Written by Paul Grobstein. Applet by Bogdan Butoi.

| Forum | Complexity | Serendip Home |

Send us your comments at Serendip