There are indeed changes in brain size over evolutionary time

• Brains don't fossilize (a point to be returned to), but one can work with endocasts.

• Increases in brain size for some groups have occurred over evolutionary time.
  
  from Harry J. Jerison, Paleoneurology and the Evolution of Mind, Scientific American, January, 1976

• Increases in brain size over evolutionary time are not unique to humans.
  
  from Harry J. Jerison, Paleoneurology and the Evolution of Mind, Scientific American, January, 1976

• Many increases in brain size seem to involve a widening size distribution rather than a replacement of smaller brained by larger brained animals.

• Such increases in brain size over evolutionary time do not indicate that larger brains are in any biological sense "better adapted" or "more advantageous" than smaller brains. They may simply result from the intrinsic "playfulness" of the evolutionary process. Such intrinsic playfulness may be a signficant contributor to the apparent trend of greater complexity over evolutionary time.