I just finished watching a NOVA series about human evolution. It wasn't particularly well produced, but it was very informative. Pre-humans Homo erectus and Homo neanderthalensis (aka Homo heidelbergensis, because the ancestors of Neanderthals were found near Heidelberg, Germany) migrated out of Africa before Homo sapiens had evolved in Africa. Actually, to be clear, only Homo erectus actually migrated out of Africa, then the population that settled in Europe evolved into the Neanderthals as they adapted to the more temperate climate (they had to deal with winter). As an aside, a population of Homo erectus that made it to Indonesia evolved a smaller stature and today it is referred to as a species of "hobbit," although it is not the hobbit of J.R.R. Tolkien fame.
The remaining population of Homo erectus in Africa experienced dramatic climactic shifts (mainly drought) that reduced the population almost to extinction, but one small coastal population adapted to sea life and became the ingeniously tool using and enterprising Homo sapiens.
The Homo sapiens population grew and then also started migrating along the same geographic paths that earlier Homo erectus had taken up through the Middle East, then spreading into Europe, Asia, Indonesia, and eventually even to Australia (it's not clear that Homo erectus ever made it as far as Australia). Homo sapiens was a lot more crafty and technologically savvy. It would be nice to think that we interbred with Neanderthals and/or Homo erectus when we encountered them already living in Europe and Asia, but the available genetic evidence suggests we didn't (no shared genetic fingerprints to date).
So we probably either out competed them for resources or killed them off or drove them out, until they had no where left to go and died off. It sounds a lot like how humans act towards outsiders today. Maybe we can evolve ourselves away from that using our other great adaptation - CULTURE.
Because evolution is still going, and that means humanity can still evolve and adapt and improve. But it also means if we don't use the brains that nature gave us, we might go extinct from our own technological excesses. We are the first species that has ever had to adapt to its own adaptiveness.
As a distinct species, we are only about 200,000 years old, tops. Virtually embryonic. Granted, we have done a lot in the time allotted.
I think the human brain size is still growing. As an aside, I know that my own skull is gigantic, as evidenced by the rarity of hats that will fit my head. However, my individual Darwinian fitness is questionable. I may be too smart for my own good. But that's for the process of natural selection to decide (noting that it does not consciously decide anything in a rational way; that is but a figure of speech).