A Random Walk?

In his book Full House: The Spread of Excellence from Plato to Darwin, Stephen Jay Gould presented the idea that the appearance of progress in evolution can be explained as “a random walk.”  According to Gould, in evolution there is a left bound, a minimum at zero complexity, but no right bound on complexity.  Evolution thus has only one direction to move in, and that is toward greater complexity over time.  There is no need to posit a direction of increased complexity or progress, only a random process, which leads to increased complexity because it can’t really lead to anything else. 

While the idea that evolution is a random walk certainly is reasonable, I find at least one reason to question it: If we accept some form of the big bang theory, then our universe starts off in a highly disorganized state.  Yet several billion years later, when the first life appears on earth, the universe has become organized into stable galaxies and planetary systems.  Just how to account for this increase in cosmic organization is a rather contentious issue, but I don’t see any way that “a random walk” describes this process.

Further, at a certain point in the history of evolution, we find one species, the human, who starts to organize his world.  Over a few hundred thousand years, we find this creature going from organizing simple shelters to creating such highly organized entities as the Library of Congress, the I-Pod, and the space program.  Again, how to account for this massive increase in organized complexity is rather contentious, yet again, it cannot be accounted for by a random walk.

So the question arises, if what happens between the big bang and the rise of life on earth seems to have a direction of progress, and the development of human learning and technology clearly has a direction of progress, should we feel so confident that evolution, which lies between these two, lacks a direction of progress? 

There is a lot of talk about a theory of everything in physics, but one thing seems clear to me – such a theory of everything won’t actually explain much of anything outside the realm of physics.  I would like to predict here that somewhere in the future there will be another kind of theory of everything that will actually explain a good deal more.  This theory will be a theory of organization – a theory that comprehensively accounts for how the universe self organizes and in the process of self organizing generates new forms of organization, such as the algorithmic organization by which genes produce organisms and ideas create buildings and machines.  Darwinian Evolution will be a part of this larger theory, rather than a theory somewhat isolated from the other forms of development and organization occurring in the universe.