Blog

What’s Driving Evolution?

Wrdcloud of what Drives Evolution


According to the Second Law of Thermodynamics we live in a universe that irreversibly decays over time.  But if this is indeed the case, then it begs the question:   How does Evolution’s Spontaneous and Progressive Complexity occur without some form of External Organizing Force?”


Evolution vs. Entropy

The Second Law of Thermodynamics (SLOT) is a law of physics that deals with “Spontaneous Change” (i.e. change that occurs without any external direction, change that happens all by itself…)

The SLOT is, more precisely, the law of physics that deals with how energy distributes itself within a thermal system, always moving spontaneously and irreversibly to “Thermal Equilibrium”.

In everyday terms the SLOT is simply the fact that hot coffee and cold milk, if left unstirred, will spontaneously mix themselves (in both composition and temperature), and will never spontaneously un-mix.

Despite the fact that this seems rather obvious and trivial behavior, the SLOT is nonetheless considered to be one of the most fundamental and important laws of physics — and the reason for this exalted status is that the SLOT is both a “Probabilistic Law” and also the “Law of Maximum Entropy”!

“Entropy” is a concept that deals with amount of “disorder” in a system, and it is widely understood that the spontaneous gravitational pull to maximum entropy is not restricted to simple thermal systems; but that all systems, if left undisturbed, will spontaneously gravitate towards a state of maximum disorder — a state that would seem to be the exact opposite of Nature’s “spontaneously self-organized complexity”.

This apparent conflict between physics and natural evolution obviously begs the question:   “How does Evolution manage to spontaneously generate such incredible Complexity in the face of the SLOT?”

How can natural complexity spontaneously arise in a universe dominated by the SLOT and its spontaneous and irreversible pull to disorder?  What exactly is the “Source” of all of Nature’s spontaneous order and complexity?

The Export of Entropy

In 1977, the Belgian chemist Ilya Prigogine won the Nobel Prize for Chemistry, for his work on his “Theory of Dissipative Structures”.  Prigogine’s theory suggests that complex ordered systems can indeed come into existence if these systems are open and capable of “exporting”  their internal disorder, to the external environment.

But while this theory would seem to go some way towards solving the paradox of how order can occur without negating the SLOT, it still does not manage to identify what fundamental forces are actually driving evolution to evermore progressive complexity.   Physics has as yet offered no explanation for “evolution’s progressive arrow of time”

As it turns out however, the resolution of this paradox is actually quite easy.  To resolve this apparent conflict  between physics and natural evolution we need merely to focus on a very simple fact that has been consistently overlooked about the “probabilistic” SLOT; the fact that it relies heavily on the “Law of Large Numbers (LLN)”…

The LLN

Most people are familiar with the concept that if we toss a coin four times, we won’t necessarily get a 50/50 split of heads and tails: indeed, we could actually get 4 tails in a row.  But if we toss the same coin a million times, we will almost certainly get something close to a 50/50 split.  It is the LLN that ensures that one million coins tosses will produce an average of 50% heads and 50% tails.

[Note: In the simplest possible mathematical terms, the reason the LLN works so well is that the number of independent tosses (i.e. 1,000,000) is significantly larger than the number of options available to each toss (i.e. 2 – heads or tails).]

The SLOT states that left undisturbed all systems gravitate towards the “most probable state”, a state that is referred to as “thermal equilibrium”.  In reality however, the achievement (and sustainment) of thermal equilibrium relies heavily on the number of independent elements (of the system) being significantly larger than the number of energy options available to each element.  Which means that the chances of any “statistical deviations” from the “most probable state” are extremely small, and consequently the system as a whole will (virtually) always exhibit uniformity.

So although on the “microscopic level” (of particle interaction) there is a lot of energetic dynamics and non-equilibrium abnormalities, these dynamics and abnormalities are normally invisible on the macro “system level” thanks to both the “Damping” and “Balancing Effects” of the LLN…

The RLLN

Our universe is fundamentally a universe of “systems”, and the probabilistic pull of equilibrium is a concept that is applicable to all fluid and fluid-like systems.

Now, in a thermal system there are billions of tiny particles which interact through collisions, but other than that we can more or less say that they behave completely independently of each other.

Systems however, where the parts – be they particles, elements, components, entities, agents, organizations, etc – behave independently of each other are actually quite rare.  Many systems are populated by adaptive elements or agents, and the behavior of these agents has a tendency to weaken the gravitational pull of equilibrium by engineering the “Reverse Law of Large Numbers (RLLN)”…

[Note: Since the LLN relies on the number of independent elements being significantly larger than the number of options available to each element, there are therefore two things can engineer the RLLN and they are:  either the number of independent elements in the system comes down, or, the number of options available to each element goes up…]

RLLN 1:  Emergent Positive Feedback

In all fluid-like systems, the LLN ensures the spontaneous movement to a “global equilibrium”; however for very small regions within these systems, there are not enough particles to ensure a “local equilibrium”.  At the very lowest level within all systems, random fluctuations are undampable and occurring all the time which means that local imbalances are constantly, and randomly, flittering in and out of existence.

Occasionally these random temporary fluctuations can randomly be very persistent.  In a thermal system this is naught but a mere statistical curiosity, but in a complex adaptive systems it can easily happen that some parts within the system will begin to adapt to these persistent fluctuations; and often such adaptation can serve to amplify the imbalance even further, and in so doing, further extend the fluctuation’s duration.  Thus random local fluctuations can lead to the localized emergence of positive feedback which reduces the independence of the elements and ultimately has an unbalancing and reversing effect on the LLN.

RLLN 2:  Insufficient Negative Feedback

Positive Feedback however is not the only thing that can engineer the RLLN.  Since the LLN effectively operates like a negative feedback system (in that it dampens a system to a equilibrium) it should be no surprise that the movement away from equilibrium could also be the result of insufficient negative feedback.

So although complex fluid-like systems might gravitate towards equilibrium, many can hold themselves some distance away from equilibrium by exhibiting excessive undampable adaptation and innovation.  Adaptation and innovation effectively increases element “Optionality” and such increased optionality among the elements of the system can also engineer the RLLN…

Self-Integration For Free

So the reality of probability driven dynamics in the natural world is that just as the LLN pulls a system to thermal equilibrium, so too the RLLN can hold, or drive, a system away from equilibrium.

But ultimately what is most interesting about all of this probabilistic behavior is that: while strong positive feedback in isolation can cause the emergence of self-reinforcing local segregation; and while insufficient negative feedback in isolation can cause the surfacing of incompressible innovative diversity; the most interesting stuff actually occurs at the intersection between the two…

Positive reinforcement in a system of great diversity can spontaneously produce surprisingly complex “Integrated Diversity”.  So in other words,

with the co-emergence of diversity

Complex-Integration comes for Free!..


Natural Complexity

Evolution’s progressive complexity is often portrayed as spontaneous “Self-Organization”, but this is not the exactly accurate.  The secret sauce of evolution’s spontaneous and progressive complexity is actually spontaneous “Self-Integration”.

In the simplest possible terms, Natural Complexity emerges from the finely-tuned self-integration of co-emergent self-organized diversity; and as a consequence “the complex whole is forever becoming greater that its less complex parts”…

Matrix of System Dynamics - Copyright - Kieran D. Kelly

So there we go, Natural Complexity explained (by mathematical probability).  “Easy Peasy Lemon Squeezy”…

In a universe supposedly dominated by the SLOT what drives nature’s progressive evolution is simply the mathematical interplay of the two distinct forms of the Reverse Law of Large Numbers…

What drives evolution’s spontaneous and progressive complexity is the interplay of insufficient negative feedback and strong positive feedback; or in other words what drives evolution is The Interplay of Random Innovation and Natural Reinforcement…

Matrix of Evolution Dynamics - Copyright - Kieran D. Kelly

The Coming Age of Accelerated Evolution!


What’s Driving The Evolution Of Everything?…  Back in 1994 I read a book called “Complexity – The Emerging Science at the Edge of Order and Chaos”.  In this book the author suggested that theoretically complexity should not really occur naturally because it seems to defy the Second Law of Thermodynamics (SLOT) and the concept of maximizing “Entropy”.

I had come across the SLOT as a student and knew that it was related to the concept of the spontaneous distribution of “Heat”, but could not remember much about the concept of “Entropy”.  A little research however revealed — much to my surprise — that seemingly “entropy” is generally associated with “disorder”; and that according to physicists because heat spontaneously distributes it means that everything in the universe spontaneously moves towards maximum disorder; and that means that the universe itself is in fact irreversibly decaying over time.  This I found surprising, because if this assertion is true then obviously it does indeed beg the question:   “How does Evolution manage to spontaneously generate such incredible Complexity in the face of the SLOT?”

This apparent conflict between physics and natural evolution sparked my curiosity.  I became curious to understand firstly why physicists would believe that the spontaneous distribution of heat must mean that we live in a universe of irreversible decay; and secondly, and more importantly, I began to wonder “what actually is the sourceof Evolution’s Self-Designing Complexity?”

Evolution Dynamics

“So?…” you might well ask.  Have I managed to formulate any sort of an answer to this tricky and paradoxical question?  Well yes, I think I have.  The answer to why physicists believe what they do is simply a matter of history; but the answer to what’s driving evolution is much more interesting…

In the simplest possible terms, the SLOT is about the spontaneous pull of thermal equilibrium, and so it effectively about how the SLOT operates like a natural “Negative Feedback System” (like a natural thermostat) in that it pulls (or dampens) a system to the equilibrium state.  However a fact that has been consistently overlooked about the physical behavior of the “Second Law of Thermodynamics” is that it relies heavily on the mathematical behavior of the “Law of Large Numbers (LLN)”.  The LLN essentially says that a very large number of independent things will most likely exhibit the “Most Probable Distribution” (which in thermodynamics is the equilibrium state).

Consequently a system can be held away from thermal equilibrium by two things.  Firstly, the negative feedback effect of the LLN can sometimes not be strong enough to dampen the system to equilibrium.  And secondly, the strength of the LLN can sometimes be weakened by a reduction in the independence of the elements within the system.

So these two things can in effect act as resistance to the pull of thermal equilibrium.  But moreover, and more interestingly, it is the interplay of these two types of resistance that can (without any external direction) naturally drive evolution away from a featureless state of thermal equilibrium, to some extraordinarily creative complexity.

In the simplest possible terms, Evolution is the result of the interplay of Insufficient Negative Feedback, and Strong Positive Feedback; simply the interplay of “Incompressible Dynamics”, and “Self-Reinforcing Dynamics”

  • Insufficient negative feedback allows the surfacing of creative diversity and “Incompressible Dynamics”…
  • Strong positive feedback drives local symmetry-breaking and emergent “Self-Reinforcing Dynamics”…
  • And the interplay of both drives the “self-integration of co-emergent diversity” and emergent “Complexity Dynamics”…

Matrix of Feedback Dynamics

The above matrix of feedback dynamics can be rewritten in terms of evolutionary dynamics, thereby clarifying Complexity Dynamics as simply the result of “The Nonlinear Interplay of Random Innovations and Natural Reinforcement”…

Matrix of Evolution Dynamics - Copyright - Kieran D. Kelly


Entropy Dynamics

Moreover, and this is what is really interesting, both of the above matrices can also be represented in terms of thermodynamic entropy and “Information Entropy”.

Matrix of Entropy Dynamics - Copyright - Kieran D. Kelly

Information entropy is not much known outside the domain of computer, but it is a different type of entropy to thermodynamic entropy.  Whereas thermodynamic entropy is a measure of disorder, information entropy is a measure of uncertainty and unpredictability.

[Note: The concept of information being a measure of uncertainty can sometimes confuse people.  But think of it this way.  By its very nature, to you “information” is unpredictable, otherwise it would not be “information” — it would just be stuff that you already knew…]


Information Creation

Essentially natural complexity is, in fact, “self-integrated information”.  All natural complex systems are characterized by the fact that they have “low thermodynamic entropy, but high information entropy”; consequently these systems can be considered to be highly organized but unpredictable nonetheless.

So while most people might think about evolution in terms of “an ecosystem of plants and animals”, the reality is that plants and animal (and the ecosystem itself) are really just “information structures”, and evolution simply keeps creating evermore complex information structures.

So despite what most people might think, evolution is not solely a theory about the emergence of life, but a more Generalized Meta-Theory about the Emergence of Everything (of which biological evolution is merely a special case).  In the broadest possible terms, “Evolution is Information Creation”…

Accelerated Evolution

Information is created from information.  Evolution is simply the constant integration of information.  Evolution’s progressive complexity is simply Mother Nature creating evermore complex information by “integrating stuff” as she goes along.  This constant integration means that “Nature’s complex whole is forever becoming greater than its less complex parts”

Evolution created us, and we in turn have created (and are continuing to create) evermore complex information structures.  But in these early years of the 21st century it could reasonably be argued that in our ever-more rapidly interconnecting world (in 2017), we are likely fast approaching a phase transition in human development, a transition to a whole new “Age of Accelerated Evolution and Information Creation”.  And this coming age will not be dominated by the old linear paradigm of predictable cause and effect, but by a whole New Nonlinear Paradigm of unpredictable “Integration and Emergence”…


The Century of Complexity and Creativity

In Conclusion:  Physics tell us that to understand the world we need simply to understand “the dynamics of cause and effect”; but the simple dynamics of cause and effect fail quite miserably when it comes to explaining “Natural Evolution and Emergent Complexity”...

However, understanding the “meta-concept” of evolution is going to turn out to be much more important than anyone might previously have thought.  Because although science may have spent the last 400 years honing its understanding of The Linear Dynamics of Cause and Effect, the reality of life in the 21st century is that the really interesting stuff will increasingly result from the “Universal Creative Dynamics” of “Adaptive Integration and Emergent Complexity”…Matrix of Universal Dynamics - Copyright - Kieran D. Kelly

Why this will be the Century of Complexity…


the-century-of-complexity


ALTHOUGH science may have spent the last 400 years honing its understanding of The Linear Dynamics of Cause and Effect”, the reality of everyday life in the 21st century is that the really interesting stuff is increasingly the result of “The Nonlinear Evolutionary Dynamics of Adaptive Integration and Emergent Complexity”…


Linear and Nonlinear Dynamics

“Physics” is the ultimate science of cause and effect.  Physicists like to describe their science as the hardest of “hard science” because physics can claim to be governed by hard and fast scientific “Laws”.  This of course would seem to imply that many of the so-called “soft sciences” are in some way not quite as elevated, not quite as good.

In truth however we could say that physics is an “easy science”, and the soft sciences are “difficult” because the “laws” of physics only really work in the absence of “noise”, and yet the everyday world of the soft sciences is full of noise because most everything is continually battered and buffeted by “constantly changing feedback” which can generate wild “nonlinear dynamics”.

In reality all dynamics have feedback (and resultant nonlinearity), it is just that some dynamics have much less feedback than others.  Physics is, in a sense, the science of “dynamics with negligible feedback”, the science of “linear dynamics” — or in other words it is the science of the nonlinear stuff that can be safely “compressed” into neat linear differential equations which express neat linear “cause and effect”.

New Paradigm

In the simplest possible terms, linear dynamics are dynamics where the effect is proportional to the cause, and nonlinear dynamics are where the effect can be disproportional to the cause.

Physics, it would be fair to say, has throughout its 400 year history, actively tried to steer clear of messy nonlinear dynamics, and in so doing has actively established a paradigm of linear dynamics; a linear paradigm of cause and effect.

But then, out of the blue, in the latter part of the 20th century, along came both “Chaos Theory” and “Complexity Theory” which between them seemed to hint strongly at a completely different paradigm; a nonlinear paradigm of “Integration and Emergence”…

Chaos Theory and Complexity Theory

Unfortunately however nobody seems to have been paying the proper attention, and so chaos theory and complexity theory, as they stand today, are still a bit of a mishmash of concepts and don’t really have agreed upon definitions.

Chaos Theory, for example, is generally associated with the relatively vague notion of the so-called “Butterfly Effect” (or as the academic community like to say “sensitivity to initial conditions”), but this association has, in my opinion, done more harm than good — for it is misguided, and its misdirection has merely served to mask the true nature of chaos.

Complexity Theory is similarly afflicted, but rather than analyse all the pros and cons of all the various definitions of both Chaos and Complexity, I will instead simply offer my own definitions…

Defining Chaos and Complexity

In my opinion chaos is not primarily characterized by sensitivity to initial conditions; but by emergence of decisions points and the resultant sensitivity to choice.  Chaos is simply “adaptive instability”; it is “unresolved internal adaptation to feedback, surfacing as turbulent diversity on the system level”.  So, in the simplest possible terms, we could say that

“Natural Chaos is Incompressible Adaptive Diversity”…

Complexity is simply the resolution of adaptive instability.  Complexity is the result of the “adaptive  integration of co-emergent diversity”, which ultimately results in “emergent complex systems” that have effectively “organised themselves into existence”.  So, in the simplest possible terms, we could say that

 “Natural Complexity is Self-Integrated Diversity”

Matrix of Universal Dynamics - Copyright - Kieran D. Kelly


Integration for Free

So is any of this important?  Absolutely it is!   We live in a world of nonlinear dynamics, some of it compressible, most of it not.  Physics and Chemistry generally deals with the compressible stuff, but Natural Evolution and Emergent Complexity deals with the rest…

To understand Natural Evolution and Emergent Complexity is to understand how in any system of adaptive entities many diverse things can randomly occur and be reinforced; but furthermore, and much more importantly, it also tells us that

With the co-emergence of diversity,

Complex-Integration comes for Free!…

This “Integration for Free” is evolution’s secret sauce.  Evolution drives the emergence of great diversity, which leads to the natural integration of co-emergent diversity, which drives the next level of emergence.  This constant interplay of integration and emergence means that evolution naturally ratchets-up complexity over time, and consequently “the complex whole is forever becoming greater than the sum of its less complex parts”

Natural Creativity

Some years after its publication, the English philosopher Herbert Spencer summarized Darwin’s theory of evolution as being the “Survival of the Fittest”, but unfortunately this description, although popular, is somewhat misleading.

Evolution is not about the “Survival of the Fittest”; evolution is about the “Integration of the Optimally Adapted” (or more precisely, the optimal integration of optimally adapted diversity).

There is a subtle difference between these two descriptions; the former implies anti-synergistic competition, while the latter implies synergistic collaboration.  When we look at the natural world it is obvious that Nature favors integrated diversity over uniformity.  Mother Nature does not employ an asymmetric “winner takes all” strategy, but prefers a more chaotic, but ultimately more creative, strategy of “mutual reinforcement”…

Accelerated Evolution

More and more in the early part of this 21st century we are being made to realize the creative power of complexity dynamics and its potential for system self-integration and emergence.  In some arenas such as a multicultural society, the economy, technology, the arts, and even our daily lives, self-integration and emergence is a source of great diversity and creativity; but in other areas such as financial markets, terror networks, and the global climate it can be a source of great instability and destruction.

Over the last 400 years cause and effect has told us a lot about the dynamics of simple systems void of feedback, but the dynamics of complex systems alive with feedback is a subject that is becoming increasingly relevant and important to understand.

Essentially natural complexity is, in fact, “self-integrated information”.  All natural complex systems are characterized by the fact that they have “low thermodynamic entropy, but high information entropy”; consequently these systems can be considered to be highly organized but unpredictable nonetheless.

So while most people might think about evolution in terms of “an ecosystem of plants and animals”, the reality is that plants and animal (and the ecosystem itself) are really just “information structures”, and evolution simply keep creating evermore complex information structures (structures that are ever more difficult to mathematically compress).

Evolution is fundamentally a “universal process of change”.  Evolution is not just about biology, but about all “information creation”.  Evolution created us, and we in turn create complex information structures.  In fact it could be reasonably argued that in our ever-more rapidly interconnecting world, we are likely fast approaching a phase transition in human development, a transition to a whole new age ; “An Age of Accelerated Evolution and Information Creation”.  And this coming age will be dominated not by the old linear paradigm of predictable cause and effect, but by a whole new Nonlinear Paradigm of unpredictable “Integration and Emergence”…


Algorithmic Search

The new physics of the 21st century and beyond, will be the physics of self-integrating systems and accelerated evolution.  Understanding this “new physics of evolution” will be essential if we want some control over our ever-increasing inter-connected, co-dependent world.  Artificial Intelligence (AI) is seen by many as a means of dealing with complexity and already AI is being used to get computers to learn, but ultimately this will turn out to be really rather small potatoes; the really big pay-off will come from getting computers to explore.

After studying chaos and complexity for so many years, it strikes me that the universe is not fundamentally (as it so often suggested) purely mathematical, but is more generally “algorithmic”.  The process of evolution is, as Darwin himself more or less suggested, a continual process of the emergence of ever greater complexity; a process which would seem to suggests that Mother Nature is, in fact, ceaselessly executing a form of “Algorithmic Search”, constantly seeking out the most successful combinations of integrated diversity.

If this is indeed the case then it begs the question, “Is what Nature finds simply random, or are some things more likely to be found than others?”  Well, as it turns out, chaos suggests the latter…

The discovery of chaos has alerted us to an algorithmic universe that was previously hidden from our awareness; a nonlinear universe of “complex strange attractors”.  The existence of such algorithmic attractors begs yet another question, “are there some, or even many, hidden gems (or dangers for that matter) in this nonlinear universe that we are as yet unaware of?”

In these early days of the 21st century, we are only just beginning to reach the computational power needed to address this question; and although computational exploration of complex system behavior is still an activity very much in its infancy, it is destined to grow to great importance because, for good or bad, it is safe to say that

The 21st century will be a century that embraces the Creative Power of Natural Evolution and Complexity Dynamics…

Matrix of Emergent Dynamics - Copyright - Kieran D. Kelly