Herding Cats

Shouldn't Complexity Theory and Emergent Systems Start with the Mathematics?

The answer of course is a resounding yes. If we can calculate something we don't have to rely on hand waving, references to populist articles that do their own hand waving, or at least references to dead people who did the hand waving before.

The very notion that a stranger is going to spend good hard earned money to make changes to something that actually belongs to someone else - possible share holders - in the absence of some sort of verification that that hard earned money is going to get returned some time in the future - is well - out of their freck'in mind.

This approach to social systems is based on the belief there are enough people out there who don't have the accountability to the share holders, a contract officer, a government procurement officer, or someone themselves accountable to look after the money, is common. It's common because those getting really excited about changing the world aren't always the ones accountable to those pesky shareholders and contract officers.

That's not to say the world doesn't need changing, because it does. The world needs people who expect it to change from their own ideas. The world needs people who come up with completely new ways to make things happen. But the world is also a cruel task master when it comes to funding these ideas. Ideas that a cheap when they don't involve spending money.

Like getting together at Starbucks to scheme on how to do things better. Like let's fix our status meeting process. That doesn't take much money, just effort. Or let's restructure our organization and not focus on earnings per share instead focus on having harmony among the product development teams first so we can improve EPS sometime in the future. Ah, you'd better have the CFO, CEO and Chairman of the Board, and the Investor Relations VP at Starbucks before you jump off that cliff.

Without some notion of the credibility of the idea of how to change anything, it simply doesn't happen. This of course doesn't stop people from trying, never has, never will. But being cranky and disappointed should not come as a surprise - change is hard.

There is a critical difference between something being possible and something being possible. It may be possible to change to world but is it probable in the domain and context you work. If not probable, it's best to look for work somewhere else.

Another View of Agile

Power to the Edge: Command and Control in the Information Age, speaks to the integration of traditional management with complex systems in the presence of agility, emergence, and variability of the real world.

This is a book about managing in the defense systems domain and the technology advances that push the decision making processes to the edge of system.

This is about architecture of the system. The human system, the techncial system, the operational system, and the governance system, and the environment in which these systems are embedded.

While the book is focused on military systems, it is applicable to other complex systems. Military systems have many if not most of the attributes of business systems.

They have strategy, emerging requirements, adversaries who want to put you at a disadvantage, limited budgets, limited time to deploy the solution, conflicts internally and externally that conspire to have your efforts fail.

The notion of centralized planning and decentralized execution has merit in modern software development processes and the business they support. Planning is NOT used in the way agile software developers use the word Planning - wrongly BTW. Planning in the defense domain, for defense processes, and the programs that deliver the systems used in those processes means Strategy Making.

All Planning is Strategy Making. So when you here an agilest say Responding the Change Over Following The Plan, they should really be saying following the schedule. The irony of course is even the formal DoD Earned Value Management world, there is a nice little form titled BCR (Budget Change Request) and its used all the time to make changes to the cost and scehdule baseline. Whym beacuse things change. But if we're changing our Strategy all the time, then there's a much bigger problem to solve.

Calculating Outcomes

To follow on a recent theme of what's wrong with populist book? is shown in this example. Integrating Air, Land, and Sea Operations, which speaks to integration of complex systems into a System of Systems that behaves in ways not definable in the beginning. This is done through intelligent agents at the nodes of the network. This is called network centric warfare.

The Integrated Master Plan (IMP) is a Strategy for the successful completion of the Program. Like all strategies, a set of hypothesis are needed to test the strategy. With these hypothesis testing processes, there is no way to known if the strategy - the Plan - will work.

This is a sample of a book where popular ideas are presented and at the same time actionable outcomes are provided.

Tell Me It Isn't True

When there are statements like this, in the absence of a domain and a context of a domain, it just reinforces why I don't recommend you read any populist books on a topic if you have any skills at all of getting to the first level of actual technical context for that topic.

Taken from a recent blog post on "complexity thinking" with 4 populist management book references

 

So let's start here with a survey of the topic from the non-populist point of view - that is actionable. That is from the point of view of people who want to calculate something, make a decision on the outcomes from that analysis.

 

Nothing wrong what so ever with those populist approaches. But one test that has served me well over the decades is can I calculate something with this person's product (book, paper, or article)?

No? Then I may not be able to calculate something directly from the author's work, but I'd better be able to find out how to do it in an appendix or a reference.

Still a No? Then is there something there that will lead me to the next level of understanding where I can start to see how I can calculate something. Here's the recent example about Moving Beyond Populist Books.

What's this means is that while the populist books may be informative to many - and we need a lot more informed people than we have now - these populist books and materials quickly run out of steam when we want to actually make a decision that involves money, time, resources, risk, commitment of someone else's stuff. In other words we need to know quantitatively and analytically what the probability of success will be if we follow that authors advise.These populist books play a role of starting the conversation, but fail to complete the conversation - in many cases - with actionable outcomes. They point out the problem but not the actions need to create the solution. Knowing the problem is necessary but not sufficient.

I can hear the populist authors squawking now.

What! How dare you critize my popularizaiton of this critically important topic with your lame need to actually apply this idea in a quantifiable manner. 

But their role is to popularize a topic, not necessary do the calculations needed to solve problems with the popularization.

So if you look at the reading list of the paper above, you'll see books, papers, articles that DO provide actionable outcomes. Once you've digested the populist ideas, take a tour of those to see if there is anything there you can put to work on your own complexity management problem.

Don't get me wrong. Populist books are fine. But they are just that populist not the engineering books I need to work the problems in front of me.

For some more background on complex systems in the world I work see:

• NDIA Human Systems
• Verification and Validation in Complex Adaptive Systems
• Cybe Space: The Ultimate Adaptive System and the Operationalizing Agility 
• The Agility Advantage: A Survivall Guide for Complex Enterprises and Endeavors. This eBook will freak out the agilest who don't understand the notion of Command and Control is mission critical paradigms, where agility must operate in the presence of 100% Mission Success.

What Does Complexity Acheive?

With all the wrangling around simple systems, avoiding complexity, emergent order out of complexity, we need to ask what does complexity achieve.

In a paper titled, Complexity and Robustness J. M. Carlson, Department of Physics, University of California, Santa Barbara, CA 93106 and John Doyley, Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125 , they tell us ...

 

 

 

 

 

Complex Adaptive Systems

When I read something like..

Self-organization is the process where a structure or pattern appears in a system without a central authority or external element imposing it through planning.

I get a smile on my face. The populist (like that above) definition of a CAS is clearly designed to inform the general public and establish a basis of discussion around the notion that organizations and the people who populate them are subject to forces outside of their control and management must act accordingly.

Good advice. But it's one of those good pieces of advice based on a naïve understanding of the underlying principles. In this case the Lagrangian of the individual elements that interact with each through a combined set of ground rules. In the mechanical systems domain, gravitational, electromagnetic, the strong, and weak forces - your choice.

But First Some Background

A vision shared by most researchers in complex systems is that certain intrinsic, perhaps even universal features capture fundamental aspects of complexity in a manner which transcends specific domains. It is in identifying these features that sharp differences arise. In disciplines such as biology, engineering, sociology, economics, and ecology, individual complex systems are necessarily the objects of study, but there often appears to be little common ground between their models, abstractions, and methods.

From Complexity and Robustness, J. M. Carlson, Department of Physics, University of California, Santa Barbara, CA 93106 and John Doyley, Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125.

It's for this very reason that statements found in the first quote - in the absence of a domain and context - have little use outside of popularizing the notion of Complex Adaptive Systems. They best serve those seeking words without the underlying tools for analysis.

It is this little common ground where the popularizations have difficulty. They want to state ideas without out all the wrangling about the details, the semantics, the gory mathematical complexities. They are popular and therefore readable in the absence of the details.

Failing to acknowledge that popularizations are just that, popular for a reason, creates issues when it comes to making calculations or decisions using other peoples money.

Back to the Populist Approach

Who cares if the statement above is based on naive principles, if it works? Well clearly those reading and writing general management books are not concerned with the underlying details. Their job is to inform the populace not provide tools to calculate solutions.I'm reminded of the advice to Stephen Hawking when he was writing A Brief History in Time. He was told sales of the book go down by ½ for every equation he used. Compared to The Large Scale Structure of Space-Time, (a graduate text I still have on my shelf), where every page  of the 381 pages is as dense like the clip to the left of this paragraph. (Click for full size)

Motivation for the Underlying Principles

Let's look beyond the simple definition for some understanding of CAS. But first let's look for some motivation...

As a physician, I learned to think from a biological perspective. When I went into management, traditional organizational theory seemed artificial, foreign to my experience. So when I started studying complexity, I was stunned. Here was a way of thinking about organizations that compared them to living things. That makes sense to me, intuitively." Richard Weinberg, MD, Vice President, Network Development, Atlantic Health System, Passaic, New Jersey

This intuitive understanding is critical for many reasons, not the least of which is to convey a broader understanding of the problem space to the general public to gain their interest in further discussion.

So here's a succinct definition from the Santa Fe Institute focused on the social aspects of CAS.

Definition: A Complex Adaptive System (CAS) is a system of individual agents, who have the freedom to act in ways that are not always totally predictable, and whose actions are interconnected such that one agent's actions changes the context for other agents. Examples of complex adaptive systems include: the stock market, a colony of termites, the human body immune system; and just about any collection of humans such as an industry, a business organization, a department within an organization, a team, a church group, a family, or the Rotary Club.

In this definition of CAS these agents act according to their own rules. These rules are defined for specific classes of agents in the system. Either one rule for all or a single rule for each of agents, or any combination.

The behavior of the resulting system emerges from the interaction among the agents executing their rules of interaction. In this case the Central Authority is the collection (one or many) Ground Rules for how these interactions take place. This is subtle but critical to the next step, when CAS is applied to business.

There are always Ground Rules. To not have ground rules defies the laws of physics. But these ground rules may be flawed. When they are, the outside observer would say the system is broken. That is the system behaves in undesirable ways. But those ground rules that central authority is still in place. It is always in place.

In order to return the system to its desired behavior, those ground rules of interaction must change.

The CAS exhibits novel behaviors resulting from the interactions. The interactions defined by the ground rules covering all objects in the system, or "N" Ground Rules one for each element, result in fundamentally unpredictable behavior. It is not a question of better understandings of the agents, better models, or faster computing; as we have come to believe erroneously, based on the machine metaphor. We simply cannot reliably predict the detailed behavior of a CAS through analysis.

But this does not mean there is no control system in place. This control system is distributed among the elements of the system. When the phrase...

where a structure or pattern appears in a system without a central authority

...is used it begs the question - what is the definition of Central? Most populist approaches fail to make this subtle and important distinction. This is why they are populist. To make these distinctions requires deep analysis and the introduction of more complex mathematics. The analogy used in the original quote suffices for the discussion at the populist level. It works, so let's move on.

But of course, we'll want to calculate something one of these days. Say the forecast close of the Stock Market tomorrow, given any non-disruptive events. While we cannot predict the precise closing value of the Dow Jones Industrial Average tomorrow, we can estimate (with a confidence interval) the market trend as bullish or bearish and take appropriate action on this estimate with a known confidence.

Note the qualifier of any non-disruptive event. This approach provides some confidence in understanding human based systems. But care is needed to not over-estimate the ability to predict what will happen outside the error bounds of our forecast.

It is popular to say ...

In a CAS, control is dispersed throughout the interactions among agents; a central controller is not needed.

Notice the difference between control and controller. The controller is distributed among the agents. But there is control, just not in a single entity.

In the end the business writers are right, but for the wrong reason.

Trying to apply external control, or a single internal controller leads to disappointment. But there is control and that control must be in place for any system to work and not turn into a dispersed set of agents flying off into the universe. This is the definition of Chaos.

The Take Away

The ground rules of the independent controllers that make up the control algorithm of the system must act in ways that produce beneficial outcomes. It is the correction of the actions of the individual (or collective) controllers that make up the control algorithm that must be the target of management processes.

Not the naive belief that there is no control needed and the control will emerge from the group of agents. There is always control at some level, be it microscopic or macroscopic. But it is always there. If it is microscopic we may not be able to see it or have any influence on its outcomes. If it is at too high a macroscopic level we are likely not going the like the outcome either.

In the end it is the semantics that are important, especially when we want to calculate something - say the probability that it will rain tomorrow on the Front Range of the Rocky Mountains. Since the weather on the Front Range is a CAS - all weather is a CAS - doesn't not remove our ability to make forecasts of it's behavior in the very short term and the long term - Global Circulation Model scales. And especially when we want to make MANAGEMENT decisions in the presence of CAS, these populist descriptions fail to provide much guidance.

We need better analogies

Jurgen Appelo has a nice post about the efforts to speed up the change in the management processes. While I don't share their goal for several reasons, it's still a good goal in principle.

My experience is that management processes in mature domains are well suited to service the needs of the domain. In inmature domain, management processes are good targets for improvement.

In building launch vehicles, manned space craft, weapons systems, commercial aircraft, the management processes are tailored to the need. They are disciplined, rigorous, methodical, and may appear to outsiders to be slow and plodding. These domains have heavy software requirements and many of the agile principles are in place, along with the management processes to get maximum benefit for them in the context of formal contracting.

In this domain, we get in trouble when we DON'T flow the processes and try to skip steps, replace known working methods with new and untried ones.

But there are many domains where management needs to be overhauled, and Jurgen and company are on a quest to do just that.

The Analogy Trap

Jurgen mentions the "attractor" as the analogy of the management processes that need to be changed.

That management is Attracted to a set of processes and those processes need to be changed. The picture above is labeled an attractor but it is in fact a picture of the path followed by the underlying equations of motion of a dynamic system. It is the artifact of this dynamic system.

Why is this an issue? Well it's simple. Jurgen mentions changing the attractor - replacing it with a new attractor.

It is NOT the attractor that needs to be changed, but the underlying equations that create the attractor that must be changed to produce a new attractor. These equations represent the dynamics of the system. When the analogy is flawed, we loose the ability to actually drive the change. We give away the connection between cause and effect. And then we're disappointed when something doesn't happen.

The garden as a self organizing system is one such analogy that produces less than expected results of good tomatoes, when we let them do their own thing.

In systems (complex or not) the equations of motion that produce a set of paths when captured over time produce pictures like the one above. These equations are deterministic, they are complex, they can emergent, and they are almost always represented in some space other than our familiar 3 dimensional world of X,Y,Z coordinates. The sample picture is in phase space not Euclidean space.

A simple example is the spherical coordinates of our world - the surface of the earth. If we were to start walking in a direction and hold our heading - or flying would be better - we'd end up in the same place we started. A spherical coordinate system returns us to the same place in the coordinate system after a 360° transformation - a 2π rotation.

The picture above is in a Phase Space, rather than a geometric coordinate system. The generation of the path in phase space is a common method in physics for showing complex relationships between the processes - a nice approach for management processes.

But the attractor does not attract in the sense of a force, pulling on an object - in the classical physics sense.The statement in the Wikipedia link is technically not correct. ... points that get close enough to the attractor remain close even if slightly disturbed. Those points follow their equations of motion they don't know that they are creating a pretty picture of the attractor, they are just following their deterministic path, guided by their equations of motion.

This is the key concept that should invert the analogy of Jurgen for the betterment of his quest. There is not attractor there are equations of management that must be altered.

The attractor is the path followed by the particle (in Jurgen's case, managers and management processes) while traveling inside the system (the management system). The picture is a plot of the path of the object moving, guided by the equations of motion. In the field I grew up in, the Path Integral describes how particle move between two locations guided by the forces that act on them.

So Back To The Analogy

For an analogy to work - or at least work well enough to be useful - it needs to inform a new understanding from previous understandings. But it has to be correct in the original space. The suggestion that a new attractor should be sought needs to be replaced with the search for a new set of equations of motion to generate the attractor - the picture above.

I got a book for Christmas, Feynman, Ottavini & Myrick. It's a graphic book of the life and accomplishments of Richard Feyman. Feyman was the father of a particle physics theory that won him the Nobel prize. He has great advice (starting on page 157) about when speaking about sticking to the real science when trying to explain something in populist terms. Once you bend the real science in an attempt to explain it in populist terms, the explanation looses its integrity and the resulting understanding looses its integrity as well.

My favorite no integrity populist book is the Tao of Physics, which was a wildly popular in the 70's trying to explain the parallels of particle physics and Eastern Mysticism and was based on completely crap (I was a particle physics grad student at the time).

Why the quest for integrity in the analogy? Once the underlying integrity is lost, the resulting benefit of he analogy is lost as well. That's the point.

Jurgen's analogy between management processes and attractors is useful - and has integrity - for management transformation. But it needs to be inverted to be useful. The underlying equations of motion are (or may be) flawed in the domain Jurgen seeks to improve. The attractor is just the  picture of the path taken by those equations over time. Change the equations - get a picture of the new attractor. Changing the attractor requires changing the equations. Not the other way around.

Seek new equations and the picture will change.

If the picture becomes more attractive in the populist sense, then mission accomplished.

Chaos, Complexity, and Entropy

When we hear about complex systems, chaos and how those systems are models of the domains we work in - software development, people based systems, systems of systems - the paper above may provide somoe insight on the limits of extending these principles into the popularizations of these principles.

• Complex system contain elements that interact nonlinearly
• These elements are interdependent
• The structure of these system spans several scales
• Such systems are capable of emergent behavior
• Complexity involves the interplay between chaos and non-chaos
• These systems involve the interplay between cooperation and competition

This is all restating the obvious, but care is needed in drawing broad conclusions.

Knowing the underlying structure of the system is the starting point. Does the system under study have linear and non-linear interdependencies? Does the system exhibit emergent behavior BECAUSE of these interdependencies or is there some other processes?

Questions like this needs some assessment before broad statements like ...

What we call “systems” are nothing more than figments of our imagination… abstractions in our minds.

The system of river fish in the Gunnison Valley is not a figment of our imagine. It is a system even when our college son is not thereto study it when he is home for Christmas. It's still a system, interacting in a non-linear manner - of water flow from Taylor Reservoir and the November snow fall and melt), through interdependent elements - of rock formations, tree roots, river bottom structure, and the Colorado Department of Wildlife hatcheries, on several scales - from the feeding flies for the trout, to the bears, to the fisherman, exhibited in an emergent behavior, with the complex interplay of chaos and non-chaos, and most of all with competition and cooperation - between the natural habitat to the man made attempts to preserve the river.