What can help to overcome complexity of the world? For quite a few weeks I couldn’t get past the thought that many products and services that we use are, well, simple. The core functionality of food delivery/taxi hailing/cleaning apps is as simple as a stick:
- Client orders something from a limited, one- or two-dimensional list
- Business links the client’s order to the actual provider
- Business keeps track of the transaction to ensure it’s executed properly
- Business transfers the transaction payment less their commission
Ridesharing/e-scooter sharing apps aren’t more complex than that. Conceptually most products we use daily are indeed pretty simple.
Google search engine, network protocols of popular online games, cloud services, messenger infrastructure, as well as infrastructures of Facebook, Zoom, or Netflix — that is where the level of technology complexity gets hardcore.
Google/Facebook/Zoom/Netflix infrastructure and core tech teams are lucky as these systems work with a strictly deterministic environment of bits; one can somewhat control the complexity and chaos.
Things get way more difficult in the physical world: in Uber infrastructure (I’ll take Uber as an example here) a driver is a few dozen kilobytes of data [including photo].
Meanwhile in the real world an Uber driver could:
- have questionable personal hygiene
- be a charming psychopath
- be severely sleep-deprived
- be angry at his wife
- go riot at the local police station
- a hundred of other plausible scenarios
All these account for megabytes of unpredictable corner cases.
And we’ve just looked into ONE dimension of the multidimensional Uber environment. There are also client segments, competitors, trends, new tech, road traffic (affected by roadwork, parades, protests, festivities, accidents), trade unions…
The external complexity of the world where our products operate is immense, and the Uber team has to process it.
This being said, for a scale-up a business has to employ more and more staff to keep processing the increasing external complexity (e.g. if we launch in France, we hire 100 more people):
- System gets more complex due to the large number of employee interactions within a team. 1 additional member in a small team doubles the number of interactions. The complexity of communication in a team is 2^n, where n equals the number of team members
- Increased number of interactions between people and internal systems
- New country means:
- new billing plans
- new ways of solving corner cases
- new interactions with external systems: tax authorities, courts, trade unions etc
- new goals and priorities of some of the top managers
- …or actually new top managers
All this leads that exponential growth of business complexity grows faster than required 2-3YoY [exponential, but slower] growth that investors demand.
This causes some unpleasant side effects:
- the speed of integration of new functionality into existing products, as well as new product launches, goes down. In their first year Slack implemented more functional scenarios than in 2019
- Startups are forced to change their teams three times, as the launching team usually lacks experience in managing complexity
- Errors in product design of the first version lead to full freeze of further development that can last for years
How Amazon manages complexity
I know of three tools that help Amazon grow while the system complexity also grows (I’m sure there are many more).
1. Working Backwards
- all communication must be written
- large-scale products (with goals that are 2 levels above yours) require a formal 6-page document
- document discussion should strictly adhere to the procedure
- = while complexity grows, the speed and quality of decision-making are maintained
2. Leadership Principles
- 70-90% of job interviews are dedicated to candidates’ fit to the Leadership Principles, the rest — to hard skills
- all decisions are made based on the Leadership Principles
- = while teams grow, their levels remain appropriate, as all the team members’ cultural fit is verified pre-emptively
3. Strict adherence to internal interaction rule
- all internal services interact with one another through interfaces — no direct linking, no direct reads of another team’s data store
- all service interfaces must be designed from the ground up to be externalisable, no exceptions
Let’s look closer at the internal services interaction for example. This format postulates: we’d rather make something less functional but more stable — something that works internally according to the strict rules of this system. And we can also sell it to others.
This means that at least for 6 months the system complexity is lower than the complexity of a system without such design requirements.
This approach allows to manage the complexity of the whole information infrastructure; this particular decision formed the basis of the most profitable subsidiary — Amazon Web Services.
Machine learning will one day disrupt organisational complexity
Managing organisational complexity (many people, many environments etc.) is an extremely rare and arduous skill.
It’s laborious to support complex systems in working order over a long period of time. It’s even harder to increase the complexity while preventing the whole system’s collapse.
The Systems Bible [book on fatal way of looking on complex systems] states that complex systems are beyond human comprehension. I once heard an urban myth (although Google search does not verify it) that the operating system OS/2 architect only managed to complete it after a three-day LSD trip. I don’t know whether it is true but it’s very believable — the capacity of a human brain is insufficient for current systems which include a company, a product, interaction with clients, and interaction with various external systems.Machine learning algorithms are already helping businesses to optimise some elements of the system:
- find optimal route for Google Maps
- find a webpage matching the text of user’s search inquiry and their search history for Google
- optimise purchasing and logistics for grocery chains
- forecast demand by city district for Uber and
So the current level of Narrow AI (narrowly specialised AI) already allows companies to operate amidst growing complexity.
My assumption is that organisational complexity can be managed with the help of AI that has virtually no analytical limits on the level of complexity, AND it would have to have an interface for working with that organisation.
So as soon as machine learning algorithms can operate tasks such as:
- forecast competitors’ behaviour, traffic (or interact with a specialised neural network that works with traffic already); forecast crime rate… and other complexity aspects of Uber
- set targets for Uber employees
- control and analyse the target achievement
…then Uber would be able to grow without having to hire more people (who are hard to manage and unpredictable meat bags susceptible to coronaviruses).
Apparently economic growth, business growth, emergence of multiple products and multiple various big complex systems will not come from human skills and habits in creating such systems, but rather from the ability to design and program an AI that would solve the issue of exponential complexity increase.
I presume that the next Microsoft, Google, Facebook and Amazon will not be created by humans — but by the human+AI team or, perchance, by an AI alone.
Technologisation of presence in the physical world.
At the same time, a MASSIVE part of our life still hasn’t even tasted high tech:
- Most people commute the old way — by public transport and private vehicles. This technology stack is 120 years old.
- Most people cook their own food, buying groceries in chain supermarkets. The cooking technology stack is 10,000+ years old. The supermarket stock optimisation tech is 70+ years old.
This is due to the fact that the complexity of the physical world is overwhelming, and to disrupt these markets we’d need a lot of cheap (=mass market) robots that we cannot yet produce.
Imagine the neural network that sets targets for Uber employees; what if it figured out that it is now optimal to check that at least 80% Uber vehicles have to smell nice, then programmed and equipped robots to clean the cars’ interiors and sent them to get it done.