Algorithms Cater To The Mean – Not The Individual
Algorithms are my income.
I game algorithms for my clients on a daily basis.
Each individual algorithm, Twitter, Facebook, Google, etc – take your pick. They make it very clear to us the metrics they like to reward, and then equally, those same metrics they enjoy punishing. But there is an inherent paradox of big algorithms…
The dirty little secret which for so long has been plain as day for anyone who bothered to look is that there is really only one metric that matters. And this is equally true across all platforms.
Ad engagement. Ad engagement is revenue and if you are a company, revenue is life. And how does one ensure maximum ad engagement? Well, that’s an important question to figure out. Watch time. The goal of each and every algorithm is to maximise watch time to the highest possible metric. Per-user, per day.
Charlie Munger famously said, “show me the incentive, and I will show you the outcome”. Well, here it is folks. Understand the algorithm incentive, and the outcome will be revealed.
It’s All Incentives… Always Has Been.
If watch time is the single driver of algorithm behaviour then what does that mean for me, or perhaps you? The content creator who hasn’t ‘niched down’ to a single theme in order for Google to ‘understand’ them better?
It’s not good news.
As I will explain in the ‘Paradox Of Big Algorithms‘, the problem is the algorithms aren’t designed for broad interests and expectations. They cater to the average, the mean, and they do so very effectively with few exceptions. Also, just to be clear I don’t mean average in quality, I mean average in predictability, average in understandability, average in interests. Typically the snow-balling feedback loop is crowding out the rest.
But why?
It’s all incentives. For an algorithm to understand you, you must be predictable. My most previous post was ‘A Manifesto On Food Waste‘, then a summation of Mongolia, and now I am writing about this, an article about algorithms, just last week it was an article about ‘Geothermal Energy‘. It’s not very predictable. What is Google to make of this?
I don’t blame Google, that job is bloody hard. But since the algorithms determine our distribution, almost exclusivly, then they do so as well, determine our fate.
Getting past the trundling snow ball? Writing online, making youtube videos, getting noticed on Twitter, these are all distributions of Extremistan, but more on that later.
The Paradox Of Big Algorithms
How often does the algorithm serve up exactly what you are looking for?
The Paradox Of Big Algorithms manages to show you simultaneously what you want without actually showing what you want. Humans are full of flaws and biases. Algorithms can easily exploit this. Sex sells. Controversy sells. Secrets sell. Some themes just end up playing over and over again.
Does your youtube home page reflect your interests or does it your procrastinations? Does your Twitter timeline show you the same people’s behaviour again and again? How often do you discover exciting, thought-provoking content online that fulfils the precise purpose of your search?
How often do you discover something that was beforehand, obscure and unknown?
Because please don’t get me wrong. The precise content of your search does exist.
If you could peel back the cover of the internet, scream into its unorganised abyss and then draw to the surface exactly what you are looking for, then you would find there is a piece of content on the internet that exactly meets the purposes of your search. But alas, one cannot simply ask Siri and have returned a satisfying response.
What separates you from exactly what you are looking for – is the algorithm.
The Paradox Explained
The Paradox Of Big Algorithms manages to somehow show you the mean of what you want without actually showing you exactly what you want. Incentives. Each algorithm is simply the accumulation of your interactions with certain nodes. Some are more sophisticated than others, but at the end of the day, it is the combination of your interactions scored against the macro interactions of everyone else operating in that same algorithm. What does that get you? It gets you a slightly biased (in your favour) version of the mean that pixelates on your screen.
Let me try and explain with an example.
The exact video you were looking for with your search has only 439 views with about 42 hours of total watch time so far. The average video you are ultimately shown meets the intersection of what you were looking for against what is most popular. In this case, a video that has 80,000 views with 10,000 hours of watch-time. More people watch the second, the algorithm favours watch time above all else.
The 80,000 view video is more accessible (more average), perhaps even one reason why it accumulated more views than the other. The average is always a better reflection of the masses. And viola! The paradox of big algorithms. You were shown what you wanted, but not exactly what you were looking for. Your attention is the north star – it’s too risky to show you the video with barely any views, your attention will more reliably be held with the validated average and your attention is the only thing that matters.
One size doesn’t fit all
There are exceptions! Obviously. But believe me, as someone who is poking around the backend of the internet every day – don’t let your exceptional experience prove the rule. There are billions if not trillions of searches made on a daily basis across all algorithms. How many of these are yielding exceptional results?
The key to this is that there does exist the exact piece of content you were looking for! But because of the mean, the easiest content to digest, the most accessible content, the content with the most watch time. This is the result which will be shown to you.
Google’s Infamous Sandbox
There is the concept within SEO which is referred to as Google’s sandbox.
When a new domain is bought and content is produced, Google takes their sweet time to index that webpage. They let the creator play around in the sandbox for a while. Google wants to see how serious this new website is going to be. Is this a post once and dump kind of deal? Or is this new website going to be maintained and updated on a regular basis?
This again goes to serve the Paradox Of Big Algorithms. Consistency is rewarded over quality. Quantity is rewarded over quality. Creating consistent quality is rare. Quantity is rewarded, and as such the content creator makes concessions to quality for the purposes of ‘consistent posting’ or the cliche, and often misunderstood, ‘SEO optimised content’.
As long as the algorithm cannot measure quality, they will take the more averaged metrics of quantity and interaction. It is a snowball effect that starts with the algorithm.
The algorithms need to be better! One hopes they will become better. But one also acknowledges the enormously difficult task of making that happen.
It’s Survivorship Bias All The Way Down
These algorithms incentivise mediocrity. They hijack the potential of creative workers and ask them to serve up more ‘digestible content. Reward them for meeting their standards. And so we are duped into thinking these are the right standards for content!
It is a big case of survivorship bias.
Youtubers who dish out advice to other people prospecting the brilliance of youtube fame will tell you quite systematically. Videos require a hook, a promise for secret content at the end, they must be fast-paced, aesthetic, plus your thumbnail must ‘get attention’. There is a big Youtuber out there who sells a course for hundreds of pounds promising, “all you need to do is put out content consistently for two years”… and of course follow all other indicators of success mentioned above. Then you will be youtube famous and successful just like me!
What we are missing is that it is all survivorship bias. And what makes it even worse, is it is self-serving. The algorithms are very sophisticated. There is no escaping this fact or the fact that they are only compounding on their sophistication. Each additional node I touch further informs the algorithm of my behaviour, then my demographic’s behaviour, then my industry’s behaviour and ultimately the algorithm’s behaviour as a whole. This iterated billions of times a day makes for an amazing piece of technology. But it also makes the world’s biggest feedback loop.
Who is the winner?
The ‘winners’ of content creation. You all know them. Whoever has the most subscribers or followers in your particular niche. They copied the winners before them, and then they encourage you to copy the process they underwent as well. Except rather through old school emulation it is now rather encouraged through monetised course work, but that’s a grievance that would derail this train of thought.
Algorithms prefer popularity over quality. Is it the case that the most popular is also of the highest quality? Any fool who’s played around in a social hierarchy knows this is total BS.
Algorithmic Outcomes = (low hanging fruit survivorship bias induced behaviour) x (billions of iterations) + (sprinkled in throughout, the exceptional outlier)
I Promised You Extremistan, But First A Conclusion.
Let’s sit on that point for a moment. Occasional and exceptional outlier.
Naval, Elon Musk, Nassim Taleb, Christopher Hitchens… etc. We know these men to be a complexity of combinations that are both high quality and also people who do not meet the algorithmic standards for ‘best practices’. They are the outliers for whom a slipstream in the algorithm allowed the world to witness their brilliance once and decided it wanted to be the only thing they saw.
Then there are men like David Perrell, or Tim Ferriss, or Shane Perish. Men who are equally brilliant and qualitative as those listed above, but crucially different because they mastered the ‘best practices’ of the algorithms. And so perpetuate themselves right to the tippy top.
But what about the myriad of people who are just as magnificent as the guys we just spoke about. Angus Fletcher, Ludvig Sunstrom, Tim Marshall, Colin Wright… this list could go on forever.
These are people whose quality of content far exceeds most of the tripe that dominates my Youtube and Twitter homepage. But they don’t adhere to the algorithm’s wishes. The algorithm wants tightly defined, understandable people who make predictable content. Pippa Malmgren is as wide an erudite as this planet has, do you think the algorithm can understand her? Only a human, like me or you, have the complexity to read the various signals that drive us towards certain people and ideas. The algorithms cannot add and subtract variables to themselves as quickly as we can.
Average Begets Average – the Paradox of Big Algorithms is the Average
And now this is the culmination of the article.
With all that preamble and context. The algorithms show us the average of what we want but not exactly what we want. If I want to hear about being a digital nomad there are fifty bums Google might throw at me before Colin Wright. Not because Colin Wright has bad content, but because Colin Wright has content about everything, and despite the fact he is one of the OG digital nomads his website tells a much deeper story. And the algorithm can’t handle that because the average can also not handle that. The algorithm serves a hyper niched, widely accessible audience. Survivorship bias then dictates what behaviour you need to emulate to become a winner.
What we see is the delicate balance between what our personal algorithm wants to show us combined with what the public algorithm wants to show us. There is a constant battle between which of these large variables is going to win out. Algorithms are defined by the low hanging fruit survivorship bias that makes up good, reliable watch time. Some sexy websites might keep you around a bit longer than some dull page of text. Even though that dull page of text had exactly what you are looking for, the sexy one kept you around distracted longer.
Maybe you even clicked on an ad.
Extremistan Explained
This is now just a touch of extra content for those keen to learn about one of the most mind-shattering insights into how things get distributed.
It is birthed from the ‘hard Twitter ego’ personified genius – Nassim Taleb. I have written extensively about this guy’s work, and myself, even host the internet’s biggest podcast on risk, randomness and uncertainty titled The Nassim Taleb & Incerto Podcast.
I will give you the skinny on Extremistan now – but for more to check out those links above. Once more, for writing on Nassim Taleb & for podcasts on Nassim Taleb.
Nassim speaks about two fictional countries and their differences concerning each owns predictive abilities. Mediocristan & Extremistan
What is Mediocristan?
In Mediocristan, the land of the average, observing the first 100 outcomes will provide analysts a good indication of the next 10,000. In Extremistan, quite the opposite occurs. Observing the first 100 outcomes provides no insight into predicting the next 10, let alone the next 10,000.
Think of how domain specific outliers influence different distributions.
If you average the weight of 100 random people, there is no alive person who can be added to the sample and weigh enough to significantly alter the average. If the average weight of 100 people is 75KG, an outlier weighing 200KG is barely going to move the needle. The same is true for calorie consumption. There is no day where you can eat enough calories to make you fat, nor is there one day where you can eat so insufficient an amount of calories to make you skinny. These are domains of Mediocristan.
Now imagine we take a sample of 100 people and calculate their average net worth. There are outliers alive (Jeff Bezos, Cristiano Ronaldo even Joe Rogan) who can totally distort your average making everyone else’s combined net worth look like a rounding error for our outlier.
There are domains subject to an outlier so extreme that it can completely distort the game for the rest. These domains are wealth, income, subscribers, deaths, size of planets, viruses, etc. Clicked links on the internet. Page 1 search results on Google.
‘You Can’t Lose A Ton Of Weight In One Day, But You Can Lose A Ton Of Money’
The problem of the Black Swan is that we think we live in Mediocristan and are able to predict the future based on the past. We can safely predict that the next person added to our average weight cannot weigh more than 500KG. However, we live in Extremistan and are exposed to its domains. All of the world’s USD can be wiped clean tomorrow should a Black Swan event so significant occur.
We live in Extremistan. The problem is… we think we live in Mediocristan.
Subscribe To My Newsletter – 🗒️ A Curious Worldview!
Cheers for reading through to the end you legend!
Every month (or couple of weeks) I write a short, popular email.
In it you will have your worldview expanded upon. It will either challenge your beliefs or open your eyes to something new. Never boring. Sometimes in will include things you don’t want to see. Sometimes it will be exactly what you were looking for.
It is free and the best thing you can do for me. If you are keen, chuck your email in below and then CHECK YOUR SPAM!
MAKE SURE YOU CHECK IF THE EMAIL GOES TO THE WRONG EMAIL FOLDER – PLEASE DRAG EMAIL INTO PRIMARY FOLDER
