Believe what people 'do', not what they say!
The True Human Mind as Captured by Google Trends

“A completely new way to study human thought!”
_Steven Pinker, author of The Better Angels of Our Nature

“Everything becomes data.
Especially lies.”
A very unexpected way to read the hidden world

While working on his PhD in economics at Harvard University, Seth Stephens-Davidowitz studied Google Trends, which shows trends in specific search terms.
It was a time when Barack Obama was elected as the President of the United States and many experts were saying, “Racism is gone now.”
But the reality captured by the data was completely different.
On the day Obama was elected president, searches for "nigger president" outnumbered those for "first black president" in some states, and white supremacist sites saw a tenfold increase in searches and signups.
His "covert racism" map, created using Google data, provided a critical reassessment of Obama's primary results in late 2008 and was instrumental in explaining Trump's political success in the 2016 US presidential election.
Why did so many Trump supporters remain hidden until after the election? Why is it so difficult to know who voters will "actually" vote for? The reason is simple: people lie.

People's search for information itself is information.
When and where they search for facts, quotes, jokes, places, people, things, and help tells us much more about what they really think, what they desire, what they fear, and what they do than any vague guesses.
Not to mention that sometimes people make confessions rather than questions in Google search boxes.
Things like, 'I hate my boss so much', 'I'm totally drunk', 'My dad hit me'.
The everyday act of typing words or phrases into small, square blank spaces leaves behind tiny traces of truth, and when millions of these traces accumulate, a profound reality is revealed.

---From "Introduction: An Overview of the Big Data Revolution"

The biggest reason Google Search is so valuable isn't because there's so much data, but because people are honest about their thoughts.
People lie to friends, lovers, doctors, surveyors, and even to themselves.
But Google shares information about sexless marriages, mental health issues, anxiety, and anti-black hostility that is hard to find elsewhere.

---From "Introduction: An Overview of the Big Data Revolution"

If we rely solely on what we hear or our personal experiences, it's easy to get the wrong idea about how the world works.
Good data science methodologies are intuitive, but the results are often counterintuitive.
Data science is a process of discovering and understanding patterns based on natural and intuitive human behavior.
Then it reinforces this by showing us that the world works in a completely different way than we thought.
---「Chapter 1.
From "Intuition is imperfect"

Big data allows us to see not just what people say they want or are doing, but what people really want and are really doing.
Providing honest data is the second power of big data.
Because there is so much data now, meaningful information exists about even small groups.
For example, we can compare the number of people who dream of cucumbers with the number of people who dream of tomatoes.
The third power of big data is that it allows us to see even small groups in close-up.
Big data has another powerful force (one that wasn't used in my brief study of Freud, but could be used in future studies).
The point is that control experiments can be conducted quickly.
This allows us to confirm not only correlation but also causality.
This type of validation is widely used by businesses today, but it will soon become a powerful tool for social scientists as well.
The feasibility of causal experiments is the fourth power of big data.

---「Chapter 2.
From "Was Freud Right?"

There was a reason why Seider and his team were so fixated on number 85, and it was a very clear reason.
The left ventricle of number 85 was in the 99.61 percentile.
That wasn't all.
All other major organs, including the heart and spleen, were unusually large.
According to Seider's findings, a larger left ventricle is generally better when racing.
However, if the other organs are all small but the left ventricle is this large, it may be a sign of disease.
The American Pharoah had larger than average size in all major organs and a giant left ventricle.
The data was screaming that number 85 was a one in a hundred thousand, or even one in a million horse.

---「Chapter 3.
From “A New Perspective on Data”

Who on earth creates prejudice against girls? It's their parents.
Parents are often excited by the idea that their child has exceptional talent.
It's not surprising.
In fact, the most common phrase that follows every Google search that begins with "My two-year-old is…" is "talented."
However, these questions are not asked equally to boys and girls.
Parents ask, "Is my son talented?" 2.5 times more often than "Is my daughter talented?"
Similar biases arise when using phrases related to intelligence.
For example, it is embarrassing to ask a question like, "Is my son a genius?"

---「Chapter 4.
From "Digital Confession"

In 2003, statistician Nate Silver developed a new model to predict player performance. Called PECOTA, the model proved remarkably accurate.
Silver found the player's doppelganger.
He built a database of every major leaguer, over 18,000 of them.
It included everything known about a player's performance each year, including height, age, position, home runs, batting average, walks, on-base percentage, and strikeouts.
Now, let's find the twenty baseball players whose performance most closely resembles Ortiz's when he was 24, 25, 26, 27, 28, 29, 30, 31, 32, and 33 years old.
Finding a doppelganger who played similarly to him at that age.
And then we check out what these doppelgangers' baseball careers were like.
Finding your doppelganger is another example of data close-up.
(Omitted) Silver predicted Ortiz's performance based on how these doppelgangers ultimately performed.
Silver discovered that they had regained their powers.
While Simmons may be right about age being a turn-off when it comes to other things, Ortiz's doppelgangers have regained their skills with age.

---「Chapter 5.
From "Close-Up"

A/B testing teaches us to be wary of common lessons.
Clark Benson, CEO of news and entertainment site ranker.com, relies heavily on A/B testing to select headlines and site design.
Benson says.
“You can’t assume anything.
“You have to test literally everything.”
---「Chapter 6.
From "The Whole World is a Laboratory"

How can we overcome the curse of dimensionality? We must be humble about our research and not fall in love with our findings.
Further experiments are needed to confirm the results.
Before you bet your life's savings on the 391st coin, you should see how it plays out over the next few years.
Social scientists call this an "out-of-sample" experiment.
The more variables you try, the more humble you have to be.
The more variables you try, the more difficult it becomes to conduct out-of-sample experiments.
It is also important to record every experiment you try.
Only then can we know exactly how likely it is that we will fall victim to this curse and how much skepticism we should have about the outcome.

---「Chapter 7.
From "Things that can't be done even with big data"

Now change will happen.
Every idea I've discussed in this book is backed up by hundreds of other important ideas that are ready to be challenged.
The research discussed here is just the tip of the iceberg, the tiniest of the tiny specks on the surface.
What will happen now?

---From "Conclusion: How many people will read the book to the end?"