#45. The Importance of Critical Analyses in Examining Social Science Evidence
Comments on Jonathan Haidt’s The Anxious Generation
Dear friends,
Much of my early career as a professor was spent writing a college-level introductory psychology textbook and then revising it for 5 more editions (before turning subsequent editions over to a co-author). (You can see more about that book, including professors’ and students’ comments on it here.) What I learned in writing and rewriting that book is this: To present an honest, accurate view of any issue in psychology I had to read critically the articles I referenced, especially the methodology sections. Does this study really show what the title or abstract claims it shows? I didn’t do that with every article I referenced, but I aimed to do it with each one that was most relevant to whatever idea I was discussing.
This approach dramatically increased the time and effort required to write my 700-page textbook and revise it with new research every 4 or 5 years. But I could not justify doing it any other way. I found that many claims that had become standard to the introductory psychology course had been passed along from textbook to textbook, as authors were essentially cribbing other texts without going back to the original studies and reading them critically.
One dramatic example of this concerns Zimbardo’s Stanford Prison Experiment, which had been described uncritically in every introductory psychology and social psychology textbook I could find. The “experiment” made for a good story, which professors loved to tell and students loved to hear, but when I read the original article I found it wasn’t an experiment at all and in no way showed what Zimbardo or the many textbooks claimed it showed. I’m glad to say that, after my critique, Zimbardo’s sham of an experiment was debunked in several other publications. (For my published critique of the study, see here.)
The point I want to make is this. When you read a book that references research studies supporting the author’s thesis, don’t assume the studies truly show what the author says or implies. This brings me to The Anxious Generation.
Jonathan Haidt’s The Anxious Generation—A Scary Story Not Supported by Science
I take no pleasure in this critique. I have tried to avoid it but no longer can. I know Jonathan Haidt and like him. We are among the cofounders of the nonprofit organization Let Grow. He and I agree that kids need much more opportunity for outdoor free play and adventure than our society presently allows and, through Let Grow, we have worked together toward that end. He and his researcher Zach Rausch even helped me get my Substack started. They are both kind and generous people. I have no question about Jon’s integrity. I am sure he believes the message he is presenting in this book and sees it as promoting valuable social reform. He really thinks that taking smartphones or at least social media away from kids will make them happier and decrease their rates of anxiety, depression and suicide, in part by providing them more time and motivation to get together and play in the physical world.
When I read, at Jon’s request, a pre-publication draft of the book, I told him I could not support it, and I explained why. I had at that time already looked quite broadly and deeply at the research pertaining to questions about effects of screens, Internet, smartphones, and social media on teens’ mental health and found that, despite countless studies designed to reveal such harmful effects, there was very little evidence for such effects. If you survey the research literature selectively, with an eye toward finding studies that seem to show the effects you are looking for, and if you don’t analyze them critically, you can make what will seem to readers to be a compelling case. But people who really know the research and have examined it fully and critically will see through it.
I was also concerned that Jon’s book would feed into and exacerbate what already was verging on a moral panic about potential harms of smartphones and social media on kids. Jon begins the book by drawing an analogy between allowing our kids unfettered use of the Internet and sending them to grow up on Mars, where we know that conditions do not support human life. Yikes!
It is not at all surprising to me that the book has been praised to the skies by people who do not do research in this field and strongly criticized by people who do. Candice Odgers, Professor of Social Science and Informatics at UC Irvine—who many would agree is the leading US researcher on the relation of social media to teen mental health—began her review of Jon’s book in the journal Nature (here) with the following words:
“Two things need to be said after reading The Anxious Generation. First, this book is going to sell a lot of copies, because Jonathan Haidt is telling a scary story about children’s development that many parents are primed to believe. Second, the book’s repeated suggestion that digital technologies are rewiring our children’s brains and causing an epidemic of mental illness is not supported by science. Worse, the bold proposal that social media is to blame might distract us from effectively responding to the real causes of the current mental-health crisis in young people.”
Evidence Contradicting the Smartphone/Social Media Theory of Teen Suffering
I am not going to repeat here my rebuttals to the arguments presented in Jon’s book. I presented them, without mentioning Haidt’s name or the book, already in my D series of letters. Here is a list of some of those arguments, with links to previous letters if you wish to look back for review.
• Mental suffering among US teens increased dramatically between 1950 and 1990. The peak of such suffering in 1990 was as great as it is today, even though there was yet no Internet or social media (Letters D2 and D4). In fact, the initial correlate of introduction of the Internet was improvement in teen mental health between 1990 and about 2010 (Letter D3).
• Reviews of studies aimed at correlating time on social media with mental suffering among teens have shown mixed results with no consistent overall conclusion. Some studies show a small positive correlation, some show a small negative correlation, some show no correlation (Letters D6 and D7). Moreover, experiments in which some participants are asked to give up or reduce social media are fundamentally flawed methodologically (Letter D6).
• Cross-nation studies fail to show a consistent relation between the widespread use of smartphones or social media and teen mental suffering. In some English-speaking countries—notably Canada, the UK, and Australia, which in many ways follow US examples—teen suffering rose when smartphones and social media became widely available. But for most of the rest of the world, including the European Union taken as a whole, it did not (Letters D8 and D9).
• When teens themselves are asked about the source of their mental suffering, the great majority say it is school pressure. Much other evidence also points to school as a cause of anxiety and depression among teens. There are strong reasons to believe that increases in school pressure and decreases in school pleasure, resulting from adoption of Common Core, is the major reason for the increase in anxiety, depression, and suicide among US teens between about 2010 and 2020 (Letter D8). I will develop this contention much further in one or more future letters. Jon makes no mention of this possibility in his book.
Example of Critical Reading of a Research Report
To illustrate the problem of uncritical use of research reports, I will present here a methodological analysis of one of the articles that Jon cites. I could do this with other articles, but one example is enough to show, at least, the value of skepticism.
On pages 147-148 of The Anxious Generation, Jon claims that random assignment controlled experiments have shown that social media is a cause (not just a correlate) of teen suffering. He cites just one example of such an experiment, so I looked it up and read the article. The reference, if you want to look it up, is this: Melissa Hunt et al (2018), “No more FOMO: Limiting social media decreases loneliness and depression. Journal of Social and Clinical Psychology, 37, pp 751-768.
The research participants in this experiment were 143 undergraduate students randomly assigned to either limit Facebook, Instagram and Snapchat use to no more than 10 minutes per day, per platform, or to use social media as usual for a three-week period. Self-report questionnaires were used to assess various indices of subjective well-being, namely their sense of social support, fear of missing out, loneliness, anxiety, depression, self-esteem, autonomy, and self-acceptance before, during, and after the three-week period. The researchers also assessed the participants’ actual social media usage throughout the study by requiring them to submit screen shots showing accumulated use of these media.
The most damaging flaws with this study, which should be obvious to any social scientist, is there are no controls for demand effects or placebo effects. I’ll describe these separately.
Social scientists have shown repeatedly that when research participants can guess the purpose of an experiment and guess the researchers’ hypothesis, they are generally motivated, consciously or unconsciously, to support that hypothesis. In other words, they are likely to believe, or at least claim, they are experiencing what they assume the researcher expects them to experience, to prove the hypothesis correct. This is called the demand effect.
I’m sure there are some contrarians, who want to show the researcher is wrong, but many studies have shown that most college students bias their own responses in the direction that fits their understanding of what the researchers want to find. In this case it would have been obvious to most if not all participants that the researchers were trying to show that reducing social media use would make them feel better. Good social science research requires that the participants have no idea what the experiment is all about or what the researchers’ hypothesis is, but in this case there was no way to hide that.
Now the placebo effect. This refers to the simple fact that when people believe they are doing something that will make them feel better, that belief by itself makes them feel better. This is why it is so difficult for drug companies to show that anti-anxiety drugs decrease anxiety or anti-depression drugs decrease depression. They must always conduct tests in which the control group is given an inactive placebo, disguised in such a way that the participants don’t know if they are getting the drug or the placebo.
In such experiments the placebo as well as the drug leads to reduced anxiety or depression. To show that the drug works, you must show that it reduces anxiety or depression more than the placebo does. That is very hard to do because the placebo effect is so powerful, especially for such subjective feelings as anxiety or depression. In this study, there was no placebo. The participants in the experimental group knew very well that they were in the experimental group, and those in the control group knew they were in that group. Only those in the experimental group had reason to believe they were doing something good for themselves.
So, even if this experiment had shown clear reductions in every subjective measure of distress used in the study, it would be impossible to draw any firm conclusion. Any reduction could be attributed to the demand effect, the placebo effect, or both.
But now here is the kicker. Despite that possibility of demand and placebo effects, the researchers found very weak and inconsistent effects of the intervention on their measures of wellbeing. They found no significant overall effect on psychological wellbeing (combining the measures), no effect on anxiety, no effect on self-esteem, no effect on autonomy, no effect on self-acceptance. They found only rather small effects on self-reported loneliness and depression, and the latter effect was only significant for those participants who showed higher than average depression at the beginning of the study.
When you measure a lot of things, without predicting beforehand which things will be affected and which will not, one or more of those things might come out, by chance, in a manner consistent with the initial hypothesis, especially when the effects are rather small. In my mind it is at least as reasonable to attribute the reported findings in this study to chance, or to the placebo effect, or the demand effect, or to some combination of all of these, as it is to attribute them to the reduced use of social media.
Good research in social science is not easy, and the task of interpreting research in social science is not easy. To decide what to convey to readers not familiar with social science methodology or the specific studies under question, it is essential to look closely at the experimental methodology, not just at the title the authors give to the report or what they say in their summary.
Concluding Thoughts
I would not be critiquing Haidt’s book if I thought it was just an intellectually interesting or entertaining read. It is, in fact, a call for action, and it is having a powerful effect on the national dialogue and policy proposals. Too many people are taking this book as an authoritative summary of the research regarding effects of smartphones and social media and concluding that we must take these tools away from children and even early teens. As a society we have almost a knee-jerk reaction to believe that the solution to any problem experienced by kids is to deprive them of yet one more freedom, and this book is helping to jerk some of those knees even further.
I do not claim there are no dangers in exploring the digital world, just as I do not claim there are no dangers in exploring the physical world. But rather than deprive kids of opportunities for such adventures and the learning that comes from them, we should be teaching safety rules. We should also develop norms about turning off phones and other devices when face-to-face interactions are possible. It is rude to answer a ding on your phone when you are face to face with me, unless it is something that concerns us both, and I see that rudeness in adults at least as much as I do in kids. I’ll say more about all this in a future post. I’ll also, in a future post, address the overuse of the word “addiction” in our culture and why its application to heavy use of smartphones or other digital technology is misleading and harmful.
As always, I am very much interested in your thoughts on what I have presented here. Your comments add to the value of each of my letters, for me and other readers. These letters are designed to promote critical thinking, and I am always interested in readers’ thoughts if they are thoughtfully and politely presented.
If you aren’t already subscribed to this Substack, please subscribe now, and let others who might be interested know about it. By subscribing, you will receive an email notification of each new letter. If you are currently a free subscriber, consider converting to a paid subscription. All profits that come to me from paid subscriptions are used to help support nonprofit organizations aimed at bringing more play and freedom to children’s lives.
With respect and best wishes,
Peter
Friends, thank you for all the thoughtful comments here. Because there are so many, and because some overlap with others, I will not attempt to respond individually. Instead, I plan to write a follow-up "thread" that will address the comments. I'm not sure when I will get to that. I hope soon, but I will certainly aim to do it within a week. My wife and I are leaving tomorrow for a few days at our cottage in the little village of Cabot, Vermont, where I graduated from high school and where they have the best Fourth of July parade you will find in any little village anywhere.
I am struggling with my emotions a bit on this one. I highly respect your work and agree with most of your conclusions—but I feel the same way about Haidt’s work. And I hate to see you stand against him, especially because in many ways, your goals align. I worry by opposing his conclusions, you may be limiting the reach and effectiveness of your own, and your conclusions work really well together.
Anecdotally, with my own gen z children, I see the negative consequences of social media pretty strongly. Now, could some of it be simply the middle school struggles we all experienced because of puberty and other universal changes that come with reaching the teen years? Sure—but I also know that my son’s access to porn would be much lower, and my daughter’s opportunity to be drawn in by the transgender “influencers” would have been next to zero. Are both of them already experiencing higher than normal levels of anxiety—absolutely, but there is also a genetic predisposition that makes this unavoidable, and social media comparisons make it worse.
They absolutely reduced their outdoor and free play time when they got phones, and since their friends did, too, no one goes out to play anymore.
My younger (right on the edge of being considered gen alpha) daughter didn’t want a phone until she got to middle school and knows she will be out of the social loop if she doesn’t have one. So she has one. Ugh! She would prefer her phone to backyard play with her younger brother—the only person around without a phone and willing to play outside.
We DO need to change school culture and increase free play time. 100%. But right now, my son is surrounded by other highschoolers just nose in the phone during any free time there is—there is no play happening. These things go hand in hand.
Another aspect of Haidt’s work includes what I would consider a vital change that took place in the middle of the last century—the meaning crisis. His work addresses that.
With the momentum his book has, you would do better to ride the wave than try to swim against it. As we all know from politics, people usually are more strongly led by how they feel than by cold reason. I hate to think you will fall on the wrong side of that equation.