August 28, 2025
Volume 23, issue 4

Download PDF version of this article PDF

Understanding the Harm Teens Experience on Social Media

A systematic approach to mitigating negative experiences online

Arturo Béjar

Every day, millions of kids are having preventable harmful experiences on social media. Internal research at Instagram, a statistically significant survey called BEEF (Bad Experiences and Encounters Framework), found that for 13- to 15-year-olds who had used Instagram over the previous seven days:

1 in 8 experienced unwanted sexual advances.
1 in 10 experienced being the target of bullying.
1 in 3 witnessed bullying happening.
1 in 5 received unwanted sexual content.
1 in 5 saw unwanted graphic violent content.
1 in 5 felt worse about themselves.

During that internal research process, Meta found that, of the people who had a harmful experience, 1 percent ended up submitting a report, and out of those who submitted a report, 2 percent resulted in an action. Out of 10,000 people who had a harmful experience, two got help by submitting a report. This critical gap means that almost all harm—99.98 percent—is not addressed by the current approach. This becomes even more concerning when you begin to understand how harm plays out for teenagers online.

If you are young, think about your own experiences: How often have you or your friends experienced these issues? Were you searching for trouble, or did it just get recommended? When you had a harmful experience, did you feel you could do anything about it? Did you get tools that made you feel that they would help protect others? Did you ever use reporting? And did it help?

If you're a parent and your kids are on social media, ask them about it: What are some of the worst things they've seen? Were they searching for them? What are some of the worst things that have happened to them or their friends?

I've talked to around 40 parents who have lost their kids, and others whose kids ended up in the hospital. I've spoken to parents and survivors of grooming, eating disorders, and body image issues and to parents of kids who have committed suicide because of bullying. When you look at the data around what each of their kids has experienced, you find that all of these issues are eminently preventable.

Somehow, in this age of technological wonders, we have come to accept that for a 13-year-old to go on social media, the harm they experience is the price of admission for the good they get from the services. When parents hand a phone to their kids, they should know how likely it is that their kids will experience sexual harassment, become the targets of hate, experience harrowing bullying, or get exposed to a firehose of content that will make them feel bad about their bodies. If you are a parent, you should know that today your kid is not given effective tools to handle distressing content, to report inappropriate contact, or to manage their time spent and attention, no matter how bad things get for them.

Social media companies have the technology and the expertise to understand and significantly reduce the harm young people experience while preserving the positive aspects of being connected to each other online. I know this because for six years (2009–2015) and for 1.2 billion people, I managed engineering, product, user research, data science, and design for security, safety, and customer care at Facebook. Teams I managed built machine learning infrastructure that evaluated billions of pieces of content a day in milliseconds. These teams built features that helped with suicidal ideation and tools designed for teens to help with harmful experiences, including bullying. All this work was done with academic experts embedded in the product and engineering teams, with full access to data in a way that was mindful of privacy and security. Results were published about the development process, lessons learned, and the effectiveness of safety tools. I have many years of experience with the extraordinary technological feats social media companies are capable of when something is a priority.

Unfortunately, in recent years, leaders at social media companies have not taken responsibility for the harm their products allow and amplify. Instead, it appears as if their safety goal has become to invest in marketing and PR and try to create a conversation that dismisses or minimizes the harm our kids experience. Social media companies do not provide the transparency that we as parents deserve for our kids, and they create flawed safety features that provide only an illusion of safety.

One of the reasons we have ended up here is that, in general, the approach to harm by most companies relies on content moderation needing to meet some objective harm definition, because if it meets that definition, it gets deleted. Content moderation is necessary but not sufficient. It is not how any of these harms are handled face to face. If a student approaches a teacher in school and talks about being sexually harassed, what happens from that point does not depend on whether a specific word was used.

People's experience is the ground truth of harm. One does not "perceive" being harassed; one experiences it, even if the content is not deemed discernibly harmful by an independent observer.

Once you understand harm from the experience of the teen and combine that with other available features, you can develop tools that are appropriate to the combination of experience, content, and context. The measure of the effectiveness of these safety tools should be done in terms of reducing the harm that people experience. It is important to set goals based on these categories.

The good thing is that we, as engineers, have always had the tools and methods to significantly understand and reduce harm. Look at the progress we've made on security and spam. The data is there, and the strategies are effective. Engineering for online safety is similar to engineering for online security. It is a mindset that drives a different way of working. Safety engineering does not require a significant new investment in terms of content review or infrastructure. Safety engineering is about capturing the right information and creating a system that includes effective feedback loops.

The key difference between safety and security is that when you engineer for safety, user feedback is an essential part of the system design. It is essential to capture user experience information, to combine that information with other features to prevent harm, and, where appropriate, to give feedback to the actors. When designing social systems, engineers must think about safety as incorporating the people and their responses to experiences into the system design.

Harm needs to be understood from the perspective of the person experiencing it.

In order to effectively reduce harm, it is essential to first understand it from the perspective of the person who is experiencing it.

If you talk to teenagers about the harm categories included in BEEF or do testing using avatar accounts, you will quickly find concrete examples for each of the categories mentioned. All of these examples were delivered to an account of a 13- or 14-year-old girl.

For unwanted sexual advances – Comments on posts or private messages asking for sex. Things like ("dtf," or "Netflix and chill"), GIFs with condoms or sexual references, or unwanted intimate pictures—sometimes from strangers, sometimes from people in their social graph.
For kids who are the targets of bullying – Coded comments, putdowns, variations on slurs, comments that contextually are profoundly distressing.
Graphically violent content where you see people jumping off buildings, getting hit by cars or hitting the sides of pools or each other and clearly breaking bones, but with no gore.
Content that triggers eating disorders – Like a plate with a single piece of broccoli, or images that manipulate body shapes to make someone look too thin or overweight, recommended in the hundreds or thousands.
Content that is related to self-harm – Black and white images with poems about how the world would be better off without you, recommended in volume.
Sexual content that does not have any nudity – Instead, graphic descriptions of demeaning sex acts using cartoons, or people graphically and enthusiastically describing demeaning sexual experiences—for example, "First day of a challenge of getting railed 365 times in 365 days."

What all of these examples have in common is that the contact, or the content, is designed to circumvent content moderation. The broken-bone videos do not show blood or gore. The sexual videos don't show private parts. Keep in mind that if one piece of content does not seem too bad on its own, the important thing is to understand what can happen when hundreds or thousands of pieces of certain kinds of content get recommended to a kid.

A key misconception here is that harmful experiences only affect vulnerable kids. All of us when we were teenagers experienced moments of vulnerability: a break-up, a comment about our body, an argument with a friend. When talking to kids who are survivors or parents of children who have died or ended up in the hospital, it becomes clear that this could happen to any of our kids. This is why it is so critical to build products that are safe enough.

Why do people post these kinds of content? Because the recommendation algorithm as currently designed rewards them, and it does not do a good enough job of separating content that is fine to recommend to adults but should not be recommended to a 13-year-old. Current algorithms, in a nutshell, notice when a kid spends time on a piece of content and are extremely efficient at recommending similar content. The main features are designed and measured so that users spend time on them; notifications are designed and measured to bring users back in. There is no upper bound to the time spent by a child, and there is no measure of whether that time is helping or hurting the child.

What these algorithms don't account for, and the product design doesn't allow for, is when the content is watched out of disgust or if it is negatively impacting the teen. Every teen I've spoken to feels powerless about the recommendations they get.

Sometimes I've heard online harms compared with face-to-face harms, that bullying at school is equivalent to bullying online, or that beauty imagery in magazines and televisions is the same as an infinite feed of content run through "beauty" filters. Online harm can be much more significant for three reasons: persistence, distribution, and scale. The situation is very different when someone makes a distressing comment that only a few people around you hear, compared with when someone posts a distressing comment that the entire school might see in a matter of minutes. It is very different to see a magazine with a few ads, compared with getting a personalized feed of thousands of posts of images that have been manipulated. If you spend time reviewing the content in question, you'll see the differences.

If you would like to gain a better understanding of these experiences, I recommend opening an avatar teen account and seeing for yourself the kind of content social media is recommending to teenagers today for these classes of harm.

So in this landscape, what can we as systems designers do? The first step is getting and understanding the data around the problem we are trying to solve. In this case, it is understanding harm from the kid's experience. As you do that you will find many ways it can be reduced and ways to measure the effectiveness of these changes. The first step to reducing harm is understanding it from the perspective of the person who is experiencing it.

Placebo Features

When I talk about lightweight negative feedback, some will point at the "Not interested" feature that some products have. That button or "Mark as spam" are two examples of placebo buttons. For many years "Mark as spam" clicks were just dropped. Testing of the "Not interested" button on the avatar account on Instagram, many times in a row, did not stop violent content from being recommended.

Different Kinds of Harm, and Strategies to Understand and Reduce it

Some of the kinds of harm that happen on social media platforms include:

Inappropriate conduct and contact – Unwanted sexual advances, bullying and other forms of aggressive or attacking contact, misogyny or violence toward women, other attacks toward people because of some part of their race, religion, or identity. These generally happen in messages or comments.
Exposure to "sensitive content" – Sexual, self-harm or depressive, violent, hateful, negative body image, drugs, etc. and the related rabbit holes or downward spirals.
Harmful usage, also called addiction – When time spent on the product has a negative impact on the child's life. Some examples include increasing anxiety, depressive behaviors, distancing from family or friends, etc.

Note that when you have enough information to prevent some kind of harm with enough confidence, that should always be the first step. This approach is necessary but not sufficient, and needs to be approached with care. Systems that rely primarily on enforcement have the simultaneous risks of over-enforcement (removing content that should not be removed), and under-enforcement (allowing and distributing content that should be removed). If your safety tools include enforcement, it is important to measure both under- and over-enforcement, as measuring only one of them inevitably results in significant problems.

Years of working on reporting tools taught me that most harmful experiences, including some of the most intense, understandably do not objectively violate content policy. Conversations with people who do sextortion or groom minors primarily consist of compliments. The most harrowing bullying can appear like positive comments to an observer. This is because many types of harm can be contextual: school, culture, slang, etc. Another kind of harm is by volume. A single image of depressive or body-image content looks innocuous on its own. But when recommended by the thousands, they can lead to a teen not eating anything that day, or worse.

Because of its nature, most of the contact and content-based harm that teens experience would not benefit from a drastic increase in moderation or content review. This is a great opportunity for innovation, and the first step is to understand the combination of contact/content, user experience/response, severity/intensity, and action taken.

Effective Reporting

In order to get the information needed to understand and mitigate harm, it is so critical to have effective reporting, which is the term I use to describe well-implemented feedback and reporting mechanisms. In order for these mechanisms to be effective they need to:

Be easy and rewarding to use.
Capture what happened (harm), where it happened (context), and how bad it was (i.e., intensity and severity). All these steps should not be required, but they should be available and rewarding to use.
Provide immediate support and protection for the user, independent of any content moderation considerations.

All of these features need to be designed to gather information and provide support independent of the outcome of content review.

The way you measure effective reporting is by asking a sample of users at the end the following questions:

Were you able to let us know what happened?
Were you able to let us know how bad or intense it was?
Were we able to provide support for the issue you are experiencing?

This is a variation of the basic structures used to assess the effectiveness of customer care.

Providing immediate relief and support is critical to reducing harm in cases of harassment, bullying, grooming, people who might be thinking of self-harm, and other areas. In these cases, the messages sent or comments made are likely not going to be found to be violating in content review. In reviewing comments and content reported for bullying, more than 90 percent did not violate any policies, and in 50 percent of the most severe cases of bullying, the content looked benign or positive to the reviewer.

When building effective reporting tools for 13- to 15-year-olds at Facebook, we learned a number of critical lessons:

Teens do not like to click on the word "Report," as they think it will get them or their friends in trouble. So, different prompts were tested, such as "I don't want to see this." Changing the initial prompt doubled or tripled usage of the reporting tool, depending on the circumstance.
Use language that matches how teens describe the issues they are experiencing. For example, the initial option was "Bullying and harassment," which no teen used to describe what happened to them. When asking teens, they said that what they were experiencing was "Somebody is threatening me" or "Someone is spreading rumors about me," etc. When the options matched the experience, usage went up significantly.
As long as the steps were helpful, teens were fine taking multiple steps when asking for help—which was counter to the understanding that fewer steps would improve completion. What we learned is that when someone asks for help, as long as each step has a clear purpose and benefit, teens will take multiple steps.

When we started, the traditional reporting flow had a completion rate of 10 percent; when we applied the findings just mentioned, the completion rate went to 82 percent, and 60 percent of people reported feeling better after using these tools.

Effective reporting is one of the most critical sensing mechanisms of harm. Well implemented, it should be rewarding for the teen to use, and at the same time capture many signals that then could be used by other systems to proactively detect and prevent harm. It is the first step in a framework that can be used to meaningfully reduce harm.

There are many opportunities for innovation here. Any innovation that helps the system designers discover and understand harm as experienced by the user can then create different kinds of features that can proactively prevent harm or provide better support when someone needs it.

Developing Safe Products

The key questions for systems designers are:

How do you first discover or become aware of harm?
How do you understand the nature and severity of the harm?
How do you help someone who is experiencing harm at the moment they are experiencing it or when they ask for help?
How do you protect other people from experiencing the same harm?

One of the things I've learned about working in the integrity and safety space for the past 30 years is that while there are no perfect solutions, there is a product-development process to reduce harm as soon as you become aware of it. The framework to approaching, understanding, and reducing the harm that teens experience online has four elements:

Effective reporting—You need to know what happened, where it happened, and how bad it was (severity or intensity).
Using the information to help the person and protect others
Where possible, giving feedback to the actor who initiated the interaction
Measuring and monitoring the effectiveness of the program

How do you understand the nature or severity of the harm? You ask people. Often in my career I've heard people talk of "bad reports," or that reporting data is "noisy." This is not an accurate description. The main reason people dismiss reports is that they don't understand why something was reported, or reported content does not meet the company's definitions of that harm. The harm is real; what is needed is to provide appropriate help to the person with the issue they are having and to have a proportionate response.

For example: A person reports for "nudity or sexual activity" a video of someone apparently performing fellatio on someone else, except you cannot see penetration. To the person submitting the report, it is clearly sexual activity, but when your internal definition of sexual content requires that you see penetration, the video is not acted on. That doesn't mean the video isn't harmful.

When reviewing the messages for unwanted sexual advances or bullying, you quickly find that most cases would not benefit from content moderation, as the messages are often not harmful to an independent observer. The question to ask is: What would you build if someone was experiencing harrowing bullying, but the messages were not objectively harmful?

An important question to explore is: How would you create a safe messaging experience if you had no access to content moderation?

One way to work on this problem is through a conduct-in-context–based approach. This relies on giving a teen an easy way to tell you when they've experienced an unwanted advance (conduct) in messages (context) or in a comment on their own posts (context). Then you can keep track of conduct in context and apply proportional responses. For example, you can provide a prominent button in any messaging feature that makes it trivial for a user to indicate when a conversation is inappropriate and the reason why—to help the user easily say if the conversation involves harassment, an unwanted sexual advance, and if they believe that the account contacting them is fake. Because the context is the user's messages, the content does not matter; what matters is that the user flagged it as inappropriate contact. In general, this is how society deals with harassment in school or in the workplace. You don't ask the person if a specific set of words was used; you treat their experience as the ground truth, and you have a proportionate response, such as giving the initiating actor feedback rather than punishment, unless there is a pattern of inappropriate contact.

This approach would be problematic if the result of a person flagging contact as inappropriate resulted in an immediate negative consequence to the sender. Instead, the process is to look for behavioral patterns before taking a structured approach to giving feedback to the actor. For example, if someone gets flagged for unwanted advances once in messages, that should not be an issue, but if they get flagged three to five times, then you respectfully notify them that unwanted advances are inappropriate. These interventions are called nudges.

During the work of Facebook's Protect and Care team, these kinds of nudges were applied, and the team consistently found that most people, given private respectful feedback, change their behavior (from 50 to 75 percent of the time). There are people who will not change harmful behavior, which then requires an escalating approach such as feature limits, etc. It is essential to have an approach like this, as it helps sift out the people who, when given a respectful reminder, would change behavior from those who will behave harmfully repeatedly. One good way to think about this is to design nudges and escalation systems as if you were the person on the receiving end of the nudge.

There are three key benefits to the conduct-in-context approach combined with nudges:

It helps address harm at the supply by respectfully communicating social norms to people. Addressing harm at the supply point is the most effective long-term strategy.
It helps identify people who will engage in harmful behavior repeatedly, separating them from others.
At each step, this approach creates a high-quality signal for all other integrity systems, which could then be used very effectively to identify similar harmful actors. For people who build integrity systems, the quality of the training dataset has a profound effect on the precision and recall of machine-learning datasets. Training on data around context-conduct-repeat could lead to effective classifiers.

This perspective is in contrast with most current thinking about moderation. Companies create systems and processes for reporting, detecting, and punishing around their interpretations of violating content. These rely on features that the content needs to have unambiguously (for example, visible private parts in order to identify nudity). These systems are necessary but not sufficient, and their design introduces a number of unintended consequences: People quickly develop conduct and content that circumvents this approach, yet the harm is the same. The content moderation toolbox isn't able to create an environment where teens do not get unwanted sexual advances.

When you understand harm from the perspective of the person experiencing it, there is an extraordinary amount of innovation available. How do you build a safe messaging experience if you don't have access to content moderation? Some of the examples covered here only scratch the surface of the innovation possible when you understand the harmful experiences people have when using your products.

Lessons Learned from Developing Tools for Teens

Initially, the reporting tools for teens on Facebook had a completion rate of around 10 percent. The main issue was that they were not designed for teenagers. The major lessons learned were:

Language matters. Avoid using the word "report" as the entry point because of findings about teens who do not "report" out of worry that they or others would get in trouble. Using different language, such as "I don't want to see this," doubled or tripled usage of the tool. For the options—the language in the product should describe what people are experiencing as they themselves describe it. Changing the options to match how users describe issues increased usage 30 percent to 60 percent.
There are great opportunities here for open text boxes and the applications of LLMs in understanding the issue that the person is experiencing.
Context matters. Interactions should be evaluated in context; it is different when someone slides into a teen's DMs and attempts sextortion than when somebody makes a comment on someone else's public post.
Intensity matters. The ability to capture intensity or severity is essential for two reasons: You cannot understand how harmful the interaction is for the teen from just the content, and without knowing it you cannot provide immediate, appropriate support or help. Capturing intensity should be optional, as to avoid an unnecessary burden in the use of the tools, but we found that as long as the tools were helpful, teens used them and that the ability to share emotion and intensity can also provide meaningful benefits in terms of self-regulation during a distressing experience.

Based on the combination of issue, context, and intensity, the teen would get a personalized intervention. If someone was being mildly annoying, the intervention was light. If a teen was feeling profoundly afraid, angry, or sad, they got a different intervention. When applying these principles, the completion rate for these tools went from 10 percent to 82 percent, and more than 50 percent of teens felt better after using the tools.

Creating Effective Safety Tools

As products increase in complexity, effective reporting is one of many safety tools that are necessary to create a safe environment. A safety tool, in order to be effective, needs four features:

Prevention – It must effectively prevent the harm from happening to a teenager.
Resiliency – A safety tool needs to be resilient to manipulation (i.e., if there is an easy workaround (teenagers are great at that), then it really is not a safety tool).
Protection for others: When harm does happen, a safety tool should help with the harm and capture information that prevents others from experiencing harm.
Ease of use: The tool should be on by default or take one click or swipe to use.

A good safety tool needs the ability to capture the correlated set consisting of issue, intensity or severity, context, and interaction or content (for example, a profile, a set of messages, or a piece of content). This dataset is a behavioral correlate.

These behavioral correlates can be used for many goals. For example:

Identify behavioral patterns that need to be changed, and test the effectiveness of different interventions.
Train classifiers to personalize content delivery in order to avoid delivering content that is distressing to someone.
Train classifiers to protect others from having similar experiences.
Identify new forms of harm or sets of bad actors as they develop.
Measure agency based around a specific behavioral correlate, which helps determine next steps to reduce harm.

Example: Unwanted sexual advances in direct messaging

How do we help teens deal with unwanted sexual advances over DMs while also communicating to the people who initiate the unwanted advance that it is not appropriate?

To help deal with unwanted sexual advances in messages, the following could be built rapidly:

1. Providing a prominent "help" button, especially during the early messages between any two people.
2. Making it possible to give the reason why and, where appropriate, indicate severity. Reasons offered in the flow can include:

"It's gross or upsetting."
"They're fake."
"They are harassing me."

3. Using the help button should feel rewarding to the person using it; immediate feedback can include:

"We've blocked the person."
Depending on the severity of the harm, other resources should be provided—for example, tools to lock down who can contact the victim, or links to offline resources—for example, links to victim services/helplines (https://cybercivilrights.org/ccri-crisis-helpline/).
Thank the person for giving feedback, because it will help make the community safer.

4. Keeping track of how many times someone has initiated unwanted contact and the reason why.
5. Giving a respectful, gentle "nudge" to someone who initiates unwanted contact that the recipient considers "gross" or "harassment" more than three (five?) times. For example: "Hey Alice/Bob, we've gotten feedback that you've sent some unwanted messages. This is a place where people are respectful to each other, so if you're taking an action, please make sure that it is welcome."
6. Escalating the feedback into feature blocking if someone ignores the nudge and continues the behavior. (for example, reducing their ability to contact new people, and/or use stronger language).

A few comments about this approach:

It doesn't matter what the content of the message is. What matters is the context in which it happened and that it was unwelcome.
It works with end-to-end encryption.
It will generate data about which actors will change behavior once they are given feedback. In my experience, more than 50 percent of people change behavior on the nudge. The goal is to separate people who respond to feedback from those who won't.
The data can identify predators, because they likely will initiate a number of unwanted contacts before finding someone who will engage with them.
The data regarding unwanted contact that is considered "fake" could lead to identifying networks of fake accounts.
Abusing this help mechanism would be difficult, as the people who could give this feedback would be limited to those who were initially contacted by the actor. As with any possible misuse, however, there should be something in the product that helps someone appeal or that would detect these cases.

Conclusion

The current approach to online safety, focusing on objectively harmful content and deletion or downranking, is necessary but not sufficient, as it addresses only a small fraction of the harm that teens experience. In order to understand harm, it is essential to understand it from their perspective by surveying and creating safety tools and reporting that make it easy to capture what happens and provide immediate help. Many of the recommendations in this article come from what you learn when you analyze behavioral correlates: that you need approaches that rely on conduct in context, better personalization, and providing feedback to actors.

The fundamental approach should be to set goals based on the harm teens experience rather than the harm the company can objectively identify. Teens' experience is the ground truth of harm.

Safety tools require the same approach as security tools. They need to be resilient, as easy to use as liking or on by default, and they need to gather information that protects the teen and protects others. It is important that they are subjected to "red team" or adversarial testing by people in the company as well as third parties.

Once you broaden the scope away from objectively harmful content, there are many opportunities for innovation: using LLMs to understand the issues people are experiencing, training classifiers on behavioral correlates to proactively identify issues, and developing new ways to integrate people's experiences into systems design.

Start by asking the following questions: What percentage of teens should have unwanted advances? Or get exposed to unwanted violent, sexual, or self-harm content? What do they do when it happens? Does that interaction capture enough data to understand what is happening? Are there proportionate systems responses that reduce the likelihood of harm? And perhaps the most important question of all: What would I build that I would want my kids to use? That is the basis on which everything we do should be executed.

Arturo Béjar was the senior leader at Facebook responsible for its engineering and product efforts on site integrity, security, safety, and customer support, from 2009 to 2015. He was also the manager for Facebook's product infrastructure team, which created core technologies (including REACT). From 2019 to 2021, Bejar returned to the company as a consultant to work on well-being issues at Instagram. Prior to that, he started as an engineer, and eventually became the head of information security (chief paranoid) at Yahoo!.

Originally published in Queue vol. 23, no. 4—
Comment on this article in the ACM Digital Library