An inside look at X’s Community Notes | Keith Coleman (VP of Product) and Jay Baxter (ML Lead) · Lenny's Podcast: Product | Career | Growth summary

Key takeaways

Community Notes utilizes a bridging-based algorithm that fosters accuracy by seeking agreement from users with differing viewpoints.
The product successfully navigated multiple leadership changes at X, highlighting its strong principles and community-driven approach.
Anonymity in contributions has significantly increased participation and improved the overall quality of notes.
The team operates with a high degree of autonomy and efficiency, foregoing traditional project management methods for a streamlined approach.
Community Notes has demonstrated real-world effectiveness in addressing misinformation, proving essential in today’s digital landscape.
Future enhancements include the potential integration of AI and machine learning to further optimize the accuracy and efficiency of community contributions.

Summary

Introduction to Community Notes

Keith Coleman, VP of Product at X, and Jay Baxter, ML Lead, introduce Community Notes, a crowdsourced information system aimed at correcting misleading posts and fostering accurate discussions. They discuss the origin of the project, highlighting how it aims to reduce misinformation on social media.

The Bridging-Based Algorithm

The podcast dives deep into the workings of Community Notes' unique algorithm that prioritizes 'bridging agreement' rather than simply majority opinion. Keith and Jay explain how the algorithm, which relies on the agreement of contributors with differing views, enhances the neutrality and accuracy of information shared.

Navigating Leadership Changes

The hosts share stories about how Community Notes has thrived through multiple CEO transitions at Twitter (now X), from Jack Dorsey to Elon Musk. They discuss maintaining focus and commitment to the product's principles during these leadership shifts.

Community Contribution and Anonymity

The discussion touches on the transition to anonymous contribution, explaining how this shift increased participation and the quality of notes. The hosts emphasized that anonymity allows users to express themselves freely without fear of backlash.

Team Structure and Operations

Coleman and Baxter detail the small, focused team structure that propels Community Notes. They reveal their unconventional approach to management—operating without standard project management tools like OKRs or Jira, instead relying on a single Google Doc to track goals and progress.

Real-World Impact of Community Notes

The hosts reflect on tangible instances where Community Notes has effectively corrected misinformation, such as major political events or viral media posts. They emphasize the importance of community involvement in fostering a more accurate information ecosystem.

Future Directions

Looking ahead, Keith and Jay discuss the further integration of AI technologies within Community Notes, such as the use of Large Language Models (LLMs) to assist in note generation. They express excitement about potential collaborations and innovations that could enhance the project.

Show notes

Keith Coleman (VP of product) and Jay Baxter (founding ML engineer), the minds behind Community Notes, reveal how a small, scrappy team inside Twitter/X built the most trusted crowdsourced information system on the internet—one that’s changing the way we understand truth online. What you’ll learn:

1. How Community Notes actually works—a deep dive into the groundbreaking algorithm that rewards “bridging agreement” instead of majority rule

2. The seemingly crazy yet brilliant way this idea survived multiple CEO changes—from Jack to Parag to Elon

3. How this project started with a dumpster fire GIF (literally)—the untold backstory of its early launch

4. The secret to running ultra-fast, high-impact product teams—no OKRs, no Jira; just one Google Doc

5. What Meta’s adoption of Community Notes means for the future of online (mis)information—why this open source system is becoming the industry standard

—

Brought to you by:

• WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs

• Productboard—Make products that matter

• Wix Studio—The web creation platform built for agencies

—

Find the transcript at: https://www.lennysnewsletter.com/p/how-x-built-the-best-fact-checking-system-on-the-internet

—

Where to find Keith Coleman:

• X: https://x.com/kcoleman

• LinkedIn: https://www.linkedin.com/in/keith-coleman-19b12b46/

—

Where to find Jay Baxter:

• X: https://x.com/_jaybaxter_

• LinkedIn: https://www.linkedin.com/in/jaybaxter/

• Website: http://jaybaxter.net/

—

In this episode, we cover:

(00:00) Introduction to Community Notes

(06:56) How the “bridging-based” algorithm works

(13:33) The impact and scale of Community Notes

(17:24) Understanding the note publishing threshold

(21:32) Challenges and philosophies

(26:26) The effect of notes on re-sharing content

(29:41) Origin story

(35:46) Embracing small teams for big impact

(40:23) The thermal project approach

(47:47) Algorithm development and internal competitions

(50:34) An inside look at how the team operates

(58:56) Working with Elon

(01:05:30) Launching Birdwatch

(01:10:48) The core principles behind Community Notes

(01:26:15) Anonymity and pseudonymity in contributions

(01:32:17) Sustaining the project through leadership changes

(01:37:57) Future directions for Community Notes

(01:42:12) Final thoughts and optimism for the future

—

Referenced:

• Community Notes on X: https://x.com/CommunityNotes

• Sign up to be a Community Notes contributor: https://communitynotes.x.com/guide/en/contributing/signing-up

• The Making of Community Notes: https://asteriskmag.com/issues/08/the-making-of-community-notes

• “Readers added a Community Note to this Tweet”: https://x.com/HelpfulNotes/status/1718103364792205704

• Note-ranking algorithm: https://communitynotes.x.com/guide/en/under-the-hood/ranking-notes#matrix-factorization

• Study: Community Notes on X could be key to curbing misinformation: https://giesbusiness.illinois.edu/news/2024/11/18/study--community-notes-on-x-could-be-key-to-curbing-misinformation

• Study Finds X’s (Formerly Twitter’s) Community Notes Provide Accurate, Credible Answers to Vaccine Misinformation: https://qi.ucsd.edu/study-finds-xs-formerly-twitters-community-notes-provide-accurate-credible-answers-to-vaccine-misinformation/

• Did the Roll-Out of Community Notes Reduce Engagement with Misinformation on X/Twitter?: https://dl.acm.org/doi/10.1145/3686967

• Kayvon Beykpour on LinkedIn: https://www.linkedin.com/in/kayvz/

• Jack Dorsey on X: https://x.com/jack

• “Birdwatch gives me the creeps” tweet: https://x.com/elonmusk/status/1589454464611540992

• Blake Scholl on LinkedIn: https://www.linkedin.com/in/blakescholl/

• Creating Truthtelling Incentives with the Bayesian Truth Serum: https://www.eecs.harvard.edu/cs286r/courses/fall12/papers/DW08.pdf

• Asana: https://asana.com/

• Spaces: https://blog.x.com/en_us/topics/product/2021/spaces-is-here

• Amazon MTurk: https://www.mturk.com/

• Community notes on GitHub: https://github.com/twitter/communitynotes

• What do I think about Community Notes?: https://vitalik.eth.limo/general/2023/08/16/communitynotes.html

• X’s community-led approach: tackling inaccurate and misleading information: https://blog.x.com/en_us/topics/company/2023/xs-community-led-approach-tackling-inaccurate-and-misleading-information

• Linda Yaccarino on LinkedIn: https://www.linkedin.com/in/lindayaccarino/

• Messi-Ronaldo rivalry: https://en.wikipedia.org/wiki/Messi%E2%80%93Ronaldo_rivalry

• Supernotes paper: https://arxiv.org/pdf/2411.06116v1

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

—

Lenny may be an investor in the companies discussed.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.lennysnewsletter.com/subscribe

An inside look at X’s Community Notes | Keith Coleman (VP of Product) and Jay Baxter (ML Lead)