Free Tweet Collector: Build a Browser Extension & Server (2025)

Introduction

Imagine you're casually scrolling through Twitter and every tweet tells an interesting story.
But as a developer, you notice that the paid Twitter API is just too expensive for your budget.
What if you could capture those tweets for free, store them on your own server, and even analyze them later?

In this guide, we'll show you how to build a Free Tweet Collector that does exactly that!
You'll create a browser extension that automatically collects tweets as you scroll, and a FastAPI server to store them.

This tutorial is designed to be beginner‑friendly, so even if you're new to browser extensions or FastAPI, you'll follow along with ease.

What You’ll Learn

• Build a Browser Extension:
  Create an extension that detects tweets on Twitter and automatically saves them.

• Understand Every Piece of Code:
  We'll break down the content script, manifest file, and popup interface so you know exactly what each part does.

• Set Up a FastAPI Server:
  Learn how to build a server that receives tweet data, validates it using Pydantic, and stores it in a database using SQLAlchemy.

• Best Practices & Troubleshooting:
  Get answers to common questions and tips to ensure your tweet collector runs smoothly.

A Quick Story: Why Build This Tool?

One day, while browsing Twitter, you come across a tweet so interesting that you wish you could save it forever. Unfortunately, relying on the paid Twitter API isn’t an option. You realize that by building your own tweet collector, you can capture and store these moments without extra cost. This project isn’t just about saving tweets—it’s about empowering you to take control of your data and learn how to build real‑world applications.

How This Guide is Structured

1. Part 1: Creating the Browser Extension
   • Overview: Get an introduction to the extension’s three main parts: the content script, manifest file, and popup interface.
   • Content Script (content.js): Learn how to detect tweets, extract data (like tweet text, username, and link), and send that data to your server.
   • Manifest File (manifest.json): Understand how to configure your extension’s settings and permissions.
   • Popup Interface (popup.html & popup.js): See how to build a simple UI to enable or disable tweet capturing.
2. Part 2: Setting Up a FastAPI Server
   • Overview: Discover how to build a FastAPI server that listens for tweet data and saves it in a database.
   • Endpoint Creation: We’ll guide you in creating a POST endpoint to receive tweets.
   • Data Validation and Storage: Learn how to use Pydantic for data validation and SQLAlchemy for database storage.
3. Conclusion & Next Steps
   • Recap what you’ve built.
   • Explore ideas for expanding the tool.
   • Find additional resources to deepen your knowledge.

Step-by-Step Walkthrough

Part 1: Building the Browser Extension

Content Script (content.js):
This is the heart of your extension. It runs on Twitter pages, looks for tweet elements, extracts essential information (like tweet text, username, profile image, and tweet link), and sends this data to your FastAPI server. Here’s a simplified version with clear comments:

// Flag to indicate whether auto-save is enabled
let autoSaveEnabled = false;

// Get auto-save setting from Chrome storage
chrome.storage.sync.get("autoSaveEnabled", (data) => {
  autoSaveEnabled = data.autoSaveEnabled || false;
  if (autoSaveEnabled) {
    startAutoSave();
  }
});

// API endpoint to send tweets
const API_URL = "http://127.0.0.1:8000/tweets";

// Queue to hold tweet data temporarily
const tweetQueue = [];
// Set to track processed tweet IDs and prevent duplicates
const sentTweetIds = new Set();

// Function to extract visible tweets from the page
function getVisibleTweets() {
  const tweets = [];
  const tweetElements = document.querySelectorAll("article");
  
  tweetElements.forEach((tweet) => {
    try {
      const user = tweet.querySelector('[dir="ltr"] span')?.innerText || "Unknown";
      const text = tweet.querySelector('[data-testid="tweetText"]')?.innerText || "No text";
      const profileImage = tweet.querySelector("img")?.src || "No image";
      const tweetLinkElement = tweet.querySelector('a[role="link"][href*="/status/"]');
      const tweetLink = tweetLinkElement
        ? `https://twitter.com${tweetLinkElement.getAttribute("href")}`
        : "No link";
      let tweetId = tweetLinkElement
        ? tweetLinkElement.getAttribute("href").split("/status/")[1]
        : "No ID";
      tweetId = tweetId.split("/")[0];
      
      // Skip if tweet ID is invalid or already processed
      if (tweetId === "No ID" || sentTweetIds.has(tweetId)) return;
      
      const createdAt = new Date().toISOString();
      const tweetData = {
        tweet_id: tweetId,
        user: { username: user, followers: 0 },
        text: text,
        likes: 0,
        retweets: 0,
        views: 0,
        replies: 0,
        bookmarks: 0,
        link: tweetLink,
        profile_image: profileImage,
        created_at: createdAt,
        sent_by_user: "extension",
      };
      
      tweets.push(tweetData);
    } catch (error) {
      console.error("Error capturing tweet:", error);
    }
  });
  
  return tweets;
}

// Function to send queued tweets to the server
function sendTweetsFromQueue() {
  if (tweetQueue.length === 0) return;
  const tweetsToSend = [...tweetQueue];
  tweetQueue.length = 0;
  fetch(API_URL, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(tweetsToSend),
  })
    .then((response) => {
      if (!response.ok) throw new Error(`HTTP error! Status: ${response.status}`);
      return response.json();
    })
    .then((data) => {
      tweetsToSend.forEach((tweet) => sentTweetIds.add(tweet.tweet_id));
    })
    .catch((error) => {
      // If sending fails, re-add the tweets to the queue
      tweetQueue.push(...tweetsToSend);
    });
}

// Function to handle scroll events and capture tweets
function handleScroll() {
  const tweets = getVisibleTweets();
  tweets.forEach((tweet) => {
    if (!sentTweetIds.has(tweet.tweet_id)) {
      tweetQueue.push(tweet);
    }
  });
}

// Start auto-save: add scroll listener and send tweets periodically
function startAutoSave() {
  window.addEventListener("scroll", handleScroll);
  setInterval(() => { sendTweetsFromQueue(); }, 5000);
}

// Stop auto-save: remove the scroll listener
function stopAutoSave() {
  window.removeEventListener("scroll", handleScroll);
}

// Listen for messages from the popup to toggle auto-save
chrome.runtime.onMessage.addListener((request, sender, sendResponse) => {
  if (request.action === "toggleAutoSave") {
    autoSaveEnabled = request.enabled;
    if (autoSaveEnabled) startAutoSave();
    else stopAutoSave();
  }
  sendResponse({ status: "success" });
});

Manifest File (manifest.json):
This file tells Chrome everything about your extension—from its name and version to the permissions it needs and which scripts to run on which pages.

{
  "manifest_version": 3,
  "name": "Tweet Saver",
  "version": "1.0",
  "description": "Save tweets from your screen for free.",
  "permissions": ["activeTab", "scripting", "storage"],
  "host_permissions": ["https://twitter.com/*", "https://x.com/*"],
  "action": {
    "default_popup": "popup.html",
    "default_icon": "icon.png"
  },
  "content_scripts": [
    {
      "matches": ["https://twitter.com/*", "https://x.com/*"],
      "js": ["content.js"]
    }
  ],
  "background": {
    "service_worker": "background.js"
  }
}

Popup Interface (popup.html & popup.js):
The popup provides a simple UI for users to control the tweet-saving feature.

popup.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <title>Tweet Saver</title>
  <style>
    body { font-family: Arial, sans-serif; width: 220px; padding: 10px; }
    label { display: flex; align-items: center; justify-content: space-between; margin: 10px 0; }
    button { background-color: #1DA1F2; color: white; border: none; padding: 8px; cursor: pointer; width: 100%; }
  </style>
</head>
<body>
  <h2>Tweet Saver</h2>
  <label>
    <span>Auto-Save Tweets</span>
    <input type="checkbox" id="auto-save-switch">
  </label>
  <div id="status"></div>
  <script src="popup.js"></script>
</body>
</html>

popup.js:

// Get UI elements
const switchElement = document.getElementById("auto-save-switch");
const statusElement = document.getElementById("status");

// Load auto-save state from Chrome storage
chrome.storage.sync.get("autoSaveEnabled", (data) => {
  switchElement.checked = data.autoSaveEnabled || false;
  statusElement.textContent = `Auto-Save is ${data.autoSaveEnabled ? "ON" : "OFF"}`;
});

// Listen for changes on the switch
switchElement.addEventListener("change", () => {
  const isEnabled = switchElement.checked;
  chrome.storage.sync.set({ autoSaveEnabled: isEnabled }, () => {
    statusElement.textContent = `Auto-Save is ${isEnabled ? "ON" : "OFF"}`;
    console.log(`Auto-Save ${isEnabled ? "enabled" : "disabled"}`);
  });
  // Notify the content script
  chrome.tabs.query({ active: true, currentWindow: true }, (tabs) => {
    if (tabs[0]) {
      chrome.tabs.sendMessage(tabs[0].id, { action: "toggleAutoSave", enabled: isEnabled });
    }
  });
});

---

Part 2: Setting Up a FastAPI Server

Overview:
Next, you’ll build a FastAPI server that receives the tweet data from your extension and stores it in a database using SQLAlchemy. This server validates the data using Pydantic models, ensuring everything is in the right format before saving.

Key Steps:

• Create a POST Endpoint:
  The server listens for JSON data at /tweets.
• Validate Data:
  Use Pydantic models to ensure incoming tweet data is correct.
• Store Tweets:
  Save the validated tweets to your database with SQLAlchemy, and use a cache to prevent duplicate entries.

Here’s a simplified example of the FastAPI server:

import os
from datetime import datetime
from fastapi import FastAPI, HTTPException, Depends
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import List
from sqlalchemy.orm import Session
from dotenv import load_dotenv
from .models import Base, UserDB, TweetDB, engine, SessionLocal

load_dotenv()
app = FastAPI()

# Allow CORS for all origins (for development)
app.add_middleware(
  CORSMiddleware,
  allow_origins=["*"],
  allow_credentials=True,
  allow_methods=["*"],
  allow_headers=["*"],
)

def get_db():
  db = SessionLocal()
  try:
    yield db
  finally:
    db.close()

class User(BaseModel):
  username: str
  followers: int = 0

class Tweet(BaseModel):
  tweet_id: str
  text: str
  likes: int = 0
  retweets: int = 0
  views: int = 0
  replies: int = 0
  bookmarks: int = 0
  link: str
  profile_image: str = None
  created_at: datetime
  sent_by_user: str
  user: User

@app.post("/tweets", status_code=201)
def create_tweets(tweets: List[Tweet], db: Session = Depends(get_db)):
  try:
    new_tweets = []
    for tweet in tweets:
      new_tweet = TweetDB(
        tweet_id=tweet.tweet_id,
        text=tweet.text,
        likes=tweet.likes,
        retweets=tweet.retweets,
        views=tweet.views,
        replies=tweet.replies,
        bookmarks=tweet.bookmarks,
        link=tweet.link,
        profile_image=tweet.profile_image,
        created_at=tweet.created_at,
        sent_by_user=tweet.sent_by_user,
      )
      new_tweets.append(new_tweet)
    db.add_all(new_tweets)
    db.commit()
    return {"message": f"{len(new_tweets)} Tweets stored successfully."}
  except Exception as e:
    db.rollback()
    raise HTTPException(status_code=500, detail=str(e))

---

Conclusion

In this comprehensive guide, you learned how to build a free tweet collector without relying on Twitter’s paid API. We covered:

• How to build a browser extension that automatically detects and saves tweets.
• How to configure your extension using a manifest and popup interface.
• How to set up a FastAPI server to receive, validate, and store tweets in a database.
• Common questions and troubleshooting tips to help you along the way.

By following this step‑by‑step tutorial, you now have a powerful, cost‑effective tool to capture and analyze tweet data. Whether you’re a beginner or looking to expand your skills, this project is a fantastic way to dive into browser extensions and FastAPI.

Next Steps:

1. Check out the full repository on GitHub for detailed code and documentation.
2. Experiment with the code to add new features or improve the UI.
3. Share your project with the community and gather feedback.

Happy coding and enjoy building your own tweet collector!