🏗️ The Challenge

I needed a tool to scrape thousands of YouTube comments and analyze their sentiment (Positive/Negative) for market research.

Writing a basic Python script is easy. But making it production-ready is a different story. I faced three major hurdles:

  1. Dependency Hell: It worked on my machine but broke elsewhere.
  2. Geolocation Lock: YouTube returned data in Arabic (because of my local IP), but I needed English for NLP analysis.
  3. Bad UX: The scraping process was a “black box”—the user had no idea if it was working or stuck.

Here is how I engineered the solution using Docker and Streamlit.

🧩 1. The Architecture: “Code as Infrastructure”

Instead of running scripts manually, I containerized the entire environment. I used Docker Compose to orchestrate the Python backend and the Streamlit frontend.

One critical trick was using Bind Mounts for development. This allowed me to edit code in Geany (my lightweight editor of choice) and see changes instantly inside the container without rebuilding.

# docker-compose.yml snippet
services:
  tube-logic:
    image: tube-bot:v2
    volumes:
      - ./src:/app/src  <-- The Magic Link
    command: 
      - python3
      - -m
      - streamlit
      - run
      - src/dashboard.py

Note: I used the list format for the command to bypass shell entrypoint issues common in Debian images.

🕵️‍♂️ 2. The Hack: Forcing English Data

The biggest issue was YouTube localizing dates (e.g., “منذ 2 أيام” instead of “2 days ago”). My NLP library needed English.

Instead of writing a complex translation function, I went to the source. I reverse-engineered the browser request and found that injecting specific Cookies and Headers could force YouTube to serve US-localized content, regardless of my actual IP.

Here is the injection logic I implemented in the scraper engine:

Here is the injection logic I implemented in the scraper engine:

# scraper.py
downloader = YoutubeCommentDownloader()

# Injecting US Locale Credentials
downloader.session.cookies.set('PREF', 'hl=en&gl=US', domain='.youtube.com')
downloader.session.headers.update({
    'Accept-Language': 'en-US,en;q=0.9',
})

Result: Clean, English metadata directly from the source. Zero processing overhead.

⚡ 3. The UX: Real-Time Feedback Loop

A scraping job can take seconds or minutes. A frozen screen is bad UX.

I refactored the engine to accept a callback function. As the scraper fetches data in the background, it pings the frontend to update the progress bar in real-time.

# The Bridge between Backend and Frontend
def update_progress(percent, count):
    bar_text = f"🕷️ Scraped {count} / {limit} comments..."
    progress_bar.progress(percent, text=bar_text)

# Passing the callback
get_comments(url, progress_callback=update_progress)

🚀 The Final Product

The result is Tube-Logic: A dark-mode, enterprise-grade dashboard that: Runs anywhere via Docker. Bypasses locale restrictions. Provides instant visual feedback. Exports UTF-8-SIG compatible CSVs for Excel. This project reinforced my philosophy: “Don’t just write code; build systems.”

Built by Radi