Changelog

All notable changes to HeadlessX will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

---

[2.0.4] - 2026-03-07

---

🖥️ UI & Dashboard UX Refactor

• Flatten Dashboard Chrome: Flattened the shared dashboard surface system by removing shadow-heavy styling, disabling blur and smooth transitions, and strengthening borders across cards, dialogs, lists, and controls.
• Normalize Layout Rhythm: Unified the dashboard shell around a single content width and flatter bordered surface system so page headers, cards, and scraper workspaces align on the same design rhythm.
• Unify Scraper Workspace UX: Aligned the Website Scraper workspace with the Google SERP workspace by matching the header shell, page framing, and configuration card structure. Converted output and config options into dropdown-style controls.
• Header Structure: Turned page headers into bordered section shells for a clearer screen structure.
• Logs Route Redesign: Added a full-width /logs request workspace with a denser table, compact filter toolbar, and an overlay log details dialog to prevent compressing the table view.
• Sidebar Adjustments: Kept Headless Mode enabled by default in settings, moved the collapse control into the header, and updated the primary logo text color to black and blue.

🛠️ Backend API Enhancements

• Automated Cloudflare Challenge Solver: Added automated Cloudflare turnstile bypass logic with simulated human mouse movements directly to the API, alongside retry logic and SSE progress rendering for the frontend.
• Markdown Improvements: Integrated Mozilla Readability & GitHub Flavored Markdown for improved markdown extraction, along with expanding/collapsing results panel UI reliability enhancements.
• Documentation: Added demo GIF and Cloudflare bypass proof images to the README.

🔗 Related GitHub Commits

• refactor(web): flatten dashboard chrome and unify scraper workspace UX (@saifyxpro)
• refactor(web): normalize dashboard layout and improve logs workspace UX (@saifyxpro)
• feat(api): add automated cloudflare challenge solver (@saifyxpro)
• feat(api): enhance cloudflare challenge solver with retry and SSE progress (@saifyxpro)
• fix(website-scraper): improve markdown extraction and results panel expand button reliability (@saifyxpro)
• chore(web): update version to v2.0.4 (@saifyxpro)
• style(web): update primary logo text color to black and blue (@saifyxpro)
• docs(readme): add demo gif and cloudflare bypass proof images (@saifyxpro)

[2.0.3] - 2026-03-07

---

🔒 Security

• Dashboard Auth Hardening: Removed the public dashboard-internal fallback key and localhost trust path from the backend auth layer.
• Protected Admin Routes: Added API key protection to dashboard, profile, proxy, and job management endpoints that were previously exposed without auth.
• Secret Storage Improvements:
  • API keys are now stored as hashes instead of plaintext.
  • Stored proxy and profile passwords are now encrypted at rest.
  • Dashboard and API responses now redact credential values instead of returning raw secrets.
• Server-Side Dashboard Proxy: The dashboard now forwards /api/* requests through a Next.js server route so browser clients no longer send reusable internal keys.

🛠️ Reliability & Behavior

• Cloudflare Handling: Website scraping now fails fast with explicit Cloudflare challenge metadata instead of returning challenge pages as successful output.
• REST/SSE Output Parity: Aligned html-js behavior between the REST API and streaming endpoint to reduce timing drift on dynamic pages like Shopee.
• DOM Stability Waiting: Added a DOM stability wait before HTML extraction and screenshot capture so selector matches do not return partially rendered documents.
• Streaming Job Cancellation: Job cancellation now interrupts active scraping work instead of only updating job status in the UI.
• PDF Capability Cleanup: Removed unsupported PDF scraping/export paths from the API contract and dashboard UI.

🖥️ Developer Experience

• Fresh Install Fixes: Fixed a frontend startup failure caused by an invalid .npmrc encoding and resolved web/API port conflicts during local development.
• Docker Fix: Corrected the web Dockerfile path mismatch so Compose builds now target the expected file.
• Security Environment Variables: Added and documented required DASHBOARD_INTERNAL_API_KEY and CREDENTIAL_ENCRYPTION_KEY values for secure local and Docker deployments.
• Browser Profile Stability: Fixed Firefox preference merging so timezone overrides no longer wipe other stealth-related preferences.

📚 Documentation & UI

• Documentation Navigation: Updated dashboard and docs links to point to the introduction page and removed obsolete sidebar references.
• Landing Page Messaging: Removed outdated PDF export claims so the marketing copy matches the current product surface.
• Changelog Coverage: This release now links the supporting GitHub issues and commit history directly for easier auditability.

🐛 GitHub Issues

• #34 Can't run Headless:
  • Fixed fresh-install startup failures in the frontend dev workflow.
  • Resolved .npmrc encoding problems and port collisions between web and API processes.
• #32 Frontend response is different from backend:
  • Fixed REST and streaming output drift for JS-rendered pages.
  • Added DOM stabilization so API responses now match dashboard-rendered results more closely.

🔗 Related GitHub Commits

• a9e2367 chore: remove obsolete files and update documentation links in Sidebar component
• e1b91d9 feat: update docs link to point to introduction
• 7eb4a45 feat(api,web): improve scrape reliability and remove unsupported PDF flow
• 27bd3e3 fix(web): resolve dev startup failures on fresh installs
• 0abfb20 fix(api): align REST and SSE HTML rendering for dynamic pages
• 3c70c7d fix(security): harden dashboard auth and secret handling

[2.0.2] - 2026-02-23

---

🔧 Chore & Infrastructure

• Monorepo Restructure: Updated project structure to modern standards (apps/web, apps/api)
• Docker Support: Added comprehensive Docker infrastructure (infra/docker) with docker-compose.yml for easy deployment
• Legacy Cleanup: Removed deprecated OS-specific scripts (install.sh, start.bat, etc.) in favor of mise/docker
• GitHub Templates: Updated issue and PR templates for the new structure
• Documentation: Added docs/docker_setup.md and updated README.md with new port (8000) and deployment instructions

[2.0.1] - 2026-01-31 ⚡ PERFORMANCE FIX

---

🚀 Performance Improvements

Major performance optimizations for profile-based scraping with stealth mode disabled.

✨ What's Fixed

• Profile Context Reuse: Profile browser contexts now stay alive between requests
• Page Reuse: Existing pages are reused instead of creating new ones (~15s saved)
• Stealth Toggle Pipeline: stealth: false now properly skips all humanize delays
• Streaming Endpoint: Fixed stealth parameter passing from frontend to backend

🔧 How to Get Maximum Speed

1. Create & Launch Profile: Go to Profiles → Create → Launch
2. Disable Stealth Mode: In Advanced Config, toggle OFF "Stealth Mode"
3. API Usage: Pass "options": { "stealth": false } for fastest scraping

Note: First request after launching a profile takes ~15-20s (Camoufox initialization). Subsequent requests are under 3 seconds.

[2.0.0] - 2026-01-28 🦊 CAMOUFOX RELEASE

---

🎉 Major Release: Complete Browser Engine Rewrite

HeadlessX V2.0 represents a complete rewrite of the browser engine, achieving 0% detection rate across all major anti-bot systems by replacing Chromium-based stealth with Camoufox (Firefox with C++ level fingerprint spoofing).

✨ Added

Browser Engine

• Camoufox Browser Engine - Firefox with C++ level fingerprint spoofing
• 0% Detection Rate - Passes CreepJS, Sannysoft, BrowserScan, and all major bot detection
• WebRTC Protection - Built-in IP leak prevention
• Binary-Level Stealth - Canvas, WebGL, and AudioContext spoofing at C++ level
• Persistent Context Pooling - 3x faster browser launches

New Scrapers

• Google SERP Scraper - Extract search results with zero detection
  • Organic results with position tracking
  • Featured snippets extraction
  • People Also Ask questions
  • Related searches
• Website Scraper - Full-featured web scraping
  • Raw HTML extraction (fast)
  • JavaScript-rendered content
  • Markdown conversion for LLMs
  • Full-page screenshots (PNG/JPEG)
  • Real-time SSE streaming

API Endpoints

• /api/website/html - Fast HTML extraction without JavaScript
• /api/website/html-js - JavaScript rendering with wait options
• /api/website/content - Clean markdown conversion
• /api/website/screenshot - Screenshot capture with quality control
• /api/website/stream - Real-time SSE stream with progress
• /api/google-serp/search - Google SERP extraction
• /api/google-serp/stream - Real-time SERP streaming

Documentation

• Complete API Reference - All endpoints documented with curl examples
• N8N Integration Guide - Workflow automation with examples
• Zapier Integration Guide - No-code automation patterns
• Make.com Integration Guide - Visual automation scenarios
• Configuration Guide - Dashboard and environment settings
• Installation Guide - Platform-specific setup (Windows, Linux, macOS)

Dashboard Features

• Modern UI - Next.js 16 with React 19 and Turbopack
• Live Configuration - Change settings without restart via Dashboard
• Request Logs - Full history with screenshots and timing
• Playground - Real-time testing interface for both scrapers
• Profile Management - Persistent browser sessions and cookies
• Proxy Management - Automatic rotation and validation

🚀 Improved

• Performance - 3x faster browser initialization with context pooling
• Stealth - From 67% detection (V1.3) to 0% detection (V2.0)
• Reliability - Better error handling and automatic recovery
• Documentation - Comprehensive guides with visual examples
• Developer Experience - Simplified installation with one-command setup

🗑️ Removed

• playwright-extra - No longer needed with Camoufox native stealth
• puppeteer-extra-plugin-stealth - Replaced by C++ level patches
• PDF endpoint (/api/website/pdf) - Removed for optimization
• Generic component previews from docs - Focused on documentation clarity
• All JavaScript-based fingerprint protection - Replaced by binary-level

🔧 Changed

• Browser Engine - Migrated from Chromium to Firefox (Camoufox)
• Config Management - Moved most settings from .env to Dashboard UI
• Frontend - Upgraded to Next.js 16 with Turbopack and React 19
• TypeScript - Upgraded to 5.9+ with stricter type checking
• Node.js - Now requires Node.js 22+ for optimal performance

🐛 Fixed

• Hydration mismatch in mobile navigation components
• Invalid SVG attributes (fill-rule → fillRule) in icon components
• Component preview build errors after library removal
• Missing module imports after Badtz-UI refactoring
• File system access errors in component library stubs
• Emoji in heading IDs causing invalid CSS selectors

📦 Dependencies

Added:

• camoufox - Stealth browser engine with Firefox
• playwright 1.58+ - Browser automation framework

Updated:

• next → 16.0.7 (with Turbopack)
• react → 19.x
• typescript → 5.9+
• node → 22+
• pnpm → 9+

Removed:

• playwright-extra and all stealth plugins
• puppeteer-extra-plugin-stealth
• Legacy fingerprinting libraries

🚨 Breaking Changes

1. Browser Engine: Chromium → Firefox (Camoufox)
2. API Response Format: All responses wrapped in { success, data, error } structure
3. PDF Endpoint Removed: /api/website/pdf no longer available
4. Configuration: Most settings moved from .env to Dashboard UI
5. Node.js Version: Requires Node.js 22+ (was 18+)

🔄 Migration Guide from V1.3

1. Update Dependencies

2. Download Camoufox

3. Update Environment Variables

4. Update API Integrations

5. Remove PDF Endpoint Usage

No Database Migration Required:

• Database schema unchanged
• Existing profiles remain compatible
• API keys remain valid

---

[1.3.0] - 2025-09-23 🛡️ ADVANCED ANTI-DETECTION & SECURITY RELEASE

---

🛡️ Major Security & Privacy Features

• Advanced Fingerprinting Protection: Comprehensive canvas, WebGL, and audio fingerprinting mitigation
• Behavioral Simulation Engine: Human-like mouse movement with Bezier curves and natural timing
• Hardware Emulation: Realistic device profiling with consistent hardware characteristics
• WAF Bypass Capabilities: Cloudflare and DataDome evasion with TLS fingerprint masking
• Enhanced Authentication: Multi-token support with admin and profile management tokens
• Comprehensive Security Audit: Full codebase security review with vulnerability fixes

🚀 Anti-Detection Technologies

Fingerprinting Protection

• Canvas Noise Injection: Dynamic noise with consistent seeding for reproducible fingerprints
• WebGL Spoofing: GPU vendor/renderer masking with hardware-specific profiles
• Audio Fingerprint Control: Hardware audio database with realistic device simulation
• WebRTC Leak Protection: ICE candidate filtering and media device enumeration control
• Hardware Noise: CPU timing, memory allocation, and performance API manipulation
• Timezone Intelligence: Automatic timezone alignment with IP geolocation

Behavioral Simulation

• Natural Mouse Movement: Bezier curve path generation with acceleration modeling
• Keyboard Dynamics: Dwell time randomization and typing rhythm simulation
• Scroll Patterns: Natural scroll behavior with reader/scanner/browser profiles
• Attention Modeling: User attention simulation with realistic interaction patterns
• Micro-movements: Subtle mouse adjustments and human-like timing variations

WAF & Bot Detection Bypass

• Cloudflare Bypass: Challenge solver with TLS fingerprint masking
• DataDome Evasion: Resource blocking and behavioral pattern bypasses
• Generic WAF Bypass: Signature detection and response analysis
• Detection Monitoring: Real-time tracking of bot detection encounters

🏗️ Enhanced Architecture

• Modular Anti-Detection Services: Organized fingerprinting, behavioral, and evasion modules
• Profile Management System: Device profile creation, validation, and rotation
• Testing Framework: Comprehensive anti-detection testing against major services
• Development Tools: Interactive fingerprint testing and profile benchmarking
• Performance Monitoring: Real-time success rate analytics and optimization

📊 New API Endpoints

• POST /api/render/stealth - Maximum stealth rendering with all anti-detection features
• GET /api/test-fingerprint - Interactive fingerprinting effectiveness testing
• POST /api/profiles - Device profile management and validation
• GET /api/analytics/detection-rate - Real-time detection rate monitoring
• POST /api/test/cloudflare - Cloudflare bypass testing and validation

🔧 Configuration Enhancements

• Expanded Environment Variables: 50+ new configuration options for anti-detection
• Profile Configuration: Custom device profiles with hardware specifications
• Stealth Mode Settings: Basic, advanced, and maximum stealth levels
• Behavioral Tuning: Configurable human behavior simulation parameters
• Monitoring Controls: Audit trails, performance tracking, and detection analytics

🔒 Security Improvements

• Authentication Timing Attack Fix: Secure token comparison using crypto.timingSafeEqual
• Log Sanitization: Token exposure prevention in application logs
• Input Validation: SSRF protection and comprehensive URL validation
• Request Throttling: Token-based and IP-based request throttling
• Security Headers: CSP, HSTS, and anti-clickjacking protection

🚨 Breaking Changes

• New Required Environment Variables: FINGERPRINT_PROFILE, STEALTH_MODE
• API Response Changes: Enhanced error responses with security context
• Browser Configuration: New profile-based browser initialization

🐛 Bug Fixes

• Fixed browser resource leaks in concurrent operations
• Resolved timing inconsistencies in behavioral simulation
• Corrected WebGL context isolation issues
• Fixed audio fingerprint noise distribution
• Resolved profile validation edge cases

📈 Performance Improvements

• 30% faster rendering with optimized browser pooling
• 50% reduction in memory usage through enhanced cleanup
• 90% improvement in detection evasion success rates
• Real-time monitoring with minimal performance impact

---

[1.2.0] - 2025-09-15 🏗️ MAJOR MODULAR ARCHITECTURE REFACTOR

---

🚀 Revolutionary Changes

• Complete Modular Refactor: Transformed 3079-line monolithic server.js into 20+ focused modules
• Separation of Concerns: Clean architecture with distinct layers for configuration, services, controllers, and middleware
• Enhanced Maintainability: Each module has a single responsibility for better code organization
• Production-Ready: Enterprise-grade error handling, logging, and monitoring capabilities
• Developer Experience: Improved development workflow with clear module boundaries

🏗️ New Modular Architecture

✨ Major Features Added

• Enhanced Error Handling: Structured error responses with correlation IDs for debugging
• Advanced Request Throttling: Intelligent request throttling with memory-based storage and cleanup
• Improved Logging: Structured logging with request correlation and detailed context
• Better Browser Management: Optimized browser lifecycle with resource monitoring
• Security Enhancements: Improved authentication middleware and request validation
• Performance Optimization: Better resource utilization and memory management

🔧 Configuration Updates

• Environment Variables: TOKEN → AUTH_TOKEN (breaking change)
• PM2 Configuration: Moved from config/ecosystem.config.js to root ecosystem.config.js
• Enhanced .env: More configuration options with validation and defaults
• Docker Optimization: Updated Docker configuration for modular structure

💥 Breaking Changes

• Environment Variable: TOKEN renamed to AUTH_TOKEN
• File Structure: PM2 configuration moved from config/ to root directory
• Import Paths: Internal imports updated for modular structure
• Script Updates: Setup scripts updated to work with new architecture

🔄 Migration Guide

---

[1.1.0] - 2024-12-19 🌐 UNIFIED ARCHITECTURE RELEASE

---

🚀 Major Features Added

• Unified Architecture: Single Node.js server now serves both website and API
• Integrated Website: Complete Next.js website served at root path (/)
• Enhanced API: All API endpoints now available under /api/* prefix
• Environment Variables: Complete .env file support for all configurations
• Domain Integration: Automatic subdomain and domain configuration from environment

🌐 Website Integration

• Next.js Website: Modern React-based website with Tailwind CSS
• API Documentation: Interactive documentation and examples
• Live Testing: Built-in API testing interface
• Responsive Design: Mobile-first design with dark/light theme support
• TypeScript Support: Full TypeScript integration for better development

⚙️ Infrastructure Improvements

• Simplified Nginx: Single proxy configuration for all routes
• Unified Server: Website and API served from same Node.js process
• Better Routing: Intelligent routing between static files and API endpoints
• Performance: Improved caching and static file serving
• Security: Enhanced security headers and token validation

🐳 Docker & Deployment

• Docker Support: Multi-stage build with optimized containers
• Docker Compose: Complete stack deployment with one command
• PM2 Integration: Production process management
• SSL Support: Ready for Let's Encrypt certificates
• Health Checks: Automatic service monitoring

💔 Breaking Changes

• API endpoints moved from root to /api/* prefix
• Configuration now requires .env file setup
• Nginx configuration changed to proxy-only
• Docker deployment process updated

🔄 Migration Guide

---

[1.0.0] - 2024-12-01 🎉 INITIAL RELEASE

---

Core API

• Complete web scraping API with Playwright
• Screenshot generation with high quality
• PDF generation from webpages
• HTML extraction (clean and raw)
• Text content extraction
• Batch processing for multiple URLs

🌐 API Endpoints

• GET /health - Health check endpoint
• GET /status - Server status with authentication
• POST /render - Full page rendering with options
• GET /html - HTML content extraction
• GET /content - Text content extraction
• GET /screenshot - Screenshot generation
• GET /pdf - PDF generation
• POST /batch - Batch URL processing

🔧 Features

• Playwright Integration: Chrome, Firefox, Safari browser support
• Human Behavior: Realistic scrolling, mouse movements, typing
• Responsive Design: Mobile and desktop viewport simulation
• Custom Headers: Support for authentication and custom headers
• Proxy Support: Route requests through proxy servers
• Timeout Handling: Configurable request timeouts
• Error Handling: Comprehensive error responses

🚀 Deployment

• PM2 Support: Production process management
• Nginx Configuration: Reverse proxy setup
• Docker Support: Container deployment
• Environment Configuration: Flexible environment setup

---

Development Roadmap

---

🔮 Planned Features (V2.1 - Q1 2026)

• Docker Support - One-command deployment with Docker Compose
• Amazon Scraper - Product listings and reviews extraction
• LinkedIn Scraper - Job listings and profile data
• Twitter Scraper - Tweets, trends, and profile information
• Instagram Scraper - Posts, stories, and profile data

🎯 Future Enhancements (V2.2 - Q2 2026)

• GraphQL API - Alternative GraphQL interface alongside REST
• Python SDK - Official Python client library
• Node.js SDK - Official Node.js client library
• Bulk Scraping - Optimized concurrent URL processing
• Advanced Scheduling - Cron-based scheduled jobs

🚀 Major Features (V3.0 - Q3 2026)

• AI-Powered Extraction - LLM-based intelligent data parsing
• Visual Testing - Screenshot comparison and diff detection
• CAPTCHA Auto-Solving - Built-in AI-powered CAPTCHA solver
• Cloud Deployment - Managed hosting with auto-scaling

---

Contributing

---

We welcome contributions! Please see our Contributing Guide for details on how to:

• 🐛 Report bugs
• 💡 Suggest features
• 🔀 Submit pull requests
• 📖 Improve documentation
• 🧪 Add tests

Security

---

If you discover a security vulnerability, please read our Security Policy or email security issues directly. All security vulnerabilities will be promptly addressed.

Support

---

• GitHub Issues: https://github.com/saifyxpro/HeadlessX/issues
• Discussions: https://github.com/saifyxpro/HeadlessX/discussions
• Documentation: https://headlessx.saify.me/docs/introduction

License

---

This project is licensed under the MIT License.

Acknowledgments

---

• Camoufox Team - For the revolutionary stealth browser engine
• Playwright Team - For the excellent browser automation framework
• Next.js Team - For the amazing React framework
• Community Contributors - For suggestions, bug reports, and improvements

---

HeadlessX v2.0.0 - The world's most advanced anti-detection scraping platform 🦊