Docs
→
Getting Started
→
Configuration

Configuration

Complete configuration guide for HeadlessX settings

HeadlessX offers comprehensive configuration options via environment variables and the Dashboard.

Environment Variables

Core Variables

These are required in your .env file:

# Database Connection
DATABASE_URL="postgresql://user:password@localhost:5432/headlessx"

# Backend Server Configuration
PORT=3001
NODE_ENV=development
FRONTEND_URL="http://localhost:3000"

# Frontend API URL
NEXT_PUBLIC_API_URL="http://localhost:3001"

Optional Variables

# Logging
LOG_LEVEL=info          # debug | info | warn | error
LOG_DIR=./logs           # Directory for log files

# Profile Storage
PROFILE_STORAGE_PATH=./profiles  # Browser profile storage

# Server Configuration
MAX_PAYLOAD_SIZE=50mb    # Maximum request size
CORS_ORIGIN=*            # CORS allowed origins

Dashboard Settings

Most configuration is managed via the Dashboard Settings page at /settings.

Access Settings

  1. Open Dashboard: http://localhost:3000
  2. Navigate to Settings
  3. Modify configuration
  4. Click Save Changes

Changes take effect immediately without restarting.

General Configuration

Headless Mode

  • Type: Boolean
  • Default: true
  • Description: Run browser without UI (faster, uses less resources)
{
  "browserHeadless": true
}

When to disable:

  • Debugging scraping logic
  • Visual CAPTCHA solving
  • Development/testing

Browser Timeout

  • Type: Number (milliseconds)
  • Default: 60000 (60 seconds)
  • Range: 5000 - 300000
  • Description: Maximum time for page load and operations
{
  "browserTimeout": 60000
}

Recommendations:

  • Fast sites: 30000ms (30s)
  • Normal sites: 60000ms (60s)
  • Slow/heavy sites: 120000ms (120s)

Max Concurrent Jobs

  • Type: Number
  • Default: 5
  • Range: 1 - 50
  • Description: Maximum simultaneous scraping jobs
{
  "maxConcurrency": 5
}

Guidelines:

  • Low RAM (4GB): 3
  • Medium RAM (8GB): 5-8
  • High RAM (16GB+): 10-20

Browser Engine (Camoufox)

Block WebRTC

  • Type: Boolean
  • Default: true
  • Description: Prevent IP leaks via WebRTC
{
  "camoufoxBlockWebrtc": true
}

Always enable for maximum stealth.

Camoufox GeoIP

  • Type: Boolean
  • Default: true
  • Description: Spoof location based on proxy IP
{
  "camoufoxGeoip": true
}

Enable when: Using proxies to match target location

Enable Cache

  • Type: Boolean
  • Default: false
  • Description: Cache resources for faster subsequent loads
{
  "camoufoxEnableCache": false
}

Enable if: Scraping same sites repeatedly

Disable if: Each scrape must be fresh

Proxy Configuration

Using Proxies

Configure proxies in the Proxies section:

HTTP Proxy:

http://username:password@proxy-server:port

SOCKS5 Proxy:

socks5://username:password@proxy-server:port

No Authentication:

http://proxy-server:port

Proxy Rotation

Enable automatic proxy rotation:

  • Upload proxy list (one per line)
  • Set rotation strategy:
    • Round Robin: Cycle through all
    • Random: Pick randomly
    • Least Used: Use least-used first

Proxy Testing

Test proxies before use:

curl -X POST http://localhost:3001/api/proxy/test \
  -H "X-API-Key: your-key" \
  -d '{"proxy": "http://proxy:port"}'

Stealth Configuration

Fingerprint Spoofing

Automatically enabled with Camoufox:

  • ✅ Canvas fingerprinting blocked
  • ✅ WebGL fingerprinting spoofed
  • ✅ AudioContext randomized
  • ✅ Fonts randomized
  • ✅ Timezone matched to proxy location

User Agent Rotation

Configure user agent behavior:

{
  "userAgent": "random",  // "random" | "chrome" | "firefox" | "custom"
  "customUserAgent": "Mozilla/5.0..."  // if userAgent = "custom"
}

Options:

  • random: Random realistic user agent
  • chrome: Latest Chrome on Windows
  • firefox: Latest Firefox

Viewport Configuration

{
  "viewportWidth": 1920,
  "viewportHeight": 1080,
  "deviceScaleFactor": 1
}

Common presets:

  • Desktop: 1920x1080
  • Laptop: 1366x768
  • Tablet: 768x1024
  • Mobile: 375x667

Advanced Configuration

Database Configuration

Connection Pool

DATABASE_URL="postgresql://user:pass@localhost:5432/db?connection_limit=10&pool_timeout=20"

Parameters:

  • connection_limit: Max connections (default: 10)
  • pool_timeout: Timeout in seconds (default: 20)

SSL Configuration

DATABASE_URL="postgresql://user:pass@host:5432/db?sslmode=require"

SSL Modes:

  • disable: No SSL
  • require: SSL required
  • verify-full: Verify certificate

Performance Tuning

Memory Management

{
  "browserMemoryLimit": 2048,  // MB per browser instance
  "maxBrowserInstances": 5
}

Request Queue

{
  "queueMaxSize": 100,
  "queueConcurrency": 5,
  "queueTimeout": 300000
}

Security Configuration

API Key Settings

Configure in Dashboard → Settings → API Keys:

  • Key Expiration: 30/60/90 days or never
  • Rate Limits: Requests per hour
  • IP Whitelist: Restrict by IP

CORS Configuration

# Allow specific origins
CORS_ORIGIN="https://yourapp.com,https://admin.yourapp.com"

# Allow all (development only)
CORS_ORIGIN="*"

Profile Management

Browser Profiles

Profiles store:

  • Cookies
  • Local storage
  • Session storage
  • IndexedDB
  • Service workers

Create Profile

Dashboard → Profiles → Create New:

  • Name: Identifier
  • Persistent: Keep across restarts
  • Proxy: Assign specific proxy

Use Profile in API

curl -X POST http://localhost:3001/api/website/html \
  -H "X-API-Key: key" \
  -d '{
    "url": "https://example.com",
    "profileId": "profile-123"
  }'

Configuration via API

Get Configuration

curl -H "X-API-Key: key" \
  http://localhost:3001/api/config

Update Configuration

curl -X PUT http://localhost:3001/api/config \
  -H "X-API-Key: key" \
  -H "Content-Type: application/json" \
  -d '{
    "browserTimeout": 120000,
    "maxConcurrency": 10
  }'

Best Practices

Production Settings

{
  "browserHeadless": true,
  "browserTimeout": 60000,
  "maxConcurrency": 10,
  "camoufoxBlockWebrtc": true,
  "camoufoxGeoip": true,
  "camoufoxEnableCache": false
}

Development Settings

{
  "browserHeadless": false,  // Watch browser
  "browserTimeout": 120000,   // More time for debugging
  "maxConcurrency": 3,        // Less resource usage
  "LOG_LEVEL": "debug"
}

High-Volume Settings

{
  "maxConcurrency": 20,
  "browserTimeout": 30000,
  "camoufoxEnableCache": true,
  "queueMaxSize": 500
}

Troubleshooting

Configuration Not Applied

  1. Verify JSON syntax in dashboard
  2. Check API response for errors
  3. Restart backend if environment variables changed

Performance Issues

  1. Reduce maxConcurrency
  2. Increase browserTimeout
  3. Enable cache if appropriate
  4. Check system resources (RAM, CPU)

Detection Issues

  1. Ensure camoufoxBlockWebrtc: true
  2. Use residential proxies
  3. Enable camoufoxGeoip
  4. Randomize user agents

Next Steps