N8N Integration

Integrate HeadlessX into your N8N workflows for powerful, undetectable web scraping automation.

Quick Setup

---

1. Add HTTP Request Node

In your N8N workflow, add an HTTP Request node with these settings:

• Method: POST
• URL: http://your-headlessx-server:3001/api/website/html
• Authentication: Header Auth
  • Name: X-API-Key
  • Value: your-api-key

2. Configure Request Body

Set the body type to JSON and add:

{
  "url": "{{ $json.url }}",
  "stealth": true
}

Complete Workflow Examples

---

Example 1: Scrape Product List

Workflow: Trigger → HTTP Request → Parse JSON → Store in Database

{
  "nodes": [
    {
      "name": "Scrape Products",
      "type": "n8n-nodes-base.httpRequest",
      "parameters": {
        "url": "http://localhost:3001/api/website/content",
        "authentication": "headerAuth",
        "method": "POST",
        "sendBody": true,
        "bodyParameters": {
          "parameters": [
            {
              "name": "url",
              "value": "https://example-store.com/products"
            },
            {
              "name": "stealth",
              "value": true
            }
          ]
        }
      },
      "credentials": {
        "httpHeaderAuth": {
          "name": "HeadlessX API Key"
        }
      }
    }
  ]
}

Example 2: Monitor Website Changes

Workflow: Schedule → Scrape → Compare → Send Alert

1. Schedule Trigger

   • Run every 1 hour

2. HTTP Request

   {
     "url": "https://target-website.com",
     "stealth": true
   }
   

3. Compare with Previous

   • Use Function node to compare HTML

4. Send Notification

   • Email/Slack if changes detected

Example 3: Bulk URL Scraping

Workflow: Spreadsheet → Split → Scrape → Aggregate

{
  "nodes": [
    {
      "name": "Split URLs",
      "type": "n8n-nodes-base.splitInBatches",
      "parameters": {
        "batchSize": 5
      }
    },
    {
      "name": "Scrape Each URL",
      "type": "n8n-nodes-base.httpRequest",
      "parameters": {
        "url": "http://localhost:3001/api/website/html",
        "authentication": "headerAuth",
        "method": "POST",
        "sendBody": true,
        "bodyParameters": {
          "parameters": [
            {
              "name": "url",
              "value": "={{ $json.url }}"
            },
            {
              "name": "stealth",
              "value": true
            }
          ]
        }
      }
    }
  ]
}

HeadlessX N8N Node (Custom)

---

Installation

Install the custom HeadlessX node for simplified integration:

cd ~/.n8n/custom
git clone https://github.com/yourusername/n8n-nodes-headlessx
cd n8n-nodes-headlessx
npm install
npm run build

Restart N8N to load the custom node.

Node Configuration

The custom node provides a cleaner interface:

Credentials:

• API URL: http://localhost:3001
• API Key: your-api-key

Operations:

1. Scrape HTML
2. Scrape with JS
3. Get Content
4. Take Screenshot
5. Google Search

Fields:

• URL (required)
• Stealth Mode (toggle)
• Timeout (ms)
• Proxy (optional)
• Wait For (load/networkidle)

Example Usage

{
  "name": "HeadlessX Scraper",
  "type": "n8n-nodes-headlessx",
  "parameters": {
    "operation": "scrapeHtml",
    "url": "={{ $json.target_url }}",
    "stealth": true,
    "timeout": 60000
  },
  "credentials": {
    "headlessxApi": {
      "name": "My HeadlessX Instance"
    }
  }
}

Advanced Patterns

---

Pattern 1: Retry on Failure

Use Error Trigger to retry failed scrapes:

{
  "nodes": [
    {
      "name": "Scrape",
      "type": "n8n-nodes-base.httpRequest",
      "continueOnFail": true
    },
    {
      "name": "Check Error",
      "type": "n8n-nodes-base.if",
      "parameters": {
        "conditions": {
          "boolean": [
            {
              "value1": "={{ $node['Scrape'].json.success }}",
              "value2": false
            }
          ]
        }
      }
    },
    {
      "name": "Retry",
      "type": "n8n-nodes-base.executeWorkflow"
    }
  ]
}

Pattern 2: Parallel Scraping

Scrape multiple URLs concurrently:

{
  "nodes": [
    {
      "name": "URLs",
      "type": "n8n-nodes-base.set",
      "parameters": {
        "values": {
          "array": ["url1", "url2", "url3"]
        }
      }
    },
    {
      "name": "Parallel Scrape",
      "type": "n8n-nodes-base.splitInBatches",
      "parameters": {
        "batchSize": 3,
        "options": {
          "parallel": true
        }
      }
    }
  ]
}

Pattern 3: AI Content Extraction

Combine with OpenAI for intelligent data extraction:

{
  "nodes": [
    {
      "name": "Scrape Page",
      "type": "n8n-nodes-base.httpRequest",
      "parameters": {
        "url": "http://localhost:3001/api/website/content"
      }
    },
    {
      "name": "Extract with AI",
      "type": "n8n-nodes-base.openAi",
      "parameters": {
        "operation": "completion",
        "prompt": "Extract product name, price, and description from:\n{{ $json.data.markdown }}"
      }
    }
  ]
}

Common Use Cases

---

E-commerce Monitoring

Monitor competitor prices:

• Trigger: Schedule (daily)
• Scrape: Product pages
• Extract: Price, stock
• Store: Database
• Alert: If price changed

Content Aggregation

Aggregate blog posts:

• Trigger: RSS feed
• Scrape: Full articles
• Convert: To markdown
• AI: Summarize
• Publish: To your site

Lead Generation

Extract contact information:

• Trigger: New company added
• Scrape: Company website
• Extract: Email, phone
• Enrich: With AI
• Add: To CRM

Troubleshooting

---

Connection Refused

Ensure HeadlessX is accessible from N8N:

# Test connection
curl http://your-headlessx-server:3001/health

Timeout Errors

Increase timeout in HTTP Request node:

• Timeout: 120000 (2 minutes)

Authentication Errors

Verify API key:

• Dashboard → Settings → API Keys
• Copy exact key without spaces

Best Practices

---

1. Use Batching: Don't scrape 1000 URLs at once
2. Add Delays: Respect target servers
3. Error Handling: Always handle failures
4. Logging: Use N8N's built-in logging
5. Credentials: Store API keys in N8N credentials

Next Steps

---

• Example Workflows
• Custom Node Source
• API Reference