Firecrawl MCP Server

Firecrawl Capabilities

Comprehensive web scraping and content extraction for enterprise use

🌐 Web Scraping

JavaScript-rendered page support
Dynamic content extraction
Multi-page crawling and sitemaps
Rate limiting and respectful crawling

📄 Content Processing

AI-powered content extraction
Markdown and structured data output
Image and media file handling
Automatic content cleaning and formatting

🔍 Data Extraction

Custom CSS selector targeting
Schema-based data extraction
Structured JSON output formats
Metadata and SEO data capture

⚡ Enterprise Features

High-volume concurrent processing
Proxy rotation and IP management
Anti-bot detection avoidance
Real-time monitoring and alerts

Step-by-Step Setup

Follow these steps to set up Firecrawl for enterprise web data collection

Step 1: Install Firecrawl and MCP Server

Install Firecrawl and the MCP server on your development machine:

# Install Firecrawl CLI
npm install -g @mendable/firecrawl-js

# Install the Firecrawl MCP server
npm install -g @mcp/firecrawl-server

# Verify installation
firecrawl --version
mcp --version

Step 2: Get Firecrawl API Access (Contact DevOps Team)

Contact your DevOps team to provision Firecrawl API access. You'll need:

Firecrawl API Key (from your enterprise Firecrawl account)
Base URL (e.g., https://api.firecrawl.dev or your self-hosted instance)
Rate limits and usage quotas for your team
Allowed domains list for scraping permissions

Tell your DevOps team you need:

Enterprise Firecrawl API subscription
Appropriate rate limits for your use case
Domain whitelist configuration
Proxy setup for enterprise network access

Step 3: Configure Firecrawl Connection

Set up the connection using your API credentials:

# Set your Firecrawl credentials (replace with actual values from DevOps)
export FIRECRAWL_API_KEY="fc-your-api-key-here"
export FIRECRAWL_BASE_URL="https://api.firecrawl.dev"
export FIRECRAWL_MAX_CONCURRENT="5"

# Configure the Firecrawl MCP server
mcp config firecrawl \
  --api-key $FIRECRAWL_API_KEY \
  --base-url $FIRECRAWL_BASE_URL \
  --max-concurrent $FIRECRAWL_MAX_CONCURRENT \
  --respect-robots-txt true \
  --default-wait-time 2000

Step 4: Test Your Connection

Verify that Firecrawl is working correctly:

# Test the connection
mcp test firecrawl

# Test a simple scrape
firecrawl scrape "https://example.com" --format markdown

# If successful, you should see:
# ✅ Firecrawl connection successful
# ✅ API authentication verified
# ✅ Scraping permissions confirmed

Step 5: Configure Scraping Policies (Optional)

Set up scraping rules and content extraction policies:

# Configure scraping policies
mcp config firecrawl policies \
  --respect-robots-txt true \
  --delay-between-requests 1000 \
  --max-pages-per-crawl 100 \
  --timeout 30000

# Set up content extraction rules
mcp config firecrawl extraction \
  --include-html false \
  --include-markdown true \
  --include-links true \
  --remove-tags "script,style,nav,footer"

Usage Examples

Leverage Firecrawl for web data collection and content extraction

Method 1: Ask GitHub Copilot (Recommended)

In your IDE with GitHub Copilot, you can ask natural language questions:

Example questions you can ask Copilot:

"Scrape the latest blog posts from our competitor's website"
"Extract product information from this e-commerce page"
"Get all the documentation from this API website"
"Crawl this news site and extract article content"
"Find all the pricing information from these SaaS websites"
"Extract contact information from company directory pages"

Copilot will automatically use Firecrawl to scrape and extract the requested content!

Method 2: Direct MCP Commands

You can also use Firecrawl directly from your terminal:

Scrape a single page:

mcp query firecrawl "scrape https://example.com/blog and extract the main content as markdown"

Crawl multiple pages:

mcp query firecrawl "crawl https://docs.example.com starting from the documentation page"

Extract structured data:

mcp query firecrawl "extract product names and prices from https://store.example.com/products"

Search and scrape results:

mcp query firecrawl "search for 'API documentation' on site:example.com and scrape the results"

Advanced Data Extraction

# Scrape with custom selectors
mcp query firecrawl "
  scrape: 'https://news.example.com'
  extract: {
    title: 'h1.article-title',
    author: '.author-name',
    date: '.publish-date',
    content: '.article-content'
  }
  format: 'json'
"

# Batch scraping with rate limiting
mcp query firecrawl "
  scrape_batch: [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3'
  ]
  delay: 2000
  format: 'markdown'
"

# Monitor website changes
mcp query firecrawl "
  monitor: 'https://example.com/pricing'
  check_interval: '24h'
  notify_changes: true
"

Enterprise Use Cases

Common enterprise applications for web data extraction

📈 Market Intelligence

Competitor pricing and product monitoring
Industry news and trend analysis
Market research data collection
Customer sentiment analysis from reviews

📊 Business Intelligence

Lead generation from business directories
Contact information extraction
Company information aggregation
Job posting and hiring trend analysis

📚 Knowledge Management

Documentation aggregation from multiple sources
Technical content extraction and indexing
Research paper and publication collection
Internal knowledge base enrichment

🔍 Compliance & Monitoring

Regulatory website monitoring
Terms of service and policy tracking
Brand mention and reputation monitoring
Compliance documentation collection

Security & Compliance

Enterprise-grade security features for web scraping

🔐 Data Security

Encrypted data transmission and storage
Secure API key management
Data retention policy compliance
PII detection and redaction

🌐 Network Security

Proxy rotation and IP anonymization
VPN and enterprise network compatibility
Firewall and network policy compliance
Request origin masking and headers

📋 Compliance

GDPR and privacy regulation compliance
Robots.txt and website policy respect
Rate limiting and ethical scraping
Audit logging and compliance reporting

Evaluation Status

Current evaluation progress and next steps

✅ Completed Evaluation

Performance testing with enterprise workloads
Legal and compliance review for web scraping
Security assessment and data handling review
Cost analysis for different usage tiers

🔄 In Progress

Enterprise proxy integration testing
Custom domain whitelist configuration
Rate limiting optimization for enterprise use
Data pipeline integration with existing systems

📋 Next Steps

Pilot deployment with business intelligence team
Integration with data warehouse and analytics tools
Training and documentation for end users
Production deployment and monitoring setup

Enterprise Web Data Extraction