firecrawl-crawl

$npx mdskill add aAAaqwq/AGI-Super-Team/firecrawl-crawl

Crawl entire websites to extract bulk content from multiple pages.

  • Extracts full site sections or follows links up to a depth limit.
  • Integrates with firecrawl CLI via Bash and npx execution.
  • Uses path filters and depth limits to control extraction scope.
  • Outputs structured JSON data to the specified output directory.

SKILL.md

.github/skills/firecrawl-crawlView on GitHub ↗
---
name: firecrawl-crawl
description: |
  Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.
allowed-tools:
  - Bash(firecrawl *)
  - Bash(npx firecrawl *)
---

# firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

## When to use

- You need content from many pages on a site (e.g., all `/docs/`)
- You want to extract an entire site section
- Step 4 in the [workflow escalation pattern](firecrawl-cli): search → scrape → map → **crawl** → interact

## Quick start

```bash
# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>
```

## Options

| Option                    | Description                                 |
| ------------------------- | ------------------------------------------- |
| `--wait`                  | Wait for crawl to complete before returning |
| `--progress`              | Show progress while waiting                 |
| `--limit <n>`             | Max pages to crawl                          |
| `--max-depth <n>`         | Max link depth to follow                    |
| `--include-paths <paths>` | Only crawl URLs matching these paths        |
| `--exclude-paths <paths>` | Skip URLs matching these paths              |
| `--delay <ms>`            | Delay between requests                      |
| `--max-concurrency <n>`   | Max parallel crawl workers                  |
| `--pretty`                | Pretty print JSON output                    |
| `-o, --output <path>`     | Output file path                            |

## Tips

- Always use `--wait` when you need the results immediately. Without it, crawl returns a job ID for async polling.
- Use `--include-paths` to scope the crawl — don't crawl an entire site when you only need one section.
- Crawl consumes credits per page. Check `firecrawl credit-usage` before large crawls.

## See also

- [firecrawl-scrape](../firecrawl-scrape/SKILL.md) — scrape individual pages
- [firecrawl-map](../firecrawl-map/SKILL.md) — discover URLs before deciding to crawl
- [firecrawl-download](../firecrawl-download/SKILL.md) — download site to local files (uses map + scrape)

More from aAAaqwq/AGI-Super-Team

SkillDescription
a-fund-monitor监控 A 股基金实时估值与盘后净值，自动判断交易日并生成提醒或分析。
add-leadAdd company/person/relationship to CRM
ads-agentAI-агент для управления Facebook рекламой. Вызывай для анализа, оптимизации, создания кампаний и отчётов.
afrexai-compliance-auditRun internal compliance audits against major governance and security
afrexai-personal-financeComplete personal finance system — budgeting, debt payoff, investing, tax optimization, net worth tracking, and financial independence planning. Use when managing money, building wealth, paying off debt, planning retirement, or optimizing taxes. Zero dependencies.
after-salesUse when managing post-purchase experience, building customer loyalty, or increasing repeat purchases
agent-contactsAI agent contacts — add, list, remove MCP contacts. Use when someone gives an agent URL, or when you need to view/remove contacts.
agent-model-switcher批量查看和切换子 agent 的模型配置，用于统一调整多 agent 的 provider/model 设置。
agent-patternsFormat SPAWN REQUEST messages to launch parallel agents, generate structured agent status reports, and define communication protocols within the sprint system. Use when the user needs to coordinate multiple agents, format spawn requests, produce agent reports, or establish inter-agent communication patterns.
agent-task-confirmUse when confirming whether a dispatched agent task was actually received, activated, and progressing after sessions_send or other task handoff actions.