Required dependencies

# Minimum: HTTP crawling
npm install crawlee

# Playwright browser crawling
npm install crawlee playwright

# Puppeteer browser crawling
npm install crawlee puppeteer

# For fingerprint injection (optional, peer dep of browser-pool)
npm install fingerprint-injector fingerprint-generator

No native build steps are required for HTTP-only usage. Playwright browsers must be installed separately:

npx playwright install
# or via the Crawlee CLI:
npx crawlee install-playwright-browsers

Project setup

1. Drop source: place the source/ directory at the root of your project (e.g., ./vendor/crawlee/). For a standard integration, install via npm instead and skip vendor steps.

2. TypeScript config: ensure moduleResolution is Node16 or Bundler and target is ES2020 or higher:

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "Node16",
    "moduleResolution": "Node16",
    "esModuleInterop": true,
    "strict": true
  }
}

3. Environment variables (optional but common):

CRAWLEE_STORAGE_DIR=./storage          # where datasets/queues are persisted
CRAWLEE_LOG_LEVEL=INFO                 # DEBUG | INFO | WARNING | ERROR
APIFY_TOKEN=your_token                 # only if deploying to Apify platform

4. Storage: by default Crawlee writes to ./storage. Import MemoryStorage from @crawlee/memory-storage to keep everything in-process for testing.

5. Entry point: use ESM or CommonJS; Crawlee ships dual builds. ESM is preferred for tree-shaking.

Public API

BasicCrawler

import { BasicCrawler, BasicCrawlerOptions } from '@crawlee/basic';

new BasicCrawler(options: BasicCrawlerOptions): BasicCrawler

The lowest-level crawler. Manages the request queue, concurrency, retries, and session rotation. Use it when you want full control over the HTTP layer or are wrapping a custom fetch function. All other crawlers extend this class.

BrowserPool

import { BrowserPool, PuppeteerPlugin, PlaywrightPlugin } from '@crawlee/browser-pool';

new BrowserPool(options: {
  browserPlugins: (PuppeteerPlugin | PlaywrightPlugin)[];
  maxOpenPagesPerBrowser?: number;
  retireBrowserAfterPageCount?: number;
}): BrowserPool

Manages a pool of browser instances across Puppeteer and/or Playwright. Use it when you need fine-grained control over browser lifecycle, fingerprinting hooks, or want to mix two browser engines. BrowserCrawler uses it internally.

PlaywrightPlugin

import { PlaywrightPlugin, BrowserPluginOptions } from '@crawlee/browser-pool';
import { chromium } from 'playwright';

new PlaywrightPlugin(launcher: typeof chromium, options?: BrowserPluginOptions): PlaywrightPlugin

Wraps a Playwright browser launcher for use inside BrowserPool. Accepts fingerprinting options and proxy configuration at the plugin level. Pass it to BrowserPool or let PlaywrightCrawler create it automatically.

LaunchContext

import { LaunchContext, LaunchContextOptions } from '@crawlee/browser-pool';

new LaunchContext(options: LaunchContextOptions): LaunchContext

Holds per-launch configuration (proxy, user-agent, browser arguments). Created by a plugin before each browser launch; mutate it inside preLaunchHooks to inject custom flags or credentials.

CheerioCrawler

import { CheerioCrawler } from '@crawlee/cheerio';
// also re-exported from 'crawlee'

HTTP crawler that parses each response with Cheerio. The request handler receives $ (a CheerioAPI instance) and body. Best for high-throughput scraping of server-rendered HTML where JavaScript execution is not required.

Working examples

Scenario - Scrape static HTML with CheerioCrawler

Fetch a list of URLs, extract data with Cheerio selectors, and save to the default dataset.

import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({
    maxRequestsPerCrawl: 50,
    async requestHandler({ $, request, enqueueLinks }) {
        const title = $('title').text();
        const url = request.loadedUrl ?? request.url;

        await Dataset.pushData({ title, url });

        // Follow internal links automatically
        await enqueueLinks({ globs: ['https://example.com/**'] });
    },
});

await crawler.run(['https://example.com']);

Scenario - Browser crawling with PlaywrightCrawler

Render JavaScript-heavy pages, take screenshots, and collect data.

import { PlaywrightCrawler, Dataset } from 'crawlee';

const crawler = new PlaywrightCrawler({
    headless: true,
    maxRequestsPerCrawl: 20,
    async requestHandler({ page, request, enqueueLinks, log }) {
        const title = await page.title();
        log.info(`Visiting: ${request.url} — "${title}"`);

        await Dataset.pushData({ url: request.url, title });
        await enqueueLinks({ globs: ['https://example.com/**'] });
    },
    async failedRequestHandler({ request, log }) {
        log.error(`Request failed: ${request.url}`);
    },
});

await crawler.run(['https://example.com']);

Scenario - Custom BrowserPool with mixed plugins

Use both Puppeteer and Playwright in the same pool for A/B testing or fallback logic.

import { BrowserPool, PuppeteerPlugin, PlaywrightPlugin } from '@crawlee/browser-pool';
import puppeteer from 'puppeteer';
import { chromium } from 'playwright';

const pool = new BrowserPool({
    browserPlugins: [
        new PuppeteerPlugin(puppeteer, { useFingerprints: true }),
        new PlaywrightPlugin(chromium, { useFingerprints: true }),
    ],
    maxOpenPagesPerBrowser: 5,
    retireBrowserAfterPageCount: 30,
});

const page = await pool.newPage();
await (page as any).goto('https://example.com');
const content = await (page as any).content();
console.log(content.slice(0, 200));
await pool.destroy();

Scenario - In-memory storage for unit tests

Replace file-system storage with MemoryStorage so tests run without touching disk.

import { BasicCrawler } from '@crawlee/basic';
import { MemoryStorage } from '@crawlee/memory-storage';
import { Configuration } from '@crawlee/core';

const storage = new MemoryStorage();
const config = new Configuration({ storageClient: storage });

const crawler = new BasicCrawler(
    {
        async requestHandler({ request, sendRequest }) {
            const response = await sendRequest();
            console.log(request.url, response.statusCode);
        },
    },
    config,
);

await crawler.run(['https://example.com']);

File walkthrough

• basic-crawler/ - Defines BasicCrawler and its options; re-exports all of @crawlee/core so callers only need one import.
• browser-crawler/ - Abstract BrowserCrawler sitting between BasicCrawler and concrete Playwright/Puppeteer crawlers; manages BrowserPool lifecycle.
• browser-pool/ - Self-contained browser-pool package: plugin abstractions, Playwright and Puppeteer concrete plugins, fingerprint injection hooks, proxy anonymization, and the pool orchestrator.
• cheerio-crawler/ - Thin package re-exporting @crawlee/http and adding CheerioCrawler; no browser dependency.
• cli/ - crawlee CLI built with yargs; commands: create (scaffold project), run (run project), install-playwright-browsers.
• core/ - Heart of the framework: RequestQueue, Dataset, KeyValueStore, Router, Session, EventManager, ProxyConfiguration.
• crawlee/ - Umbrella package; re-exports every sub-package so users can do import { ... } from 'crawlee'.
• http-crawler/ - HttpCrawler using a configurable HTTP client; parent of CheerioCrawler, JSDOMCrawler, etc.
• impit-client/ - HTTP client with TLS/JA3 fingerprint spoofing to evade bot detection at the TCP layer.
• jsdom-crawler/ - JSDOMCrawler; useful when a lightweight DOM API is needed without a real browser.
• linkedom-crawler/ - LinkedOMCrawler; fastest DOM-emulation crawler, good for simple HTML parsing.
• memory-storage/ - Drop-in storage implementations backed by plain objects; ideal for testing and serverless environments with no disk.
• playwright-crawler/ - PlaywrightCrawler; extends BrowserCrawler with Playwright-specific defaults and pre-navigation hooks.
• puppeteer-crawler/ - PuppeteerCrawler; extends BrowserCrawler with Puppeteer-specific launch options.
• stagehand-crawler/ - StagehandCrawler; experimental crawler that layers LLM-driven actions on top of Playwright pages.
• templates/ - Starter templates (TypeScript, JavaScript) cloned by crawlee create.
• types/ - Shared interfaces and enums used across multiple packages; not a runtime dependency.
• utils/ - Helpers: Cheerio type re-exports, URL utilities, logging wrappers, social URL parsing.

Common pitfalls and fixes

• Playwright browsers not found: running PlaywrightCrawler without installing browsers throws browserType.launch: Executable doesn't exist; fix with npx playwright install chromium.
• ESM/CJS mismatch: Crawlee ships both builds; if you get ERR_REQUIRE_ESM, set "type": "module" in package.json or use .mjs entry points.
• CRAWLEE_STORAGE_DIR not set in Docker: the default path is relative to process.cwd(); set it explicitly so storage persists to a mounted volume.
• Fingerprint peer deps missing: BrowserPool with useFingerprints: true requires fingerprint-injector and fingerprint-generator installed separately; omitting them silently disables fingerprinting without an error in some versions.
• Node.js version: Crawlee requires Node.js 16+; running on Node 14 produces cryptic SyntaxError for optional chaining; pin engines in your package.json.
• maxRequestsPerCrawl not resetting between runs: the request queue persists to disk by default; call await crawler.teardown() or point CRAWLEE_STORAGE_DIR to a temp directory between test runs.

Vibe-coding prompt template

I have the Crawlee monorepo source in `./source/` and a usage guide at `./USAGE.md`.
The upstream npm package is `crawlee` (and sub-packages like `@crawlee/playwright`,
`@crawlee/cheerio`, `@crawlee/browser-pool`, etc.).

Please help me integrate Crawlee into my existing Node.js/TypeScript project step by step:

1. Read `USAGE.md` and `source/` to understand the real exported symbols.
2. Install the required npm packages for my use case (specify: HTTP-only / Playwright / Puppeteer).
3. Create a `src/crawler.ts` file that:
   - Uses [CheerioCrawler / PlaywrightCrawler / PuppeteerCrawler — pick one] 
   - Starts from a seed URL I provide
   - Enqueues internal links matching a glob pattern I specify
   - Pushes { url, title } records to the default Dataset
   - Respects a `maxRequestsPerCrawl` limit I configure via an env var
4. Show me how to read the collected dataset back after the crawl.
5. Show me how to swap in MemoryStorage for unit tests.
6. Point out any tsconfig or ESM settings I need to change.

Seed URL: <YOUR_URL>
Link glob: <YOUR_GLOB>
Max requests: <NUMBER>

License and credit

Crawlee is released under the Apache 2.0 License (see source/LICENSE if present, or the GitHub repository). It is developed and maintained by Apify. Full documentation and guides are available at crawlee.dev. The upstream npm package is crawlee.

This product’s unique contribution is clean room implementation of valuable code block into AVCP compatible standard.

The full install guide and integration prompts unlock after purchase.