Crawl

Crawl a site and all it's links.

Install Hyperbrowser

npm install @hyperbrowser/sdk dotenv

yarn add @hyperbrowser/sdk dotenv

pip install hyperbrowser python-dotenv

uv add hyperbrowser python-dotenv

Setup your Environment

To use Hyperbrowser with your code, you will need an API Key. You can get one easily from the dashboard. Once you have your API Key, add it to your .env file as HYPERBROWSER_API_KEY .

Crawl a Site

Next, you can crawl any site by simply setting up the Hyperbrowser client and providing the site's url.

import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const crawlResult = await client.crawl.startAndWait({
    url: "https://hyperbrowser.ai",
    maxPages: 5,
  });
  console.log("Crawl result:", crawlResult);
};

main();

import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import StartCrawlJobParams

# Load environment variables from .env file
load_dotenv()

# Initialize Hyperbrowser client
client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))


def main():
    # Start crawling and wait for completion
    crawl_result = client.crawl.start_and_wait(
        StartCrawlJobParams(url="https://hyperbrowser.ai", max_pages=5)
    )
    print("Crawl result:\n", crawl_result.model_dump_json(indent=2))


main()

View Crawl in Dashboard

You can view all your crawls in the dashboard and see all their related information.

To view more details check out the Crawl page.

PreviousScrape NextExtract

Last updated 1 month ago