Skip to content
Forjed Crawler · Volunteer

Run a crawler, contribute to the index.

You can contribute to the index by hosting your own crawler (Forjed Crawler Worker) on Linux, MacOS, or Windows WSL

  • Linux
  • macOS
  • Windows (WSL)

How it works

Three steps, then it runs by itself.

01

Run one command

The script downloads, signs your worker up with Forjed and installs what it needs.

02

It crawls for Forjed

Configure your crawler by setting limits on pages crawled per second, bandwidth consumption, and other crawling parameters.

03

It's start crawling

Congrats! You are now contributing on growing the Forjed Crawler index!

Get started

Start in one command.

Copy a command and paste it into a terminal on the computer or server you want to use.

  1. 1

    Download and start

    The first run signs up with Forjed, installs what it needs and starts crawling.

    curl -fsSLo forjed-worker.sh https://crawler.forjed.xyz/worker.sh
    sh forjed-worker.sh
  2. 2

    Give it a name

    Optional, and only on the first run: the name your worker shows under.

    sh forjed-worker.sh --name office-pc
  3. 3

    Keep it running on a server

    Leave it crawling after you log out. Its output goes to a log file.

    nohup sh forjed-worker.sh > forjed-worker.log 2>&1 &
  4. 4

    Stop and carry on

    Ctrl+C stops it. Run the same command again to carry on; newer versions download by themselves.

    sh forjed-worker.sh

Your limits

Decide how much it uses.

Each command saves the setting, and a running worker picks it up within seconds. Downloads and uploads both count.

sh forjed-worker.sh limit 50GB
At most 50 GB per month.
sh forjed-worker.sh limit 200GB --period week
At most 200 GB per week.
sh forjed-worker.sh rate 20Mbit/s
At most 20 Mbit/s.
sh forjed-worker.sh pace on
Spread the allowance evenly over the period.
sh forjed-worker.sh concurrency 4
Pages at the same time, from 1 to 16 (2 by default).
sh forjed-worker.sh status
Your settings and the data used so far.
sh forjed-worker.sh forget
Forget the server and token; the next run signs up again.

Good to know

Before you start.

Does it respect websites?

Always. Workers respect robots.txt, whatever else is set.

What if Forjed pauses crawling?

Your worker finishes the page it is on, then waits, checking every 30 seconds. It carries on by itself when crawling resumes.

Where are its settings kept?

In ~/.forjed-worker on your computer, or in the folder named by FORJED_WORKER_HOME.

Were you given a token?

Start the worker with --token followed by your token, instead of signing up.

Can I run it on Windows?

Yes, inside WSL (Windows Subsystem for Linux).

Questions before you start?

Write to us, or read more about the crawler.