
Run a crawler, contribute to the index.
You can contribute to the index by hosting your own crawler (Forjed Crawler Worker) on Linux, MacOS, or Windows WSL
- Linux
- macOS
- Windows (WSL)
How it works
Three steps, then it runs by itself.
Run one command
The script downloads, signs your worker up with Forjed and installs what it needs.
It crawls for Forjed
Configure your crawler by setting limits on pages crawled per second, bandwidth consumption, and other crawling parameters.
It's start crawling
Congrats! You are now contributing on growing the Forjed Crawler index!
Get started
Start in one command.
Copy a command and paste it into a terminal on the computer or server you want to use.
- 1
Download and start
The first run signs up with Forjed, installs what it needs and starts crawling.
curl -fsSLo forjed-worker.sh https://crawler.forjed.xyz/worker.sh sh forjed-worker.sh - 2
Give it a name
Optional, and only on the first run: the name your worker shows under.
sh forjed-worker.sh --name office-pc - 3
Keep it running on a server
Leave it crawling after you log out. Its output goes to a log file.
nohup sh forjed-worker.sh > forjed-worker.log 2>&1 & - 4
Stop and carry on
Ctrl+C stops it. Run the same command again to carry on; newer versions download by themselves.
sh forjed-worker.sh
Your limits
Decide how much it uses.
Each command saves the setting, and a running worker picks it up within seconds. Downloads and uploads both count.
sh forjed-worker.sh limit 50GB- At most 50 GB per month.
sh forjed-worker.sh limit 200GB --period week- At most 200 GB per week.
sh forjed-worker.sh rate 20Mbit/s- At most 20 Mbit/s.
sh forjed-worker.sh pace on- Spread the allowance evenly over the period.
sh forjed-worker.sh concurrency 4- Pages at the same time, from 1 to 16 (2 by default).
sh forjed-worker.sh status- Your settings and the data used so far.
sh forjed-worker.sh forget- Forget the server and token; the next run signs up again.
Good to know
Before you start.
Does it respect websites?
Always. Workers respect robots.txt, whatever else is set.
What if Forjed pauses crawling?
Your worker finishes the page it is on, then waits, checking every 30 seconds. It carries on by itself when crawling resumes.
Where are its settings kept?
In ~/.forjed-worker on your computer, or in the folder named by FORJED_WORKER_HOME.
Were you given a token?
Start the worker with --token followed by your token, instead of signing up.
Can I run it on Windows?
Yes, inside WSL (Windows Subsystem for Linux).
Questions before you start?
Write to us, or read more about the crawler.
