Rustle

Usage

First, ensure that the config file exists at the following location (per your OS):

OS	Path
Linux	`$HOME/.config/Rustle/config.toml`
MacOS	`$HOME/Library/Application Support/Rustle/config.toml`

Then just use:

rustle

Configuration

Example config.toml file:

origin_url = "https://example.com"
depth = 6
database_name = "crawler"

Logging

To configure logging, this program uses the RUST_LOG environment variable, with options:
- error
- warn
- info
- debug
- trace
Example:
```
RUST_LOG=info rustle
```

Roadmap

Abstract code & functionality into structs & other files
Use SQLite to store information about websites, instead of downloading HTML
Recursion fix, specify depth
config file parsing to specify origin url & depth
Parallel / distributed crawling
Obey robots.txt