73 lines
1.6 KiB
Markdown
73 lines
1.6 KiB
Markdown
# rule34vault-downloader
|
|
|
|
Python script to download public bookmarks from `rule34vault.com` for a username. No authorization is required for public bookmark pages.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
python -m venv .venv
|
|
. .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Usage
|
|
|
|
Download bookmarks for a user:
|
|
|
|
```bash
|
|
python r34vault_downloader.py <user> --out downloads
|
|
```
|
|
|
|
Download only a page range:
|
|
|
|
```bash
|
|
python r34vault_downloader.py <user> --start-page 1 --end-page 3 --out downloads
|
|
```
|
|
|
|
Process only the first N bookmark items, useful for testing:
|
|
|
|
```bash
|
|
python r34vault_downloader.py <user> --start-page 6 --end-page 6 --limit 1 --out downloads
|
|
```
|
|
|
|
## Output
|
|
|
|
Files are grouped by Rule34Vault post prefix:
|
|
|
|
```text
|
|
downloads/
|
|
1263/
|
|
1263322.jpg
|
|
996/
|
|
996379.mp4
|
|
download_order.txt
|
|
seen.txt
|
|
failed.txt
|
|
skipped.txt
|
|
```
|
|
|
|
`download_order.txt` records successful downloads in bookmark order:
|
|
|
|
```text
|
|
000001 1263322 downloads/1263/1263322.jpg https://rule34vault.com/posts/1263/1263322/1263322.jpg
|
|
```
|
|
|
|
## Behavior
|
|
|
|
- Uses browser-like headers and post-page `Referer` headers.
|
|
- Uses streaming `GET`; it does not rely on `HEAD` requests.
|
|
- Parses bookmark data from `script#ng-state` and preserves bookmark order.
|
|
- Tries original image URLs first, for example `1263322.jpg`.
|
|
- If a page image is only `1263303.small.jpg`, it tries `1263303.jpg` before falling back to the small file.
|
|
- For videos, parses real `<video>` and `<source src="...">` elements from the post page.
|
|
- Retries transient failures with exponential backoff.
|
|
- Uses plain text logs only; no SQLite database.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
. .venv/bin/activate
|
|
pip install pytest
|
|
pytest -q
|
|
```
|