init commit

2026-07-07 05:35:14 +03:00
parent 63352e1128
commit 77ff644f89
5 changed files with 1031 additions and 1 deletions
@@ -0,0 +1,329 @@
+# rule34vault downloader ideas
+
+## User requirements / comments to keep
+
+- Download bookmarks for a public username; no authorization required.
+- Bookmark pages are available as `https://rule34vault.com/u/<username>?page=<n>`.
+- Only username should be required for normal use.
+- Use browser-like headers to reduce chance of request blocking/blocklisting.
+- Do **not** rely on `HEAD` only. Some servers may reject or mis-handle `HEAD`; use `GET` with streaming/range checks when needed.
+- For normal image posts, first try predictable original-file URLs:
+  - `https://rule34vault.com/posts/<prefix>/<post_id>/<post_id>.jpg`
+  - `https://r34xyz.b-cdn.net/posts/<prefix>/<post_id>/<post_id>.jpg`
+- `<prefix>` is the first digits group from the post id path. Example: post `1263322` uses prefix `1263`.
+- Some videos require opening the post page and extracting real `<video>/<source src=...>` URLs from the rendered HTML.
+- Create a plain text list of downloaded files in the same order they appear in bookmarks.
+- Plain txt state/log files are enough. Do **not** use SQLite.
+- Use multiple tries and exponential sleep after failures.
+- Project must use a new venv.
+
+## Real site structure inspected
+
+Inspected with browser-like headers from the project venv.
+
+### Video post example
+
+URL:
+
+```text
+https://rule34vault.com/post/738794
+```
+
+The page contains a real video element in returned HTML:
+
+```html
+<video controls loop class="video" style="aspect-ratio: 1280/720;">
+  <source type="video/mp4" src="https://r34xyz.b-cdn.net/posts/738/738794/738794.480.mp4">
+</video>
+```
+
+So video extraction should parse the post HTML with BeautifulSoup and collect:
+
+1. `video[src]`
+2. `video source[src]`
+3. fallback: any `source[src]` where `type` starts with `video/`
+
+For post `996379`, real page had multiple video sources:
+
+```text
+https://r34xyz.b-cdn.net/posts/996/996379/996379.720.hevc.mp4
+https://r34xyz.b-cdn.net/posts/996/996379/996379.480.mp4
+```
+
+Prefer broadly-compatible sources first, e.g. non-HEVC MP4 before HEVC if both exist.
+
+### Image post examples
+
+For:
+
+```text
+https://rule34vault.com/post/1263322
+```
+
+The page HTML contains a small display image:
+
+```html
+<img class="img" src="/posts/1263/1263322/1263322.small.jpg">
+```
+
+But the original image is available directly:
+
+```text
+https://rule34vault.com/posts/1263/1263322/1263322.jpg
+https://r34xyz.b-cdn.net/posts/1263/1263322/1263322.jpg
+```
+
+Both returned `200 image/jpeg` during inspection.
+
+Another inspected example:
+
+```text
+https://rule34vault.com/post/1263303
+```
+
+Sometimes the post page `img[src]` points only to a small/preview file:
+
+```text
+https://rule34vault.com/posts/1263/1263303/1263303.small.jpg
+```
+
+In this case the downloader must normalize the source URL to the original filename by removing the size suffix before the extension:
+
+```text
+https://rule34vault.com/posts/1263/1263303/1263303.jpg
+```
+
+This URL returned `200 image/jpeg` during inspection. Do not download `.small.jpg` when the corresponding original image exists.
+
+### Bookmark page structure
+
+On:
+
+```text
+https://rule34vault.com/u/krosh?page=6
+```
+
+Post links were **not** regular `<a href="/post/...">` links in the DOM. The bookmark data was inside:
+
+```html
+<script id="ng-state" type="application/json">...</script>
+```
+
+Relevant key looked like:
+
+```json
+"post:/api/v2/post/search/bookmarked/50311": {
+  "items": [
+    {"id": 1075647, "type": 0, ...},
+    {"id": 1081537, "type": 0, ...},
+    ...
+  ]
+}
+```
+
+Therefore bookmark extraction should prefer parsing `script#ng-state` JSON and reading `items` from the key starting with:
+
+```text
+post:/api/v2/post/search/bookmarked/
+```
+
+Keep item order exactly as found in that `items` array.
+
+The API also works directly as POST:
+
+```text
+POST https://rule34vault.com/api/v2/post/search/bookmarked/<user_id>
+```
+
+Examples observed:
+
+- Body `{"skip": 100, "take": 20}` returned 20 bookmark items.
+- Body `{"take": 20, "page": 6}` returned page-like bookmark items.
+
+Simplest robust path:
+
+1. Fetch profile page `/u/<username>?page=<page>`.
+2. Parse `ng-state`.
+3. Get user id from key `get:/api/v2/account/user/<username>`.
+4. Get bookmark items from `post:/api/v2/post/search/bookmarked/<user_id>`.
+5. Optionally use the direct POST API for pagination once user id is known.
+
+## Headers
+
+Use one shared `requests.Session()` with realistic headers:
+
+```python
+HEADERS = {
+    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
+                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
+    "Accept-Language": "en-US,en;q=0.9",
+    "Connection": "keep-alive",
+    "Referer": "https://rule34vault.com/",
+}
+```
+
+For direct API POST, use:
+
+```python
+API_HEADERS = {
+    **HEADERS,
+    "Accept": "application/json,text/plain,*/*",
+    "Origin": "https://rule34vault.com",
+    "Content-Type": "application/json",
+}
+```
+
+For media download, set referer to the post page:
+
+```python
+headers = {**HEADERS, "Referer": f"https://rule34vault.com/post/{post_id}"}
+```
+
+## Retry policy
+
+Every network operation should use retry wrapper:
+
+```python
+for attempt in range(max_tries):
+    try:
+        ...
+    except requests.RequestException:
+        sleep = base_delay * (2 ** attempt)
+        time.sleep(sleep)
+```
+
+Also retry on temporary HTTP statuses:
+
+```text
+429, 500, 502, 503, 504
+```
+
+Do not retry permanent `404` for candidate media URLs; just try the next candidate.
+
+## Download order log
+
+Create a txt manifest in download order:
+
+```text
+download_order.txt
+```
+
+Append one line per successfully downloaded post, exactly in bookmark order:
+
+```text
+000001 1075647 downloads/1075/1075647.jpg https://rule34vault.com/posts/1075/1075647/1075647.jpg
+000002 1081537 downloads/1081/1081537.jpg https://r34xyz.b-cdn.net/posts/1081/1081537/1081537.jpg
+000003 996379 downloads/996/996379.mp4 https://r34xyz.b-cdn.net/posts/996/996379/996379.480.mp4
+```
+
+Also keep simple txt logs:
+
+```text
+seen.txt      # post ids already processed successfully
+failed.txt    # post id + reason
+skipped.txt   # already existing files
+```
+
+No SQLite.
+
+## Media resolution strategy
+
+For each bookmark item in order:
+
+1. If already downloaded, skip and log to `skipped.txt`.
+2. If `type == 0` image:
+   - Try original candidate URLs by extension and host.
+   - Use streaming `GET`, not only `HEAD`.
+   - Validate `status_code == 200` and `Content-Type` starts with `image/`.
+3. If candidates fail, fetch post page and parse actual HTML.
+   - For image posts, inspect `img[src]` values from the real post page.
+   - If the only discovered image is a display derivative such as `<post_id>.small.jpg`, `<post_id>.preview.jpg`, `<post_id>.thumbnail.jpg`, etc., derive and try the original URL without the size suffix before downloading:
+     - `1263303.small.jpg` -> `1263303.jpg`
+     - `1263303.preview.jpg` -> `1263303.jpg`
+   - Download the higher-resolution original when it returns a valid image response; use the derivative only as a last fallback if the original does not exist.
+4. For videos (`type == 1`) or if image candidates fail:
+   - Fetch `https://rule34vault.com/post/<post_id>`.
+   - Parse `<video>` and `<source>` tags.
+   - Choose best compatible source.
+5. Download media to grouped directory:
+
+```text
+downloads/<prefix>/<post_id>.<ext>
+```
+
+## Candidate URL generation
+
+For post id `1263322`:
+
+```python
+prefix = post_id[:4]
+```
+
+Candidate originals:
+
+```python
+hosts = [
+    "https://rule34vault.com",
+    "https://r34xyz.b-cdn.net",
+]
+
+extensions = ["jpg", "png", "webp", "gif"]
+
+for host in hosts:
+    for ext in extensions:
+        yield f"{host}/posts/{prefix}/{post_id}/{post_id}.{ext}"
+```
+
+For video direct guesses, possible candidates can be tried after HTML parsing or as fallback:
+
+```text
+<post_id>.480.mp4
+<post_id>.720.mp4
+<post_id>.720.hevc.mp4
+<post_id>.webm
+```
+
+But actual post HTML source extraction is preferred for videos.
+
+## CLI idea
+
+```bash
+python -m venv .venv
+. .venv/bin/activate
+pip install requests beautifulsoup4 tqdm
+python r34vault_downloader.py krosh --start-page 1 --end-page 10 --out downloads
+```
+
+Arguments:
+
+```text
+username              required
+--start-page          default 1
+--end-page            optional; stop when no items if omitted
+--out                 default downloads
+--tries               default 4
+--delay               default 0.5 seconds between posts
+--base-retry-delay    default 1.5 seconds
+--dry-run             print planned downloads without saving media
+```
+
+## Core functions
+
+```python
+make_session()
+request_with_retries(method, url, **kwargs)
+fetch_user_page(username, page)
+parse_ng_state(html)
+extract_user_id(ng_state, username)
+extract_bookmark_items_from_state(ng_state, user_id)
+fetch_bookmark_items_api(user_id, page=None, skip=None, take=20)
+candidate_image_urls(post_id)
+stream_download(url, dest, referer)
+extract_media_sources_from_post_html(post_id)
+choose_video_source(sources)
+already_downloaded(post_id, out_dir)
+append_download_order(index, post_id, path, source_url)
+append_failed(post_id, reason)
+main()
+```