one more test on real sysiphus packets

.clang-format

@@ -1,3 +1,4 @@
 BasedOnStyle: Google
 IndentWidth: 2
-ColumnLimit: 100
+BracedInitializerIndentWidth: 2
+ColumnLimit: 100

Docs/more/compare-sisyphus-set-versions.md

@@ -1,125 +0,0 @@
-# Сравнение set-зависимостей Sisyphus с `newset.c`
-
-Скрипт [`scripts/compare_sisyphus_set_versions.sh`](../scripts/compare_sisyphus_set_versions.sh)
-сравнивает set-строки из RPM-заголовка с результатом `reimplement/newset.c`.
-
-## Что именно сравнивается
-
-1. Из изолированных APT-индексов `Sisyphus/x86_64` и `Sisyphus/noarch`
-   берутся `Package`, `Architecture`, `Version`, `Filename`, `MD5Sum`,
-   `Provides` и `Depends`/`Pre-Depends`.
-2. RPM скачивается непосредственно из `Sisyphus/files/<arch>/RPMS` и
-   проверяется по `MD5Sum` из индекса.
-3. Эталонные зависимости читаются из RPM-заголовка через массивы тегов:
-   `PROVIDENAME/PROVIDEFLAGS/PROVIDEVERSION` и
-   `REQUIRENAME/REQUIREFLAGS/REQUIREVERSION`.
-4. Для воспроизведения зависимостей вызываются установленные вместе с
-   `rpm-build` программы `%_rpmlibdir/find-provides` и
-   `%_rpmlibdir/find-requires` с методом `none,lib`.
-5. В каталоге инструментов заменяется **только** `mkset`: вместо штатного
-   `mkset` подставляется совместимая обвязка над `newset.c`. `provided_symbols`,
-   `ldd --bindings`, `eu-readelf`, `objdump`, `file` и остальные средства
-   отбора/извлечения символов остаются штатными инструментами `rpm-build`.
-
-Для `Requires` APT получает точный уже скачанный RPM-файл как вход и скачивает
-его полный runtime dependency closure в пустой пользовательский RPM root. Так
-версия и архитектура целевого пакета не выбираются повторно по имени. APT
-запускается с пустым `APT_CONFIG`; source list, package cache, archives, lists,
-status и preferences перенаправлены во временный каталог, поэтому настройки,
-hooks и RPM-база основной системы не участвуют в разрешении зависимостей.
-
-Перед распаковкой проверяются имена элементов cpio и цели символических ссылок.
-Абсолютная RPM-ссылка считается путём внутри виртуального package root: например,
-`/usr/sbin/update-alternatives -> /bin/true` безопасно переписывается в
-`../../bin/true`, а не направляется в `/bin/true` основной системы. Все symlink
-сначала собираются из RPM-заголовков и создаются в порядке от родительских путей
-к дочерним; затем `cpio` извлекает только нессылочные элементы. Благодаря этому
-usrmerge-ссылки вроде `/bin -> usr/bin` создаются до файлов `/bin/*`.
-
-Имена элементов с выходом через `..` и относительные symlink, фактически
-выходящие из временного root, по-прежнему отклоняются. После проверки RPM-файлы
-распаковываются без установки. Это позволяет штатному `rpm-build/ldd`
-использовать интерпретатор и библиотеки из того же снимка Sisyphus, не изменяя
-систему.
-
-## Ограниченный тест
-
-На ALT p11 с установленным `rpm-build`:
-
-```sh
-bash scripts/compare_sisyphus_set_versions.sh \
-    --package strace \
-    --report strace-set-report.tsv
-```
-
-Несколько пакетов:
-
-```sh
-bash scripts/compare_sisyphus_set_versions.sh \
-    --package strace \
-    --package libdw \
-    --report sample-set-report.tsv
-```
-
-Либо первые десять записей индекса:
-
-```sh
-bash scripts/compare_sisyphus_set_versions.sh \
-    --limit 10 \
-    --report sample-set-report.tsv
-```
-
-Без `--all`, `--limit` или `--package` скрипт завершится до обращения к
-репозиторию. Это защищает от случайного полного запуска.
-
-## Полный запуск
-
-Команда предусмотрена, но в ходе разработки не запускалась:
-
-```sh
-bash scripts/compare_sisyphus_set_versions.sh \
-    --all \
-    --report sisyphus-set-report.tsv
-```
-
-Продолжение после прерывания:
-
-```sh
-bash scripts/compare_sisyphus_set_versions.sh \
-    --all \
-    --resume \
-    --report sisyphus-set-report.tsv
-```
-
-`--resume` пропускает только уже завершённую комбинацию
-`package/architecture/version`; обновившийся пакет будет обработан заново.
-Завершённость подтверждается маркером `complete=1` в полной 11-польной строке
-`SUMMARY`, поэтому оборванная при записи строка не считается результатом.
-
-Во время работы отчёт можно наблюдать отдельно:
-
-```sh
-tail -f sisyphus-set-report.tsv
-```
-
-`START` записывается до обработки пакета, а `DEPENDENCY` и `SUMMARY` — сразу
-после сравнения пакета. Поэтому файл обновляется в процессе, а не только после
-завершения всего обхода.
-
-## Статусы отчёта
-
-- `match` — set-строка из RPM совпала с результатом `newset.c`;
-- `mismatch` — capability и оператор совпали, set-строки различаются;
-- `missing_generated` — RPM содержит set-зависимость, но повторный запуск
-  `rpm-build` её не сгенерировал;
-- `extra_generated` — повторный запуск сгенерировал зависимость, которой нет в
-  RPM-заголовке;
-- `no_set_metadata` — в APT metadata пакета нет set-строк;
-- `*_error` — ошибка скачивания, контрольной суммы, распаковки либо генератора.
-
-`extra_generated` показывается в строках `DEPENDENCY` и делает итоговый статус
-пакета равным `mismatch`. При анализе нужно учитывать, что бинарный RPM не
-содержит исходных spec-фильтров (`filter_from_requires` и аналогичных), поэтому
-часть таких расхождений может относиться не к `newset.c`, а к невозможности
-полностью воспроизвести фильтрацию исходного spec. Их число отдельно указано в
-поле `extras=N` строки `SUMMARY`.

Docs/more/for_me.md

@@ -1,71 +0,0 @@
-# Notes
-
-## to read
-
-https://www.altlinux.org/Features/Specific
-
-## Hash
-
-в данный момент используется Jenkins OAAT. Считается чуть устаревшим(check), главным аналогом является:
-
-https://github.com/cyan4973/xxhash
-
-xxHash64 (XXH64): относительно простая имплементация на си, хвалят за скорость, 64 бит, параллель
-
-необходимы тесты и графики:
-
-1. насколько быстрее/медленнее
-2. коллизии на наборе букв в сравнении с рандомными данными
-3. коллизии у первого и второго хэшей между собой
-
-> В XXH3 внутренний цикл, который оптимально обрабатывается векторизацией. Благодаря этому функция использует аппаратную поддержку на наборах инструкций SSE2, AVX2 и NEON. Производительность зависит от комилятора. Неожиданно оказалось, что версия, скомпилированная clang, намного превосходит остальные. Ян Колле даже подумал, что производительность хеш-функции здесь превысит пропускную способность памяти. Этой версии на графике соответствует пунктирная линия.
-
-есть ещё cityhash от гугла, но она c++
-
-в случае XXH64 строк кода ~200, можно вставить в файл
-в случае XXH3 строк кода ~2к, стоит использовать библиотеку напрямую, бессмысленно
-
-## about collizions
-
-> А вы пробовали побитовую вероятностную карту строить?
-> Есть у нас по условию множество из входных значений, которые замапились в N хешей (включая коллизии!). Эти N хешей имеют одинаковый размер и, по сути, являются массивами из K бит.
-> Так вот можно среди этих хешей посчитать 0 и посчитать 1, получится K распределений. Чем они все ближе к пополамному ± 6%, тем лучше.
-> https://raw.altlinux.team/arseny/snippets/2026-06/QSIrJSuwxp.txt
-> https://raw.altlinux.team/arseny/snippets/2026-06/QSIrJSuwxp.c
-
-## free
-
-free не делает очистку самой структуры, valgrind
-
-## data
-
-алфавит для символ библиотек:
-
-```
-.0123456789@ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz
-```
-
-- предоставленные/определенные символы из библиотек обычно используют @@VERSION
-- обязательные/неопределенные символы из двоичных файлов обычно используют @VERSION
-
-## additional about set:version.md
-
-[коды](https://altlinux.space/arseny/atsv-research) от Арсения для наглядности происходящего
-
-some funny [msg's](https://lists.pld-linux.org/mailman/pipermail/pld-devel-en/2013-November/012467.html)
-
-по коду неоднократно раскидано `bpp < 10 || bpp > 32` проверки
-
-запись `char = [1,0]` мне не нравится.
-
-про "отрицательные значения = код ошибки" встречается в многих местах, стоит вынести отдельно
-проверить кодом примеры, особенно base62
-
-кэш вечно копируется и переносится, немного странно
-PIVOT_SIZE странный
-
-прыжки IFLT8 и IFLT4 я бы возможно делал как c1/c2
-
-учитывая оптимизации, возможно стоит самостоятельно менять массивы местами до начала всех операций (но проблема с кэшем возможна)
-
-хочу себе день, чтобы переписать это всё на English

Docs/more/raw_data.md

@@ -1,46 +0,0 @@
-# raw_data
-
-## messages from tg
-
-[6/23/26 9:57 PM] Arseny: Кстати. У меня есть серьёзная математическая, она же очень серьёзная алгоритмическая задача на исследование. ALT set:version (https://git.altlinux.org/gears/r/rpm.git?a=blob;f=lib/set.c) нужен майнтейнер.
-[6/23/26 9:57 PM] Arseny: Вы хотели алгоритмов? Вот, расскажите мне про этот.
-[6/23/26 9:57 PM] Arseny: весь алгоритм можно разбить на 4 последовательных стадии.
-
-0) Задача в том, чтобы проверять, выполняется ли R ⊂ P.
-множества, стоящие в позиции P, даются командой навроде nm --dynamic -j -U /usr/lib64/libyourfavourite.so.N.
-множества, стоящие в позиции R, даются командой навроде nm --dynamic -j -u /usr/bin/yourfavouriteprog, или там может быть иной .so.
-Тут, наверное, ещё и введение в предметную область должно быть.
-
-1) hash: N элементов сбрасываются в этот самый хеш; его эффективность бы стоило оценить хорошенько. (гипотеза от меня: он вообще-то плохой и даёт много коллизий на интересных нам инпутах), в результате K <= N отсортированных хешей, чисел из [0; 2**10) (по умолчанию 10 бит на хеш);
-2) delta: K <= N чисел из прошлого шага заменяются на разности с предыдущими, и получаются по-другому распределённые числа, чаще маленькие и очень, очень мало сверхбольших; есть мнение, что они для равномерно распределённых хешей распределены геометрически;
-3) golomb: эти K разностей кодируются по Rice-Golomb (https://en.wikipedia.org/wiki/Golomb_coding), делитель по умолчанию 2**7; получается префиксный код из K битострок, которые можно слева направо прочесть без lookahead и восстановить массив разностей;
-4) (якобы) base62: их конкатенация кодируется в буквы.
-
-(2) и (3) близки к очевидно-понятным; я поизучал, реализовал на питоне (https://altlinux.space/arseny/atsv-research) и снова поизучал.
-А вот (1) и (4) я не занимался совсем.
-От (4) у меня тоже башка взрывается, а (1) можно очень хорошо так поизучать, и существует, наверное, обширная литература.
-[6/23/26 9:57 PM] Arseny: grep -A16 Jenkins lib/set.c
-вот в этом дереве исходников (https://git.altlinux.org/gears/r/rpm.git?p=rpm.git;a=blob;f=lib/set.c;h=9474a2ee6d7c;hb=a01a87db6e8a)
-[6/23/26 9:57 PM] Arseny: Пусть параметр-делитель равен M, и энкодеру на вход поступает число A. Тогда ищутся такие q и r < q, что A = qM + r. Частное кодируется унарной кодировкой из единиц, за ним следует нулик, а далее ровно log₂M бит на остаток. Подобрать M нужно с умом.
-```sh
-python3 stringset.py 10 7 <<EOF
-foo
-bar
-quux
-EOF
-hashes=[298, 487, 588]
-deltas=[298, 189, 101]
-codes=[b'1100101010', b'100111101', b'01100101']
-golomb divisor: 128 (minimum word bit length: 8)
-total bit length: 27
-per-elem bit length: 9.000
-```
-Но здесь не хеш из файла выше, а питонический hash() обычный. Чтобы смотреть на актуальные результаты, нужно подсунуть актуальный.
-[6/23/26 9:58 PM] Arseny: А вот страничка set:version на ALT Wiki (https://www.altlinux.org/Set-version)
-[6/23/26 9:58 PM] Arseny: Putze, F.; Sanders, P.; Singler, J. (2007),
- 253  * "Cache-, Hash- and Space-Efficient Bloom Filters",
- 254  * http://algo2.iti.uni-karlsruhe.de/singler/publications/cacheefficientbloomfilters-wea2007.pdf
-[6/23/26 9:58 PM] Arseny: Сайта нет, но статью поискать https://t.me/c/1622441719/1359/9771
-[6/23/26 9:59 PM] Arseny: И ещё наследие Алексея Турбина:
-https://ftp.altlinux.org/pub/people/at/
-https://github.com/svpv

Docs/more/rpmquery-alt-linux.md

@@ -1,444 +0,0 @@
-# `rpmquery` в ALT Linux
-
-## Краткий вывод
-
-`rpmquery` — отдельная точка входа в тот же механизм запросов RPM, что и
-`rpm -q`. В ALT Linux p11 `/usr/bin/rpmquery` является символьной ссылкой на
-ELF-программу `/usr/lib/rpm/rpmq`; имя запуска заранее включает режим запроса,
-поэтому `-q` обычно не нужен:
-
-```sh
-rpmquery rpm
-rpmquery -q rpm
-rpm -q rpm
-```
-
-Все три команды дают один и тот же результат. При этом `rpmquery` не является
-интерфейсом к репозиторию APT: без `-p` он читает только установленную базу RPM,
-по умолчанию `/var/lib/rpm`. Для поиска доступных, но не установленных пакетов
-нужны `apt-cache`, `apt-get` или веб/RDB-интерфейсы ALT.
-
-Исследование проверено на официальном контейнере `alt:p11`, обновлённом до
-`rpm-4.13.0.1-alt45.x86_64`, и по ветке `p11` исходников RPM, commit
-`7e71ed3da438289d916f1cccc9c0a83e594455dc`.
-
-## Где ищет `rpmquery`
-
-| Вызов | Источник данных |
-|---|---|
-| `rpmquery NAME` | установленная база RPM |
-| `rpmquery -a` | все пакеты из установленной базы RPM |
-| `rpmquery -p FILE.rpm` | заголовок указанного RPM-файла, установка не требуется |
-| `rpmquery --root ROOT ...` | база и файловая система относительно `ROOT` |
-| `rpmquery --dbpath DIR ...` | явно указанная база RPM |
-
-Практическая граница была проверена так: `apt-cache show strace` находил пакет
-в p11, тогда как `rpmquery strace` отвечал `package strace is not installed` и
-завершался с кодом `1`.
-
-## Работа с APT-репозиторием
-
-APT-RPM загружает индексы подключённых репозиториев в `/var/lib/apt/lists`.
-Обычный порядок работы:
-
-```sh
-# Показать настроенные источники ALT, если установлен пакет apt-repo.
-apt-repo
-
-# Обновить локальные индексы после изменения источников.
-apt-get update
-```
-
-В минимальных контейнерах `apt-repo` может отсутствовать; источники всё равно
-задаются файлами `/etc/apt/sources.list` и `/etc/apt/sources.list.d/*.list`.
-
-Сами `pkglist.*` являются бинарными header-list файлами RPM, поэтому искать в
-них обычным `grep` не следует. Для доступа к кэшу предназначен `apt-cache`:
-
-| Задача | Команда |
-|---|---|
-| Поиск по имени и описанию | `apt-cache search REGEX` |
-| Перечень имён | `apt-cache pkgnames` |
-| Выбранная версия и приоритеты источников | `apt-cache policy PACKAGE` |
-| Полная доступная запись, включая версии зависимостей | `apt-cache show PACKAGE` |
-| Граф разрешения зависимостей | `apt-cache depends PACKAGE` |
-| Обратные зависимости | `apt-cache whatdepends PACKAGE` |
-| Низкоуровневое представление версий и зависимостей | `apt-cache showpkg PACKAGE` |
-| Метаданные исходного пакета | `apt-cache showsrc PACKAGE` |
-| Все доступные записи репозиториев | `apt-cache dumpavail` |
-| Проверка плана установки без изменений | `apt-get -s install PACKAGE` |
-| Только скачать бинарный пакет и зависимости | `apt-get -d install PACKAGE` |
-| Скачать исходники | `apt-get source PACKAGE` |
-
-`apt-cache depends` удобен для просмотра найденных провайдеров, но в его
-обычном выводе set-версия зависимости теряется. Для полных set-строк нужны
-`apt-cache show`, `apt-cache showpkg`, скачанный RPM или RDB API.
-
-## Set-строки пакета из APT-репозитория
-
-### Способ 1: прямо из локального кэша APT
-
-Это самый короткий способ, установка и скачивание RPM не нужны:
-
-```sh
-pkg=libdw
-
-# Сначала проверить, какую версию и из какого источника выбрал APT.
-apt-cache policy "$pkg"
-
-# Полные поля с set-Requires и set-Provides.
-apt-cache show "$pkg" |
-sed -n -E '/^(Pre-Depends|Depends|Provides):/p' |
-grep -F 'set:'
-```
-
-В проверенном p11 поля имели вид:
-
-```text
-Depends: ... liblzma.so.5()(64bit) (>= set:kiyIz7cr3p0), ...
-Provides: libdw.so.1()(64bit) (= set:ldm5ZH1pjPBqjZ9bdJbLDnKGbC1hkJSX...)
-```
-
-`Depends` и `Pre-Depends` соответствуют требованиям пакета, `Provides` — тому,
-что пакет предоставляет. `apt-cache show` может вывести несколько записей,
-если в подключённых источниках доступны разные версии; выбранный кандидат
-виден в `apt-cache policy`.
-
-Для выгрузки всего подключённого репозитория можно использовать:
-
-```sh
-apt-cache dumpavail >available.txt
-grep -E '^(Pre-Depends|Depends|Provides):.*set:' available.txt
-```
-
-В отличие от HTML-страниц `packages.altlinux.org`, кэш APT содержит полные
-set-строки без визуального многоточия.
-
-### Способ 2: скачать RPM через APT и запросить его заголовок
-
-Это наиболее точный локальный способ: выводятся исходные массивы тегов RPM и
-их удобно преобразовать в TSV. `apt-get -d` скачивает также отсутствующие
-зависимости, но ничего не устанавливает.
-
-```sh
-pkg=strace
-cache=$(mktemp -d)
-trap 'rm -rf "$cache"' EXIT
-mkdir -p "$cache/partial"
-
-apt-get -y -d \
-  -o "Dir::Cache::archives=$cache" \
-  install "$pkg"
-
-rpmfile=$(find "$cache" -maxdepth 1 -type f \
-  -name "${pkg}_*.rpm" -print -quit)
-```
-
-Set-Requires выбранного пакета:
-
-```sh
-rpmquery -p --qf \
-  '[%{REQUIRENAME}\t%{REQUIREFLAGS:depflags}\t%{REQUIREVERSION}\n]' \
-  "$rpmfile" |
-awk -F '\t' '$3 ~ /^set:/'
-```
-
-Set-Provides выбранного пакета:
-
-```sh
-rpmquery -p --qf \
-  '[%{PROVIDENAME}\t%{PROVIDEFLAGS:depflags}\t%{PROVIDEVERSION}\n]' \
-  "$rpmfile" |
-awk -F '\t' '$3 ~ /^set:/'
-```
-
-Для `strace-7.0-alt1.x86_64.rpm` в p11 получены два требования:
-
-```text
-libdw.so.1()(64bit)       >=  set:lgHyMNBkvSTjjY0en8Yi3vFojgjVM7jRsRF4zCFBzNVmhhGF
-libselinux.so.1()(64bit)  >=  set:liZ0N709Wr2dbKPs2
-```
-
-Set-Provides у этого пакета нет: исполняемый `strace` требует библиотеки, но не
-экспортирует собственную разделяемую библиотеку. У скачанного вместе с ним
-пакета `libdw` присутствует длинный Provide
-`libdw.so.1()(64bit) = set:...`.
-
-Если пакет уже установлен и APT не скачивает его повторно, его заголовок можно
-прочитать напрямую командой `rpmquery PACKAGE`; для гарантированного получения
-конкретного репозиторного RPM удобнее использовать RDB API.
-
-### Способ 3: структурированные данные RDB API
-
-RDB удобен для автоматизации, другой ветки/архитектуры и больших выборок. Он
-возвращает полные строки и явно различает `provide`, `require` и `conflict`:
-
-```sh
-branch=p11
-arch=x86_64
-pkg=libdw
-
-hash=$(curl -fsS \
-  "https://rdb.altlinux.org/api/site/pkghash_by_binary_name?branch=$branch&name=$pkg&arch=$arch" |
-  jq -r '.pkghash')
-
-curl -fsS \
-  "https://rdb.altlinux.org/api/dependencies/binary_package_dependencies/$hash" |
-jq -r '
-  .dependencies[]
-  | select(
-      (.type == "provide" or .type == "require") and
-      (.version | startswith("set:"))
-    )
-  | [.type, .name, .version]
-  | @tsv
-'
-```
-
-Проверенный результат для `libdw` содержит один set-Provide и четыре
-set-Requires. Поле `flag_decoded` того же JSON позволяет восстановить оператор:
-`RPMSENSE_EQUAL` для обычного Provide и
-`RPMSENSE_GREATER` + `RPMSENSE_EQUAL` для `>=` Require.
-
-Через RDB можно также получить URL самого RPM:
-
-```sh
-curl -fsS \
-  "https://rdb.altlinux.org/api/site/package_downloads_bin/$hash?branch=$branch&arch=$arch" |
-jq -r '.downloads[].packages[].url'
-```
-
-## Основные запросы
-
-### Пакет и его метаданные
-
-```sh
-# Версия установленного пакета в стандартном формате NVRA.
-rpmquery rpm
-
-# Подробная информация. В ALT вывод включает DistTag.
-rpmquery -i rpm
-rpmquery --info rpm
-
-# Стабильный формат для скрипта.
-rpmquery --qf '%{NAME}|%{EPOCHNUM}|%{VERSION}|%{RELEASE}|%{DISTTAG}|%{ARCH}\n' rpm
-
-# Доступные имена тегов заголовка.
-rpmquery --querytags
-```
-
-В `rpmquery` короткая опция `-i` означает `--info`, а не установку пакета. Это
-ALT-алиас из `rpmpopt`; он появился в changelog RPM в версии
-`4.13.0-alt7`. Для установки используется другой режим программы `rpm`, но в
-обычной работе с ALT зависимости и репозитории следует поручать APT.
-
-### Файлы
-
-```sh
-# Какому установленному пакету принадлежит файл.
-rpmquery -f /usr/bin/rpmquery
-
-# Все файлы пакета.
-rpmquery -l rpm
-
-# Только конфигурационные или документационные файлы.
-rpmquery -c rpm
-rpmquery -d rpm
-
-# Пакет из файла без установки.
-rpmquery -p ./package.rpm
-rpmquery -lp ./package.rpm
-rpmquery -ip ./package.rpm
-```
-
-Для аргумента-пути `--whatprovides` сначала обращается к индексу установленных
-файлов, затем к индексу capability. Поэтому для существующего установленного
-файла эти команды обычно совпадают:
-
-```sh
-rpmquery -f /usr/bin/rpmquery
-rpmquery --whatprovides /usr/bin/rpmquery
-```
-
-### Provides, Requires и обратные запросы
-
-```sh
-# Что объявляет один пакет.
-rpmquery --provides rpm
-rpmquery --requires rpm
-rpmquery -R rpm
-
-# Какие установленные пакеты объявляют capability или требуют его.
-rpmquery --whatprovides 'libpopt.so.0()(64bit)'
-rpmquery --whatrequires 'libpopt.so.0()(64bit)'
-```
-
-`--provides`, `--requires`, `--info`, `--last`, `--scripts` и ряд других
-удобных режимов реализованы как `popt`-алиасы над `--queryformat`, а не как
-отдельные алгоритмы запросов.
-
-## Формат запросов для автоматической обработки
-
-Стандартный вывод удобен человеку, но надёжнее не разбирать его пробелами.
-Массивы тегов RPM обходятся форматом в квадратных скобках. Чтобы внутри такого
-итератора повторять скалярный тег для каждого элемента массива, знак `=` ставят
-внутри имени тега: `%{=NAME}`.
-
-### Все set-Requires в TSV
-
-```sh
-LC_ALL=C rpmquery -a --qf \
-  '[%{=NAME}\t%{REQUIRENAME}\t%{REQUIREFLAGS:depflags}\t%{REQUIREVERSION}\n]' |
-awk -F '\t' '$4 ~ /^set:/'
-```
-
-Поля: пакет, имя capability, оператор, версия зависимости.
-
-Пример:
-
-```text
-rpm	libpopt.so.0()(64bit)	>=	set:jgtcU6BLBccTnteGxrE0
-```
-
-### Все set-Provides в TSV
-
-```sh
-LC_ALL=C rpmquery -a --qf \
-  '[%{=NAME}\t%{PROVIDENAME}\t%{PROVIDEFLAGS:depflags}\t%{PROVIDEVERSION}\n]' |
-awk -F '\t' '$4 ~ /^set:/'
-```
-
-Пример:
-
-```text
-libpopt	libpopt.so.0()(64bit)	=	set:jdtcJcAdqxTmPJUyuYcVIbLNhqmj...
-```
-
-Это предпочтительнее `grep` по человекочитаемому выводу, если нужны имена
-пакетов, операторы или однозначное разделение полей.
-
-## Что `rpmquery` делает и не делает с `set:version`
-
-### Показывает сохранённые зависимости
-
-`rpmquery --requires` и `rpmquery --provides` извлекают из заголовков пакетов
-имя capability, флаги отношения и строку версии. Они не декодируют `set:` и не
-показывают исходные ELF-символы.
-
-Для установленного `rpm` в p11 наблюдалось:
-
-```text
-libpopt.so.0()(64bit) >= set:jgtcU6BLBccTnteGxrE0
-```
-
-Установленный провайдер находился по одному имени capability:
-
-```sh
-rpmquery --whatprovides 'libpopt.so.0()(64bit)'
-# libpopt-1.18-alt1.x86_64
-```
-
-### Не сравнивает выражение, переданное в `--whatprovides`
-
-Передача полного выражения не запускает `rpmsetcmp()`:
-
-```sh
-rpmquery --whatprovides \
-  'libpopt.so.0()(64bit) >= set:jgtcU6BLBccTnteGxrE0'
-```
-
-Проверенный результат:
-
-```text
-no package provides libpopt.so.0()(64bit) >= set:jgtcU6BLBccTnteGxrE0
-```
-
-Для непутевого аргумента `--whatprovides` делает точный поиск по индексу
-`PROVIDENAME`. Поэтому сначала нужно искать провайдера по имени capability, а
-его set-строку получать отдельным запросом:
-
-```sh
-name='libpopt.so.0()(64bit)'
-provider=$(rpmquery --whatprovides "$name" | head -n1)
-rpmquery --qf \
-  '[%{PROVIDENAME}\t%{PROVIDEFLAGS:depflags}\t%{PROVIDEVERSION}\n]' \
-  "$provider" |
-grep -F "$name"
-```
-
-Настоящее сравнение двух `set:`-версий происходит в `rpmdsCompareEVR()` при
-проверке совместимости зависимостей. Если обе версии начинаются с `set:`, код
-вызывает `rpmsetcmp()`; если `set:` имеет только одна сторона, зависимости не
-пересекаются. Следовательно, `rpmquery` полезен для извлечения корпуса строк,
-но не является CLI для непосредственного сравнения двух наборов.
-
-## Как `rpmquery` используется при сборке самого ALT RPM
-
-В `alt/rpm.spec` ветки p11 после первой сборки выполняется:
-
-```sh
-rpmquery -a --provides | fgrep '= set:' | sort >P
-rpmquery -a --requires | fgrep '= set:' | sort >R
-join -o 1.3,2.3 P R | shuf >setcmp-data
-time ./setcmp <setcmp-data >/dev/null
-```
-
-Смысл конвейера:
-
-1. Из локальной базы сборочного окружения собираются все set-Provides и
-   set-Requires.
-2. Строка поиска `= set:` захватывает как Provides с `= set:`, так и Requires с
-   `>= set:`, потому что вторая строка тоже содержит подстроку `= set:`.
-3. После сортировки `join` соединяет файлы по первому полю — имени capability.
-4. `-o 1.3,2.3` оставляет только третьи поля: set провайдера и set требования.
-5. Полученные реальные пары подаются в `tools/setcmp` для замера реализации
-   `rpmsetcmp()` и профиль-управляемой пересборки `lib/set.c`.
-
-В обновлённом минимальном контейнере p11 точный конвейер дал:
-
-```text
-P: 75 строк
-R: 109 строк, из них 13 полных дублей
-setcmp-data: 109 пар
-```
-
-Эти числа описывают только состав минимального контейнера, а не весь p11 или
-Sisyphus. Также конвейер spec не сохраняет имя capability и пакет-владелец;
-для исследовательского корпуса удобнее TSV-форматы выше.
-
-Современная замена устаревающего `fgrep` без изменения смысла:
-
-```sh
-grep -F '= set:'
-```
-
-## Практические ограничения
-
-- `rpmquery -a` означает все **установленные** пакеты, а не весь репозиторий.
-- `--whatprovides` и `--whatrequires` ищут заголовки по имени capability; они
-  не принимают полноценное выражение зависимости как запрос сравнения.
-- Несколько установленных версий или провайдеров могут дать несколько строк.
-- Set-строки крупных библиотек очень длинные; лучше писать полный вывод в файл,
-  а в терминале показывать только статистику или начало строки.
-- Для скриптов следует фиксировать `LC_ALL=C` и использовать `--queryformat`.
-- `--requires`/`--provides` отражают метаданные RPM, а не текущее содержимое ELF
-  на диске. Для восстановления исходных символов нужны `nm`, `readelf` и
-  генераторы автозависимостей rpm-build.
-- При запросе другой системы безопаснее явно использовать `--root`; `--dbpath`
-  меняет только путь к БД и требует внимательности к относительным путям.
-- ALT предоставляет `--nowait-lock` как popt-алиас, отключающий ожидание
-  блокировки БД. Его стоит применять только когда отказ предпочтительнее
-  ожидания, а не как способ обходить активную транзакцию RPM/APT.
-
-## Источники
-
-1. [Документация платформы ALT 10.1: утилита RPM](https://docs.altlinux.org/ru-RU/platform/10.1/html/alt-platform/utilita_komandnoj_stroki_rpm.html) — RPM работает с файлами, пакетами, зависимостями и собственной БД, но не знает о репозиториях.
-2. [Настройка списка репозиториев APT в ALT](https://docs.altlinux.org/ru-RU/platform/10.1/html/alt-platform/nastrojka_spiska_repozitoriev_apt.html).
-3. [Пакет `rpm` в p11](https://packages.altlinux.org/en/p11/srpms/rpm/) и [specfile](https://packages.altlinux.org/en/p11/srpms/rpm/specfiles/).
-4. [RDB: текущий бинарный пакет `rpm` для p11/x86_64](https://rdb.altlinux.org/api/site/pkghash_by_binary_name?branch=p11&name=rpm&arch=x86_64).
-5. [`rpmqv.c`](https://git.altlinux.org/gears/r/rpm.git?p=rpm.git;a=blob;f=rpmqv.c;hb=7e71ed3da438289d916f1cccc9c0a83e594455dc) — выбор режима Query по имени `rpmquery`/`rpmq`.
-6. [`rpmpopt.in`](https://git.altlinux.org/gears/r/rpm.git?p=rpm.git;a=blob;f=rpmpopt.in;hb=7e71ed3da438289d916f1cccc9c0a83e594455dc) — ALT-алиасы `--requires`, `--provides`, `--info`, `-i`, `--nowait-lock`.
-7. [`lib/query.c`](https://git.altlinux.org/gears/r/rpm.git?p=rpm.git;a=blob;f=lib/query.c;hb=7e71ed3da438289d916f1cccc9c0a83e594455dc) — индексы для `--whatprovides`, запросы файлов и формат `%{nvra}` по умолчанию.
-8. [`lib/rpmds.c`](https://git.altlinux.org/gears/r/rpm.git?p=rpm.git;a=blob;f=lib/rpmds.c;hb=7e71ed3da438289d916f1cccc9c0a83e594455dc) — вызов `rpmsetcmp()` при сравнении двух set-версий.
-9. [`alt/rpm.spec`](https://git.altlinux.org/gears/r/rpm.git?p=rpm.git;a=blob;f=alt/rpm.spec;hb=7e71ed3da438289d916f1cccc9c0a83e594455dc) — построение `setcmp-data` через `rpmquery`.
-10. [Официальный контейнер ALT](https://hub.docker.com/_/alt) — среда воспроизведения `alt:p11`.

Docs/more/usage_setc_rpm.md

@@ -1,80 +0,0 @@
-rg rpmsetcmp:
-
-```
-tools/setcmp.c
-12-{
-13: int cmp = rpmsetcmp(s1, s2);
-14- switch (cmp) {
-
-alt/rpm.spec
-709-_ Mon Nov 25 2019 Andrew Savchenko <bircoph@altlinux.org> 4.13.0.1-alt15
-710:- Support rpmsetcmp profiling on E2K.
-711-
---
-713-- Added triggers circumvention for packagekit offline update (by Aleksei Nikiforov).
-714:- Imported rpmsetcmp optimization from rpm-build.
-715-
---
-1094-- set.c: Increased cache size from 160 to 256 slots, 75 percent hit ratio.
-1095:- set.c: Implemented 4-byte and 8-byte steppers for rpmsetcmp main loop.
-1096-
---
-1314-_ Mon Sep 20 2010 Alexey Tourbin <at@altlinux.ru> 4.0.4-alt98.47
-1315:- set.c (rpmsetcmp): Fixed check for set2 decoding error.
-1316-- brp-cleanup: Updated for /usr/lib64/perl5 and /usr/share/perl5.
-
-lib/rpmds.c
-1087- if (aset && bset) {
-1088: sense = rpmsetcmp(AEVR, BEVR);
-1089- if (sense < -1) {
-
----
-
-rg set_new:
-none
-
----
-
-rg set_add
-none
-
----
-
-rg set_fini:
-none
-
----
-
-rg set_free:
-none
-```
-
----
-
-rg setcmp:
-
-```
-Makefile.am
-218-
-219:noinst_PROGRAMS = setcmp
-220:setcmp_SOURCES = tools/setcmp.c
-221:setcmp_LDADD = lib/librpm.la
-222-
-
-alt/rpm.spec
-311-rpmquery -a --requires |fgrep '= set:' |sort >R
-312:join -o 1.3,2.3 P R |shuf >setcmp-data
-313:time ./setcmp <setcmp-data >/dev/null
-314-rm lib/set.lo lib/librpm.la
---
-318-%if_with profile
-319:time ./setcmp <setcmp-data >/dev/null
-320-rm lib/set.lo lib/librpm.la
-321-%make_build -C lib set.lo librpm.la CFLAGS="$set_c_cflags -fprofile-generate"
-322:./setcmp <setcmp-data >/dev/null
-323-%ifnarch %e2k
---
-332-%make_build
-333:time ./setcmp <setcmp-data >/dev/null
-334-
-```

Docs/more/usage_setc_rpm_build.md

@@ -1,99 +0,0 @@
-rg rpmsetcmp:
-
-```
-tools/setcmp.c
-12-{
-13:    int cmp = rpmsetcmp(s1, s2);
-14-    switch (cmp) {
-
-lib/depends.c
-124-    if (aset && bset) {
-125:	sense = rpmsetcmp(AEVR, BEVR);
-126-	if (sense < -1) {
-
-rpm-4_0.spec
-1231-- set.c: Increased cache size from 160 to 256 slots, 75 percent hit ratio.
-1232:- set.c: Implemented 4-byte and 8-byte steppers for rpmsetcmp main loop.
-1233-
---
-1451-* Mon Sep 20 2010 Alexey Tourbin <at@altlinux.ru> 4.0.4-alt98.47
-1452:- set.c (rpmsetcmp): Fixed check for set2 decoding error.
-1453-- brp-cleanup: Updated for /usr/lib64/perl5 and /usr/share/perl5.
-
-build/reqprov.c
-175-	if (aset && bset) {
-176:	   sense = rpmsetcmp(Aevr, Bevr);
-177-	   if (sense < -1)
-```
-
----
-
-rg set_new:
-
-```
-tools/mkset.c
-11-    assert(bpp <= 32);
-12:    struct set *set = set_new();
-13-    char *line = NULL;
-```
-
----
-
-rg set_add:
-
-```
-tools/mkset.c
-21-	   continue;
-22:	set_add(set, line);
-23-	added++;
-```
-
----
-
-rg set_fini:
-
-```
-tools/mkset.c
-25-    assert(added > 0);
-26:    const char *str = set_fini(set, bpp);
-27-    assert(str);
-```
-
----
-
-rg set_free:
-none
-
----
-
-rg mkset:
-
-```
-rpm-4_0.spec
-431-# set-version helpers
-432:%rpmattr %_rpmlibdir/mkset
-433-%rpmattr %_rpmlibdir/setcmp
---
-1458-- rpmlibprov.c: Added rpmlib(SetVersions) feature.
-1459:- %_rpmlibdir/mkset: Command-line helper for making set-versions.
-1460-- lib.prov: Implemented soname set-versioning with exported symbols.
-
-tools/Makefile.am
-37-	relative \
-38:	mkset \
-39-	setcmp \
---
-62-
-63:mkset_SOURCES = mkset.c
-64-setcmp_SOURCES = setcmp.c
-
-autodeps/lib.prov.in
-112-		Info "$f: $n symbols, $bpp bpp"
-113:		set=$(printf '%s\n' "$sym" |"${RPMB_TOOLS_DIR-@RPMCONFIGDIR@}/mkset" "$bpp")
-114-		printf '%s = %s\n' "$provname" "$set"
-
-autodeps/lib.req.in
-306-	#printf '%s\n' "$reqsym" |LC_ALL=C sort -c -u
-307:	set=$(printf '%s\n' "$reqsym" |"${RPMB_TOOLS_DIR-@RPMCONFIGDIR@}/mkset" "$bpp")
-308-	printf '%s >= %s\n' "$dep" "$set"
-```

Docs/set9_ru.md

@@ -0,0 +1,586 @@
+# Реализация set:version в `set9.c`
+
+Документ описывает экспериментальную реализацию [`reimplement/set9.c`](../reimplement/set9.c).
+Общий смысл механизма set:version и устройство исходного [`set.c`](../set.c) разобраны в
+[`set:version_ru.md`](set:version_ru.md). Здесь основное внимание уделено отличиям `set9.c`:
+потоковому кодированию и декодированию, сортировке, двум LRU-кэшам и
+новому алгоритму сравнения отсортированных множеств.
+
+## Назначение
+
+ALT RPM записывает наборы ELF-символов в версии зависимостей вида:
+
+```text
+libfoo.so.X()(64bit) = set:<encoded-set>
+libfoo.so.X()(64bit) >= set:<encoded-set>
+```
+
+`Provides` содержит хэши предоставляемых символов, а `Requires` — хэши символов, требуемых
+от конкретной библиотеки. `rpmsetcmp()` сравнивает декодированные множества по включению,
+а не как обычные RPM-версии.
+
+`set9.c` сохраняет формат строк и Jenkins OAAT из исходной реализации. Поэтому его задача —
+изменить внутреннее представление и горячие пути, не меняя смысл корректных set-строк.
+Совпадение формата не устраняет фундаментальную вероятность коллизий: сравниваются усечённые
+32-битные хэши, а не исходные имена символов.
+
+## Основные отличия от исходного `set.c`
+
+| Область                   | Исходный `set.c`                                     | `set9.c`                                                         |
+| ------------------------- | ---------------------------------------------------- | ---------------------------------------------------------------- |
+| Хранение имён             | отдельный `xstrdup()` для каждого имени              | общая растущая строковая арена и массив смещений                 |
+| Сортировка                | всегда `qsort()`                                     | `qsort()` для `< 128` элементов, LSD radix sort для остальных    |
+| Кодирование               | отдельные массивы delta, битов и Base62              | один поток `hash -> delta -> Golomb-Rice -> Base62`              |
+| Декодирование             | сложный табличный декодер пар символов               | простой потоковый декодер с 64-битным аккумулятором              |
+| Кэш                       | один кэш первого операнда на 256 записей             | два независимых кэша по 512 записей, по одному на каждый операнд |
+| Поиск в кэше              | линейный просмотр и `memmove()`                      | хэш-бакеты и двусвязный LRU со сменой позиции за `O(1)`          |
+| Нормализация `bpp`        | выполняется для каждого сравнения                    | выполняется при промахе кэша и сохраняется в записи кэша         |
+| Сравнение                 | флаги `ge`/`le`, sentinel-элементы и макросы прыжков | проверка мощности, `memcmp()` и `sorted_subset()`                |
+| Освобождение `struct set` | сама структура не освобождается                      | освобождаются арена, массив символов и сама структура            |
+
+## Публичный API
+
+Как и исходная реализация, файл предоставляет пять функций:
+
+```c
+int rpmsetcmp(const char *set1, const char *set2);
+
+struct set *set_new(void);
+void set_add(struct set *set, const char *sym);
+const char *set_fini(struct set *set, int bpp);
+struct set *set_free(struct set *set);
+```
+
+### `set_new()`
+
+Создаёт пустой `struct set`. Все счётчики и ёмкости устанавливаются в ноль, указатели — в
+`NULL`.
+
+### `set_add()`
+
+Добавляет копию строки символа во внутреннюю строковую арену:
+
+1. массив `symbols_v` увеличивается блоками по 1024 элемента;
+2. строковая арена при первом выделении получает 4096 байт;
+3. при нехватке места ёмкость арены удваивается, пока не вместит новую строку;
+4. в `symbols_v` сохраняются смещение строки в арене и нулевое начальное значение хэша.
+
+В отличие от хранения отдельных указателей, перемещение арены через `xrealloc()` не делает
+записи массива недействительными: в них находятся смещения, а не адреса строк.
+
+### `set_fini()`
+
+Хэширует, сортирует и кодирует добавленные имена. Возвращаемая строка выделена через
+`xstrdup()` и **не содержит** префикс `set:`. Префикс добавляет вызывающий код, например
+`mkset`.
+
+В `set9.c` контракт проверяется через `assert()`:
+
+```text
+set != NULL
+set->cnt > 0
+10 <= bpp <= 32
+```
+
+Это отличается от исходного `set.c`, где пустое множество или недопустимый `bpp` приводят к
+`NULL`. При сборке с `NDEBUG` проверки `assert()` исчезают, поэтому передавать некорректные
+аргументы нельзя.
+
+### `set_free()`
+
+Освобождает строковую арену, массив `symbols_v` и сам `struct set`, затем возвращает `NULL`.
+Типичный вызов:
+
+```c
+set = set_free(set);
+```
+
+### `rpmsetcmp()`
+
+Сравнивает две set-строки. Префикс `set:` у каждого операнда необязателен.
+
+Результаты совместимы с исходным API:
+
+|  Код | Значение                                             |
+| ---: | ---------------------------------------------------- |
+|  `1` | первое множество строго содержит второе              |
+|  `0` | множества равны                                      |
+| `-1` | первое множество строго содержится во втором         |
+| `-2` | множества несравнимы по включению                    |
+| `-3` | ошибка метаданных или декодирования первого операнда |
+| `-4` | ошибка метаданных или декодирования второго операнда |
+
+В типичном RPM-вызове первым операндом остаётся `Provides`, вторым — `Requires`.
+
+## Формат set-строки
+
+После необязательного `set:` строка имеет структуру:
+
+```text
+<bpp><Mshift><payload>
+```
+
+Первые два символа кодируют числа формулой:
+
+```c
+value = character + 7 - 'a';
+character = value - 7 + 'a';
+```
+
+Ограничения:
+
+```text
+10 <= bpp <= 32
+7 <= Mshift <= 31
+Mshift < bpp
+payload не пуст
+```
+
+`payload` использует алфавит:
+
+```text
+0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
+```
+
+Его числовые значения расположены как `0..9`, `a..z`, `A..Z`. Символ `Z` имеет значение
+61 и одновременно служит escape-маркером для исходных шестибитных значений 61, 62 и 63.
+
+## Внутреннее представление создаваемого множества
+
+```c
+struct set {
+    size_t cnt;
+    size_t symbols_cap;
+    size_t strings_len;
+    size_t strings_cap;
+    char *strings;
+    struct symbols {
+        size_t offset;
+        unsigned hash;
+    } *symbols_v;
+};
+```
+
+Здесь:
+
+- `cnt` — число добавленных имён;
+- `symbols_cap` — ёмкость массива метаданных;
+- `strings_len` и `strings_cap` — занятая и выделенная части строковой арены;
+- `strings` — последовательность NUL-терминированных имён;
+- `offset` — начало конкретного имени относительно `strings`;
+- `hash` — хэш, вычисляемый только при `set_fini()`.
+
+Такое представление сокращает число отдельных выделений памяти при построении больших наборов.
+
+## Работа `set_fini()`
+
+Полный путь выглядит так:
+
+```text
+строковая арена и offsets
+  | Jenkins OAAT + mask(bpp)
+  v
+массив struct symbols с хэшами
+  | qsort или LSD radix sort
+  v
+отсортированные хэши
+  | предупреждения о коллизиях + удаление повторов
+  v
+массив уникальных хэшей
+  | потоковый delta + Golomb-Rice + Base62
+  v
+строка <bpp><Mshift><payload>
+```
+
+### Jenkins OAAT и усечение
+
+Для каждого имени вычисляется тот же Jenkins one-at-a-time hash, что и в исходном файле.
+Начальное состояние равно `0x9e3779b9`. Результат ограничивается младшими `bpp` битами:
+
+```c
+unsigned mask = (bpp < 32) ? (1u << bpp) - 1 : ~0u;
+hash_value = hash(symbol) & mask;
+```
+
+### `sort_symbols()`
+
+Для малых наборов (`count < 128`) используется `qsort()`. Для больших наборов применяется
+стабильная LSD radix sort по байтам хэша:
+
+1. число проходов равно `ceil(bpp / 8)`;
+2. на каждом проходе строятся 256 счётчиков/смещений;
+3. элементы стабильно распределяются по текущему байту;
+4. источник и приёмник меняются местами;
+5. если итог оказался во временном массиве, он копируется назад.
+
+Поскольку перед сортировкой хэш уже ограничен `bpp` битами, обработки
+`ceil(bpp / 8)` байтов достаточно. Временный массив размером `count` размещается на стеке.
+
+### Коллизии и повторяющиеся символы
+
+После сортировки соседние равные хэши проверяются попарно. Для разных строк печатается:
+
+```text
+warning: hash collision: <left> <right>
+```
+
+Две одинаковые строки предупреждения не создают. Затем все одинаковые **хэш-значения**
+схлопываются в одно. Поэтому настоящая хэш-коллизия после предупреждения всё равно становится
+одним элементом кодируемого множества — это свойство исходного формата.
+
+### Выбор `Mshift`
+
+Параметр Golomb-Rice вычисляется как:
+
+```text
+Mshift = bpp - floor(log2(count)) - 1
+```
+
+После этого значение ограничивается диапазоном `7..31` и проверяется условие
+`Mshift < bpp`. Идея та же, что в исходной реализации: при примерно равномерном распределении
+хэшей средняя дельта близка к `2^bpp / count`.
+
+### Потоковый `encode_set()`
+
+Исходный `set.c` сначала создаёт массив дельт, затем массив по одному байту на бит и лишь потом
+Base62-строку. В `set9.c` эти стадии объединены.
+
+Для каждого отсортированного хэша вычисляется:
+
+```c
+unsigned delta = current - previous;
+unsigned q = delta >> Mshift;
+unsigned r = delta & ((1u << Mshift) - 1);
+```
+
+В `encode_writer` последовательно добавляются:
+
+```text
+q нулевых битов
+1 — разделитель
+Mshift младших битов r
+```
+
+Все биты идут младшими вперёд (LSB-first), как и в исходном формате.
+
+### `struct encode_writer`
+
+"Писатель" содержит:
+
+- `uint64_t bits` — накопленные ещё не выведенные биты;
+- `filled` — число занятых битов;
+- `escaped` — ожидание второй половины escape-пары;
+- `pending_high` — старшие два бита значения 61, 62 или 63;
+- `output` — текущую позицию в выходной строке.
+
+`encode_writer_put()` добавляет обычное поле битов. `encode_writer_zeros()` добавляет длинную
+unary-последовательность нулей блоками не более 56 бит, чтобы аккумулятор оставался в пределах
+`uint64_t`. `encode_writer_flush()` выводит накопленные Base62-цифры.
+
+В обычном состоянии потребляется шесть потоковых битов. Если значение равно 61, 62 или 63:
+
+1. выводится `Z`;
+2. различие между 61/62/63 сохраняется как два старших бита следующего символа;
+3. из потока для следующего символа берутся только четыре бита.
+
+При завершении неполный символ дополняется нулями. После payload записывается `\0`.
+
+## Метаданные декодируемой строки
+
+Для декодирования используется `struct set_meta`:
+
+```c
+struct set_meta {
+    const char *str;
+    const char *payload;
+    size_t len;
+    size_t payload_len;
+    int bpp;
+    int Mshift;
+    int bit_capacity;
+    int value_capacity;
+};
+```
+
+Подготовка разделена на две части.
+
+### `set_meta_init()` — быстрая проверка
+
+Функция читает только начало строки:
+
+1. проверяет наличие двух метасимволов и хотя бы одного символа payload;
+2. декодирует и проверяет `bpp`;
+3. декодирует и проверяет `Mshift`;
+4. сохраняет `str`, `payload`, `bpp` и `Mshift`.
+
+На этой стадии `strlen()` не вызывается. Это позволяет сначала выполнить дешёвую проверку и
+поискать уже готовую запись в кэше.
+
+### `set_meta_fini()` — вычисление размеров
+
+Функция вызывается только при промахе кэша:
+
+```text
+len            = strlen(str)
+payload_len    = len - 2
+bit_capacity   = payload_len * 6
+value_capacity = bit_capacity / (Mshift + 1)
+```
+
+`bit_capacity` является верхней границей: обычный символ даёт шесть битов, а два символа
+escape-пары дают десять битов, то есть не больше двенадцати. Каждое Golomb-Rice-значение
+требует как минимум `Mshift + 1` бит. Если места нет даже для одного значения,
+строка отклоняется.
+
+## Потоковое декодирование
+
+Путь декодирования объединяет три логических стадии:
+
+```text
+Base62 payload
+  | decode_chunk()
+  v
+6- или 10-битные блоки
+  | q-state + r-state
+  v
+Golomb-Rice delta
+  | previous += delta
+  v
+отсортированные хэши
+```
+
+Отдельные массивы битов и дельт не создаются.
+
+### Таблица `char_to_num[]`
+
+Таблица из 256 элементов классифицирует любой байт:
+
+- `0..60` — обычная Base62-цифра;
+- `61` — `Z`;
+- `0xff` — конец строки;
+- `0xee` — недопустимый символ.
+
+Индекс берётся как `unsigned char`, поэтому байты с установленным старшим битом тоже безопасно
+попадают в диапазон таблицы и отклоняются как недопустимые.
+
+### `decode_chunk()`
+
+Обычный символ со значением меньше 61 возвращает шесть битов. При `Z` функция читает следующий
+символ и:
+
+1. отклоняет конец строки сразу после `Z`;
+2. отклоняет не-Base62 символ;
+3. отклоняет комбинацию старших битов `11`, которая создала бы новый `Z`;
+4. собирает один 10-битный блок из значения 61/62/63 и четырёх младших битов второго символа.
+
+Таким образом escape-пара сразу преобразуется в исходные десять потоковых битов.
+
+### `decode_set()`
+
+Декодер хранит ещё не обработанные биты в `uint64_t bits`, а их количество — в `filled`.
+Для каждого значения он проходит два состояния.
+
+#### Unary-часть `q`
+
+Если аккумулятор пуст, загружается следующий блок. Полностью нулевой блок целиком прибавляется
+к `q`. Иначе `__builtin_ctzll(bits)` находит число нулей до разделительной единицы. Нули и сама
+единица удаляются из аккумулятора.
+
+#### Остаток `r`
+
+Пока доступно меньше `Mshift` битов, загружаются следующие блоки. Затем младшие `Mshift` битов
+образуют остаток, а дельта восстанавливается как:
+
+```c
+unsigned delta = (q << Mshift) | r;
+```
+
+Дельта сразу превращается в исходный хэш:
+
+```c
+previous += delta;
+hash_arr[count++] = previous;
+```
+
+#### Завершение строки
+
+Конец payload допустим в состоянии чтения `q`, если осталось не более пяти нулевых битов
+Base62-дополнения. Более длинный нулевой хвост даёт внутреннюю ошибку `-10`.
+
+Конец строки во время набора `Mshift` битов остатка означает незавершённое значение и даёт
+внутреннюю ошибку `-11`. Ошибки `decode_chunk()` также возвращаются вверх. Публичный
+`rpmsetcmp()` скрывает конкретный внутренний код и преобразует его в `-3` или `-4` в зависимости
+от операнда.
+
+## Нормализация точности: `downsample_set()`
+
+Строки с разным `bpp` нельзя сравнивать напрямую. Обе стороны приводятся к:
+
+```text
+target_bpp = min(bpp1, bpp2)
+```
+
+Один вызов `downsample_set()` уменьшает точность с `bpp + 1` до `bpp`:
+
+1. маска равна `(1u << bpp) - 1`;
+2. бинарным поиском находится первый элемент с удаляемым старшим битом;
+3. исходный отсортированный массив делится на две отсортированные половины;
+4. у второй половины удаляется старший бит;
+5. половины сливаются как два отсортированных массива;
+6. появившиеся после усечения дубликаты удаляются.
+
+Если требуется убрать несколько битов, операция повторяется по одному биту. В `set9.c` результат
+нормализации сохраняется в кэше, поэтому та же строка при том же `target_bpp` не проходит этот
+цикл повторно.
+
+## Два кэша декодированных множеств
+
+`cache_decode_set()` обслуживает два независимых кэша:
+
+```text
+cache_id = 0 — первый операнд
+cache_id = 1 — второй операнд
+```
+
+Каждый кэш содержит до `CACHE_SIZE = 512` записей и 1024 бакета. Разделение не даёт потоку
+часто повторяющихся `Requires` вытеснять кэш `Provides` и наоборот, хотя реальный эффект зависит
+от порядка вызовов решателя зависимостей.
+
+### Ключ записи
+
+Для быстрого предварительного отбора вычисляется fingerprint:
+
+```c
+str[0] | (str[2] << 8) | (str[3] << 16)
+```
+
+Он смешивается с `target_bpp`, после чего выбирается бакет. Совпадения fingerprint недостаточно:
+успешный hit требует также равного `target_bpp` и полного `strcmp()` строки. Поэтому коллизия
+fingerprint влияет только на длину цепочки, но не подменяет декодированное множество.
+
+`target_bpp` входит в ключ, поскольку одна и та же исходная строка может участвовать в
+сравнениях с операндами разной точности и, следовательно, иметь разные нормализованные массивы.
+
+### Устройство записи
+
+Одним `xmalloc()` выделяются:
+
+```text
+struct cache_ent
+массив unsigned для хэшей
+копия исходной строки
+```
+
+Запись хранит fingerprint, исходную строку, число элементов и `target_bpp`.
+
+### Промах
+
+При промахе происходят следующие шаги:
+
+1. `set_meta_fini()` вычисляет длины и верхнюю границу числа значений;
+2. выделяется новая запись;
+3. `decode_set()` создаёт отсортированный массив хэшей;
+4. при необходимости массив поэтапно уменьшается до `target_bpp`;
+5. строка и метаданные копируются в запись;
+6. запись добавляется в хэш-бакет и в начало логического LRU.
+
+### LRU
+
+Все записи одного кэша образуют двусвязный список от `oldest` к `newest`. При hit запись
+отцепляется от текущего места и становится `newest`; это не требует сдвига массива записей.
+При заполненном кэше удаляется `oldest`, включая поиск ссылки на него в соответствующей цепочке
+бакета.
+
+## Сравнение в `rpmsetcmp()`
+
+Последовательность работы:
+
+1. снять `set:`, если он присутствует;
+2. выполнить `set_meta_init()` для обеих строк;
+3. выбрать минимальный `target_bpp`;
+4. получить первый операнд из кэша `0` или декодировать его;
+5. если строки полностью одинаковы, вернуть `0`;
+6. получить второй операнд из кэша `1` или декодировать его;
+7. сравнить мощности и содержимое нормализованных массивов.
+
+Быстрый возврат для одинаковых строк выполняется после полноценного декодирования первого
+операнда. Поэтому равенство строк не позволяет принять некорректный payload без проверки.
+Метаданные второго операнда к этому моменту также уже прошли начальную проверку.
+
+### Равная мощность
+
+Нормализованные массивы отсортированы и не содержат дубликатов. Если `cnt1 == cnt2`, два
+множества могут быть либо равны, либо несравнимы. Поэтому достаточно:
+
+```c
+memcmp(hash_arr1, hash_arr2, cnt1 * sizeof(unsigned))
+```
+
+Совпадение даёт `0`, различие — `-2`.
+
+### Разная мощность
+
+Множество с большим числом уникальных хэшей не может быть строгим подмножеством меньшего.
+Поэтому проверяется только одно возможное направление:
+
+```text
+cnt1 > cnt2: set2 ⊆ set1 ? 1  : -2
+cnt1 < cnt2: set1 ⊆ set2 ? -1 : -2
+```
+
+Саму проверку выполняет `sorted_subset(small, large)`.
+
+### `sorted_subset()`
+
+Алгоритм выбирается по отношению размеров:
+
+```c
+jump = large_count / small_count;
+```
+
+Если `jump < 4`, используется обычное линейное слияние: указатель большого массива двигается
+до текущего элемента малого. Для близких по размеру множеств это последовательный проход с
+хорошей локальностью.
+
+Если `jump >= 4`, вызывается `step_lower_bound()`. Функция сначала делает шаги примерно на
+среднее расстояние между искомыми элементами, затем делит шаг пополам, пока не найдёт первый
+элемент, который не меньше искомого. Это позволяет пропускать части большого массива при
+разреженном малом множестве без sentinel-значений и выхода за границы.
+
+При первом отсутствующем элементе `sorted_subset()` возвращает ложь.
+
+## `SELF_TEST`
+
+При `SELF_TEST` компилируется `main()`, который проверяет только публичный API на коротких
+наборах:
+
+1. строгое надмножество возвращает `1`;
+2. равные множества возвращают `0`;
+3. строгое подмножество возвращает `-1`;
+4. пересекающиеся несравнимые множества возвращают `-2`;
+5. выделенные строки и оба объекта освобождаются.
+
+В отличие от исходного `set.c`, встроенный тест является smoke-тестом, а не полной проверкой совместимости.
+
+## "Карта функций"
+
+| Функция                  | Роль                                         |
+| ------------------------ | -------------------------------------------- |
+| `set_new()`              | создание контейнера символов                 |
+| `set_add()`              | добавление имени в строковую арену           |
+| `set_free()`             | освобождение контейнера                      |
+| `hash()`                 | Jenkins OAAT                                 |
+| `sort_symbols()`         | выбор `qsort()` или radix sort               |
+| `encode_golomb_Mshift()` | вычисление параметра Golomb-Rice             |
+| `encode_writer_*()`      | потоковая упаковка битов в Base62            |
+| `encode_set()`           | объединённые delta, Golomb-Rice и Base62     |
+| `set_fini()`             | полный путь от имён до set-строки            |
+| `set_meta_init()`        | быстрая проверка заголовка строки            |
+| `set_meta_fini()`        | длины и верхние границы буферов              |
+| `decode_chunk()`         | чтение обычного или escape Base62-блока      |
+| `decode_set()`           | объединённые Base62, Golomb-Rice и delta     |
+| `cache_decode_set()`     | два бакетных LRU-кэша                        |
+| `downsample_set()`       | уменьшение точности на один бит              |
+| `step_lower_bound()`     | прыжок и уточнение позиции в большом массиве |
+| `sorted_subset()`        | проверка включения отсортированных множеств  |
+| `rpmsetcmp()`            | публичное сравнение set-строк                |

Docs/test_result.md

@@ -0,0 +1,48 @@
+## original set.c w/out optimizations
+
+функция `decode_set` идёт по пути функций:
+
+1. `decode_base62`
+2. `decode_golomb`
+3. `decode_delta`
+
+| command                     | average_seconds | run1_seconds | run2_seconds | run3_seconds | exit_status |
+| :-------------------------- | :-------------: | :----------: | :----------: | :----------: | ----------: |
+| -s check                    |    4.217343     |   4.310882   |   4.179343   |   4.161804   |           0 |
+| -s autoremove               |    4.884413     |   4.876135   |   4.882347   |   4.894757   |           0 |
+| -s install rpm-build        |    4.634789     |   4.652499   |   4.623077   |   4.628792   |           0 |
+| -s install openuds-server   |    12.704538    |  12.640266   |  12.975482   |  12.497867   |           0 |
+| -s install password-store   |    8.560463     |   8.561515   |   8.557596   |   8.562277   |           0 |
+| --enable-upgrade -s upgrade |    36.271838    |  35.511443   |  35.908641   |  37.395430   |         100 |
+| -s dist-upgrade             |    90.550086    |  94.702198   |  92.174160   |  84.773901   |         100 |
+
+## original set.c
+
+функция `decode_set` идёт по пути функций:
+
+1. `decode_base62_golomb`
+2. `decode_delta`
+
+| command                     | average_seconds | run1_seconds | run2_seconds | run3_seconds | exit_status |
+| :-------------------------- | :-------------: | :----------: | :----------: | :----------: | ----------: |
+| -s check                    |    1.570747     |   1.621496   |   1.545001   |   1.545744   |           0 |
+| -s autoremove               |    2.294502     |   2.312574   |   2.287133   |   2.283798   |           0 |
+| -s install rpm-build        |    2.108121     |   2.142784   |   2.093474   |   2.088105   |           0 |
+| -s install openuds-server   |    8.028098     |   8.147214   |   8.001140   |   7.935940   |           0 |
+| -s install password-store   |    3.373377     |   3.396943   |   3.361290   |   3.361899   |           0 |
+| --enable-upgrade -s upgrade |    33.915979    |  33.895262   |  33.770192   |  34.082484   |         100 |
+| -s dist-upgrade             |    88.722136    |  95.484250   |  85.808085   |  84.874073   |         100 |
+
+## set9.c
+
+Переписанная реализация `set.c`
+
+| command                     | average_seconds | run1_seconds | run2_seconds | run3_seconds | exit_status |
+| :-------------------------- | :-------------: | :----------: | :----------: | :----------: | ----------: |
+| -s check                    |    1.550342     |   1.596307   |   1.532768   |   1.521950   |           0 |
+| -s autoremove               |    2.200188     |   2.237645   |   2.185378   |   2.177540   |           0 |
+| -s install rpm-build        |    2.031055     |   2.045757   |   2.023173   |   2.024234   |           0 |
+| -s install openuds-server   |    7.580058     |   7.572994   |   7.582557   |   7.584622   |           0 |
+| -s install password-store   |    3.337012     |   3.367696   |   3.309826   |   3.333514   |           0 |
+| --enable-upgrade -s upgrade |    31.494775    |  31.535714   |  31.585522   |  31.363090   |         100 |
+| -s dist-upgrade             |    85.326580    |  85.389775   |  85.604566   |  84.985398   |         100 |

README.md

@@ -3,3 +3,112 @@
 Research on ALT Linux RPM set.c and dependency comparison using encoded symbol sets.
 
 - [Docs about current set:version implementation](Docs/set:version.md)
+- [Docs about rewrite implementation (ru)](Docs/set9_ru.md)
+
+# Compare results
+
+| command                     | original set.c | orig set (w/out optim) | set9.c |
+| :-------------------------- | :------------: | :--------------------: | :----: |
+| -s check                    |       x1       |         x3.134         | x1.041 |
+| -s autoremove               |       x1       |         x2.564         | x1.053 |
+| -s install rpm-build        |       x1       |         x2.578         | x1.036 |
+| -s install openuds-server   |       x1       |         x1.946         | x1.094 |
+| -s install password-store   |       x1       |         x2.941         | x1.073 |
+| --enable-upgrade -s upgrade |       x1       |         x1.351         | x1.100 |
+| -s dist-upgrade             |       x1       |         x1.204         | x1.044 |
+
+Более подробные данные сравнения приведены [здесь](Docs/test_result.md)
+
+# Testing
+
+Для проверки используются четыре скрипта из каталога `scripts`.
+
+## Случайные наборы символов
+
+### `scripts/rpmsetinit/compare_random_sets.py`
+
+Проверяет совместимость кодировщиков: компилирует `set.c` и
+`reimplement/newset.c`, генерирует случайные списки символов и сравнивает
+полученные set-строки при случайном `bpp` от 10 до 32.
+
+Пути до наобходимых файлов приведены в коде скрипта.
+Минимальный запуск из корня репозитория:
+
+```sh
+python3 scripts/rpmsetinit/compare_random_sets.py
+```
+
+Нужны Python 3 и C-компилятор `cc`. Скрипт работает непрерывно до `Ctrl-C`;
+входы, на которых результаты различаются, сохраняются в
+`scripts/rpmsetinit/error/`.
+
+### `scripts/rpmsetcmp/compare_random_sets.py`
+
+Проверяет совместимость `rpmsetcmp()` из `set.c` и
+`reimplement/newset.c`. Скрипт генерирует равные, вложенные, несравнимые и
+некорректные множества, проверяет оба направления сравнения и сопоставляет
+коды результатов двух реализаций.
+
+Пути до наобходимых файлов приведены в коде скрипта.
+Минимальный запуск:
+
+```sh
+python3 scripts/rpmsetcmp/compare_random_sets.py
+```
+
+Нужны Python 3 и `cc`. Тест также работает до `Ctrl-C`, а найденные
+расхождения сохраняет в `scripts/rpmsetcmp/error/`.
+
+## Проверка зависимостей Sisyphus
+
+### `scripts/rpmsetcmp/check_sisyphus_set_relations.sh`
+
+Скрипт загружает в изолированный временный APT-каталог метаданные Sisyphus,
+извлекает set:version `Provides` и `Requires`, компилирует компаратор из
+`reimplement/newset.c` и проверяет, что требования пакетов удовлетворяются
+хотя бы одним провайдером. По умолчанию проверяются первые 100 уникальных
+требований; лимит задаётся переменной `REQUIRE_LIMIT` в начале файла.
+`REQUIRE_LIMIT=0` для снятия лимита
+
+Минимальный запуск на ALT Linux с доступом к зеркалу Sisyphus:
+
+```sh
+bash scripts/rpmsetcmp/check_sisyphus_set_relations.sh
+```
+
+Нужны `apt-get`, `apt-cache`, `cc`. В конце выводятся числа совместимых требований,
+требований без провайдера, несовместимых пар и ошибок декодирования.
+
+## Сравнение производительности
+
+### `scripts/run-setc-bench.sh`
+
+Скрипт берёт все `*.c` из `~/setc`, для каждого варианта собирает отдельные
+`rpm-build` и `librpm7` в hasher, подставляя файл как `lib/set.c`. Затем он
+запускает с этой `librpm` одинаковый набор симуляций APT (`check`,
+`autoremove`, `install`, `upgrade`, `dist-upgrade`), записывает коды возврата и
+время трёх прогонов в TSV. При включённых `COLLECT_PERF` и
+`PERF_RECORD` дополнительно создаются результаты `perf stat`, `perf record`,
+`perf report` и `perf annotate`.
+
+Минимальная подготовка:
+
+1. Запускать на ALT Linux p11 с настроенными `gear-hsh`/`hasher` и разрешённым
+   монтированием `/proc`.
+2. Скопировать сравниваемые реализации в `~/setc/`, по одному файлу `*.c` на
+   вариант.
+3. Проверить настройки в начале скрипта: `PACKAGER`, `APT_SOURCE`, `CPU` и
+   каталоги `SETC_DIR`, `WORK_ROOT`, `RESULT_DIR`.
+4. Для запуска без perf установить `COLLECT_PERF=0`; для профилирования
+   оставить `COLLECT_PERF=1`, установить пакет `perf` и разрешить доступ к
+   счётчикам производительности.
+
+Запуск:
+
+```sh
+bash scripts/run-setc-bench.sh
+```
+
+По умолчанию рабочие каталоги создаются в `~/setc-bench`, а итоговые TSV и
+perf-артефакты — в `~/res`. Первый запуск скачивает исходники и собирает RPM,
+поэтому занимает значительно больше времени, чем случайные проверки.

articles/for-alt/plan.md

@@ -0,0 +1,74 @@
+# Работа над `set.c`: краткий план статьи
+
+## 1. Зачем ALT RPM нужны set-строки
+
+Обычная зависимость от версии библиотеки не гарантирует, что в ней остались все нужные программе символы: символ можно удалить, не сменив SONAME, а одинаковые SONAME могут скрывать разные наборы экспортов. Поэтому ALT RPM использует версии зависимостей вида `set:<encoded-set>`. В `Provides` такая строка описывает символы, предоставляемые библиотекой, а в `Requires` - символы, которые конкретный потребитель требует именно от неё. Сравниваются не номера версий, а включение множеств: все хэши из `Requires` должны присутствовать в `Provides`. Гарантия вероятностная, поскольку вместо имён хранятся усечённые хэши, но ответ "символ отсутствует", когда он есть мы не получим (ложноположительное срабатывание)
+
+## 2. Как строится set-строка
+
+Список символов формируется автодепами `rpm-build`. Для `Provides` из ELF извлекаются отфильтрованные экспортируемые базовые имена символов. Для `Requires` `ldd --bindings` связывает каждый сильный неопределённый символ с конкретной библиотекой, после чего для каждого провайдера строится отдельное множество.
+
+Далее `mkset` использует API `set_new()` → `set_add()` → `set_fini()`:
+
+1. для каждого имени вычисляется Jenkins OAAT и оставляются младшие `bpp` бит;
+2. хэши сортируются, повторы удаляются, о коллизиях выдаётся предупреждение;
+3. абсолютные значения заменяются дельтами;
+4. дельты сжимаются кодом Голомба-Райса с параметром `Mshift`;
+5. битовый поток переводится в RPM-безопасную Base62-строку.
+
+При сравнении `rpmsetcmp()` выполняет обратное декодирование, приводит строки с разным `bpp` к общей точности и проверяет включение отсортированных наборов. В рабочем вызове первым операндом выгодно передавать `Provides`: исходный код кэширует прежде всего его декодирование. (переформулировать в сторону кэш есть только для)
+(сказать про подсёт bpp и Mshift на стороне)
+
+## 3. Разбор исходного кода
+
+Работа началась с изучения примерно десятилетнего `lib/set.c`. Код оказался большим по объёму и очень плотным: объединённый Base62/Golomb-декодер, таблица всех пар входных байтов, макросы состояний, прыжки по массиву через defined инструкции и подобие LRU-кэш. Отдельно были прослежены реальные точки вызова в `rpm` и `rpm-build`: во время сборки бинарник `mkset` создаёт строки, а `rpmsetcmp()` через `rpmRangesOverlap()` участвует в проверке зависимостей RPM и APT.
+
+Результатом этого этапа стала документация: описание главных пяти функций API, формата строки, кодирования и декодирования, нормализации `bpp`, кэша, макросов сравнения и встроенного `SELF_TEST`. Таким образом, прежде чем менять алгоритм, для него была построена читаемая модель и зафиксированы его неочевидные инварианты.
+
+## 4. Переписывание и последовательные оптимизации
+
+Сначала алгоритм был перенесён в Python как понятная проверочная реализация. Затем появился читаемый C-вариант, а после него - оптимизированный `set9.c`, сохраняющий старый wire-format и публичный API. В нём были опробованы:
+(добавить про все промежуточные тоже)
+
+- строковая арена вместо отдельного выделения памяти под каждое имя;
+- `qsort()` для малых наборов и radix sort для больших;
+- потоковое кодирование и декодирование без промежуточных массивов битов и дельт; (было и в оригинале, но не было в читаемом)
+- два раздельных bucketed LRU-кэша для `Provides` и `Requires`; (теперь даже LRU, нет постоянных realloc)
+- кэширование результата понижения `bpp`; (надо лучше вспомнить разницу)
+- более простой адаптивный поиск включения в отсортированных массивах;
+- явная проверка метаданных, размеров и кодов внутренних ошибок. (в оригинале по несколько раз)
+
+Не все «читаемые» замены оказались быстрыми. Удаление специальных оптимизаций исходного файла замедляло обычные APT-сценарии примерно в 2–3 раза. `set9.c` вернулся к уровню оригинала и в проведённых симуляциях оказался быстрее/медленне него примерно на 1–10%, однако отдельный анализ показал, что старый слитый декодер и прыгающий проход по массивам всё ещё сильны. Поэтому был собран гибридный `set_frank.c`: новые структуры, encoder и кэш соединены с декодером и сравнением из исходного `set.c`.
+(сказать, что прироста не сильно, читаемость хуже, проверить реальные числа)
+
+## 5. Как проверялись изменения
+
+Тесты строились вокруг инвариантов формата: (логично, его и надо сохранить)
+
+- исходный `SELF_TEST` и расширенные проверки публичного API;
+- побайтовое совпадение результата `set_fini()` старой и новой реализации для `bpp=10…32`; (уточнить, что на ранд данных)
+- дифференциальные случайные тесты равенства, включения, несравнимости и обоих направлений сравнения;
+- повреждённые строки, граничные `bpp`/`Mshift`, разные точности, заполнение и вытеснение кэша;
+- проверки освобождения памяти; (надо добавить про алгоритм, что этого не было, хе)
+- проверка реальных `Provides`/`Requires` из Sisyphus;
+- холодные и прогретые микробенчмарки, затем симуляции и настоящие вызовы ALT APT с проверкой одинакового результата.
+
+(более того, потом всё равно предполагается запуск в сборочнице. вроде.)
+
+(куда-то эти мысли про хэш надо впихнуть)
+Отдельно сравнивались Jenkins OAAT, CityHash и xxHash по скорости и коллизиям на случайных данных и ASCII-строках. Этот эксперимент помог отделить свойства хэш-функции от стоимости остального формата, но простая замена хэша не стала готовым решением: она меняет совместимость строк и не устраняет основную цену декодирования и поиска.
+
+## 6. Альтернативные форматы и результаты
+
+Параллельно были проверены варианты, не совместимые со старым форматом.
+
+**Roaring Bitmap.** Создание bitmap могло быть быстрым, но равномерно распределённые хэши плохо подходят для такого представления. Для 1000 символов при `bpp=32` строка выросла с 3994 до 19924 символов; холодное сравнение было примерно в 4,5 раза, прогретое - в 76 раз медленнее `set9`. Zstd уменьшал строку до 14126 символов, но добавлял ещё больше работы при сравнении. Вариант оставлен как отрицательный эксперимент.
+
+**Прямой массив хэшей в Base64.** Отказ от delta/Golomb ускорял отдельные микротесты: в одном прогоне холодное сравнение занимало `0,87×`, а прогретое `0,35×` времени `set9`, при росте строки с 3994 до 5340 символов. Но полный поток реальных пар Sisyphus показал обратное (эту стороны надо добить) - текущая реализация была примерно в 1,4 раза медленнее. На отдельном синтетическом APT-графе resolver, напротив, ускорялся на 7–18%, зато `gencaches` замедлялся примерно на 12%. Кроме того, стандартный Base64 использует padding `=` (да, я забыл это пофиксить), который RPM запрещает внутри версии зависимости. Поэтому прямой формат интересен как компромисс между размером и стоимостью декодирования, но в текущем виде не готов заменить set-строки.
+
+**Другие направления.** Рассматривались отрицательный Bloom-подобный prefilter с обязательной точной проверкой, адаптивный индекс для больших `Provides`, Elias–Fano и глобальные идентификаторы символов. Общий вывод: (в это возможно стоит углубиться и сделать его)
+
+## 7. Итого
+
+Суть работы была в понимании "магии" кода. Сначала были восстановлены назначение, формат и реальные пути использования; создана документация и читаемая реализация; после этого каждая оптимизация проверялась на совместимость, безопасность и скорость.
+Эксперименты показали, что код содержит оправданные низкоуровневые решения, а локально более простой или быстрый формат не обязательно выигрывает на полном потоке RPM/APT (still check). Наиболее практичное продолжение - (что-то)

articles/for-asvk/plan.md

@@ -0,0 +1,241 @@
+# Исследование алгоритма разрешения зависимостей в ALT RPM
+
+## Замысел и структура статьи
+
+Статья будет как решение одной формальной задачи. В ALT RPM пакет-потребитель задаёт множество
+требуемых символов $R$ (`Requires`), а пакет-поставщик - множество предоставляемых
+символов $P$ (`Provides`). Зависимость выполнима тогда и только тогда, когда
+
+$$
+R \subseteq P.
+$$
+
+Явное перечисление тысяч ELF-, Python- и файловых символов раздувает RPM-метаданные,
+поэтому множества представлены компактными set-строками.
+
+## 1. Постановка задачи и данные ALT Linux
+
+Ввести обозначения для bpp(b), множеств, хэш функций(H), длин строк и т.д.
+
+Проверяемое программой условие имеет вид
+
+$$
+H_b(R)\subseteq H_b(P).
+$$
+
+Отмечаем вероятностную природу: из $R\subseteq P$ следует
+$H_b(R)\subseteq H_b(P)$, поэтому коллизия не создаёт ложного отказа. Обратное
+неверно: отсутствующий символ может совпасть по хешу с символом из $P$ и дать
+ложное принятие зависимости.
+
+(Представить данные ниже табличкой)
+Привести срез Sisyphus `x86_64 + noarch`: 47 367 пакетов, 14 761 `Provides`-отношение и 68 857
+`Requires`-отношений. Для 68 198 реально сопоставимых пар медианы равны
+$p=480$, $r=13$, $p/r=28.1$; p75 - 1886, 37 и 80; p90 - 6219, 104 и 257.
+В 92.4% пар выполняется $p/r\ge 4$. Это к теме, почему рассматриваем разреженные множества.
+
+## 2. Устройство существующей `set:`-строки
+
+Даём схему формата:
+
+```
+массив строк
+  | (Jenkins OAAT)
+  v
+массив хэшей
+  | (qsort)
+  v
+отсортированный массив хэшей
+  | (вычисление разницы между элементами)
+  v
+массив delta
+  | (Rice-Golomb преобразование)
+  v
+битовый массив
+  | (base62 преобразование)
+  v
+set-строка
+```
+
+Нужно отдельно разобрать заголовок, диапазоны `bpp` и `Mshift`, внешний префикс
+`set:` и роль Base62: строка должна оставаться допустимым токеном версии RPM.
+Полезна небольшая иллюстрация на 5-8 символах: исходные имена, хеши,
+отсортированный массив, дельты, биты кода и итоговая строка.
+Но. Это много альт-специфики, может настолько глубоко не стоит (хотя и не очень глубоко)
+
+### 2.1. Хеширование и вероятность коллизий
+
+! Всё ещё нет нормального анализа коллизий
+
+Текущий код использует 32-битный Jenkins OAAT и оставляет $b$ бит. Для модели
+равномерного независимого хеширования ожидаемое число столкнувшихся пар среди
+$n$ различных символов равно
+
+(будет мудрёная формула, теория)
+
+В статье формулы дополняем разными табличками по реальным данным из Sisyphus: распределения $p,r,b$, ожидаемые коллизии, реальные.
+
+(Другие хэши идут далее)
+
+### 2.2. Кодирование Golomb-Rice
+
+Для отсортированных значений $0\le x_1<\dots<x_n<N$ определить
+$\delta_1=x_1$, $\delta_i=x_i-x_{i-1}$.
+
+Занимаеи столько и столько бит. В реализации параметр выбирается из
+$b$ и $n$ приблизительно как
+
+$$
+m=b-\lfloor\log_2 n\rfloor-1
+$$
+
+с ограничением допустимым диапазоном формата. При средней дельте
+$\mathbb E\delta\approx 2^b/n$ это даёт ожидаемую длину порядка
+
+...
+
+### 2.3. Base62
+
+Пояснить экзотичность Base62, его алфавит, принцип кодирования, Z-escape,
+
+## 3. Построение `set:`-строк в `set.c`
+
+(фактически его можно в 2 упихать)
+
+Рассказать путь `set_new()` - `set_add()` - `set_fini()` и отдельно объяснить
+сложные участки:
+
+(а тут и нет сложного, всё в cmp запихали)
+Можно рассказать о слитом энкодере, но это есть в переписанной части
+
+В оригинальном варианте используется `qsort()`
+
+Записать итоговую сложность построения:
+
+...
+
+И доп памяти:
+
+...
+
+## 4. Сравнение `set:`-строк
+
+### 4.1. Обратное декодирование
+
+Кратко пройти обратную цепочку:
+
+```
+set-строка
+  | (обратное base62 преобразование)
+  v
+битовый массив
+  | (обратное Rice-Golomb преобразование)
+  v
+массив delta
+  | (вычисление изначальных значений)
+  v
+массив хэшей
+```
+
+Если `bpp` различается, более точное множество проецируется на меньшее число бит,
+после чего снова удаляются дубликаты. (btw, тут есть сортировка слиянием, интересный момент)
+
+### 4.2. Кэш декодированных множеств
+
+В исходном `set.c` кэш содержит 256 записей (`CACHE_SIZE=256`), при вытеснении
+использует позицию 243 (`PIVOT_SIZE=243`), `free()` и два `memmove()`. Ключ включает
+короткий fingerprint, а окончательная проверка требует сравнения строки.
+
+- что именно кэшируется;
+- стоимость линейного поиска, `strcmp`, перемещений и аллокаций;
+- слабое повторное использование при потоке десятков тысяч различных пар;
+
+(всё надо сильно короче расписывать)
+
+### 4.3. Проверка включения и «прыжок» по массивам
+
+После декодирования задача сводится к двум возрастающим массивам. Базовый вариант
+
+рассказ про IFLT4 и IFLT8
+
+Худшая асимптотика остаётся
+
+$$
+T_{\subseteq}(p,r)=O(p+r),
+$$
+
+Полная сложность алгоритма холодного сравнения:
+
+...
+
+При попадании в кэш:
+
+...
+
+## 5. Оптимизации с сохранением текущего формата
+
+### 5.1. Новый кэш
+
+Заменить небольшой кэш на два раздельных bucketed LRU для `Provides` и `Requires`, увеличить вместимость. Хранить длину и более сильный fingerprint, избегать лишних
+`strlen`/`strcmp`, `free` и полных `memmove`. Отдельно убрать повторные `realloc` при построении входных наборов.
+Проверить cold, warm, поток меняющихся `Requires` к одному `Provides` и поток
+полностью уникальных пар.
+
+### 5.2. Radix sort
+
+Для больших наборов заменить $O(n\log n)$ `qsort()` стабильной LSD radix sort по
+байтам хеша. Число проходов
+
+$$
+k=\left\lceil\frac b8\right\rceil,
+\qquad T_{\mathrm{radix}}=O(kn),
+$$
+
+дополнительная память $O(n+256)$. Для малых $n$ оставить `qsort()`; в `set9.c`
+порог равен 128.
+
+### 5.3. Оптимальный проход по массивам
+
+Сделать адаптивный алгоритм:
+
+- равные мощности - `memcmp` и быстрый вывод «равны/несравнимы»;
+- близкие мощности - обычное линейное слияние;
+- $p/r$ велико - galloping/jump search с монотонной нижней границей;
+- немедленный выход при первом отсутствующем хеше.
+
+## 6. Варианты со сменой формата или API
+
+### 6.1. Roaring Bitmap - отрицательный результат
+
+Равномерно распределённые хеши не образуют плотных диапазонов, ради которых создан
+Roaring. Для 1000 символов при `bpp=32` получено:
+
+| Представление        | Длина строки | Cold compare | Warm compare |
+| -------------------- | -----------: | -----------: | -----------: |
+| Golomb/Base62 `set9` |         3994 |   293.87 мкс |     5.24 мкс |
+| raw Roaring/hex      |       19 924 |  1321.17 мкс |   399.20 мкс |
+
+То есть строка длиннее в 4.99 раза, cold-сравнение медленнее в 4.50 раза, warm -
+в 76.14 раза. Zstd уменьшал строку до 14 126 символов, но оставлял её в 3.54 раза
+длиннее `set9`; в соответствующем прогоне cold- и warm-сравнение проиграли в 4.96
+и 101.86 раза. Этот вариант нужен в статье как важный отрицательный эксперимент.
+
+### 6.2. Прямые хеши и Base64
+
+В формате D1 отсортированные усечённые хеши хранятся фиксированной шириной без
+дельт и Golomb-кода, затем упаковываются в Base64. Теоретическая длина payload:
+
+...
+
+Для 1000 символов и `bpp=32` строка выросла с 3994 до 5340
+символов (+33.7%). Приложить разные результаты тестов. (тесты бы прогнать снова)
+
+Отдельно про увеличение размера и влияние на apt команды от этого
+
+выводы насчёт формата
+
+### 6.3. Альтернативные хэш функции
+
+тут про разные хэши, их быстрдействие и тесты коллизий
+
+## 8. Вывод

hash_testing/new_testing/corpus_bit_distribution.py

@@ -0,0 +1,385 @@
+#!/usr/bin/env python3
+"""Measure P(hash output bit = 1) over a line-oriented ASCII corpus."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import hashlib
+import subprocess
+import tempfile
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+from statistics import fmean
+from typing import TextIO
+
+from plot_probability_map import ProbabilityRow, render_bit_probabilities
+from probability_map import HASHES, HASH_FUNCS_DIR
+
+ROOT = Path(__file__).resolve().parent
+DEFAULT_OUTPUT_DIR = ROOT / "corpus_bit_distribution"
+
+
+class DistributionError(RuntimeError):
+    """Raised when a corpus or hash adapter cannot be processed."""
+
+
+@dataclass(frozen=True)
+class HashSpec:
+    bits: int
+    expression: str
+
+
+HASH_SPECS = {
+    "jenkinsOAAT": HashSpec(32, "jenkins_oaat(word, length)"),
+    "xxh64": HashSpec(64, "xxh64(word, length, XXH64_SEED)"),
+    "t1ha2": HashSpec(64, "t1ha2_atonce(word, length, T1HA2_SEED)"),
+}
+
+
+@dataclass(frozen=True)
+class Distribution:
+    hash_name: str
+    samples: int
+    bits: int
+    ones: tuple[int, ...]
+
+    def __post_init__(self) -> None:
+        if self.samples < 1:
+            raise ValueError("число образцов должно быть положительным")
+        if self.bits < 1 or len(self.ones) != self.bits:
+            raise ValueError("число счётчиков должно совпадать с шириной хэша")
+        if any(count < 0 or count > self.samples for count in self.ones):
+            raise ValueError("счётчик единиц должен находиться между 0 и samples")
+
+    @property
+    def probabilities(self) -> tuple[float, ...]:
+        return tuple(count / self.samples for count in self.ones)
+
+
+def c_include_path(path: Path) -> str:
+    """Return a C string-safe absolute include path."""
+    return str(path.resolve()).replace("\\", "\\\\").replace('"', '\\"')
+
+
+def make_adapter_source(source: Path, spec: HashSpec) -> str:
+    """Build a C program that aggregates bit counts using an existing hash source."""
+    return f'''#define _POSIX_C_SOURCE 200809L
+#define main arsv_original_hash_cli_main
+#include "{c_include_path(source)}"
+#undef main
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/types.h>
+
+int main(void)
+{{
+    char *buffer = NULL;
+    size_t capacity = 0;
+    uint64_t samples = UINT64_C(0);
+    uint64_t ones[{spec.bits}] = {{UINT64_C(0)}};
+    ssize_t bytes_read;
+
+    while ((bytes_read = getline(&buffer, &capacity, stdin)) >= 0) {{
+        size_t length = (size_t)bytes_read;
+        const unsigned char *word = (const unsigned char *)buffer;
+
+        while (length > 0U && is_ascii_trailing_space(word[length - 1U])) {{
+            --length;
+        }}
+        if (length == 0U) {{
+            fprintf(stderr, "corpus contains an empty line\\n");
+            free(buffer);
+            return EXIT_FAILURE;
+        }}
+        for (size_t index = 0; index < length; ++index) {{
+            if (word[index] > 0x7fU) {{
+                fprintf(stderr, "corpus must contain ASCII characters only\\n");
+                free(buffer);
+                return EXIT_FAILURE;
+            }}
+        }}
+        if (samples == UINT64_MAX) {{
+            fprintf(stderr, "sample counter overflow\\n");
+            free(buffer);
+            return EXIT_FAILURE;
+        }}
+
+        uint64_t hash = (uint64_t)({spec.expression});
+        for (unsigned int bit = 0U; bit < {spec.bits}U; ++bit) {{
+            ones[bit] += (hash >> bit) & UINT64_C(1);
+        }}
+        ++samples;
+    }}
+    if (ferror(stdin)) {{
+        fprintf(stderr, "failed to read corpus\\n");
+        free(buffer);
+        return EXIT_FAILURE;
+    }}
+    free(buffer);
+
+    printf("%" PRIu64 ",{spec.bits}", samples);
+    for (unsigned int bit = 0U; bit < {spec.bits}U; ++bit) {{
+        printf(",%" PRIu64, ones[bit]);
+    }}
+    putchar('\\n');
+    return EXIT_SUCCESS;
+}}
+'''
+
+
+def measure_hash(
+    hash_name: str,
+    corpus: Path,
+    hash_funcs_dir: Path = HASH_FUNCS_DIR,
+) -> Distribution:
+    """Compile a batch adapter and count set output bits for every corpus line."""
+    try:
+        spec = HASH_SPECS[hash_name]
+    except KeyError as error:
+        supported = ", ".join(HASH_SPECS)
+        raise DistributionError(
+            f"нет batch-адаптера для {hash_name!r}; доступны: {supported}"
+        ) from error
+
+    source = hash_funcs_dir / hash_name / "bin_hash.c"
+    if not source.is_file():
+        raise DistributionError(f"не найден исходник хэша: {source}")
+    if not corpus.is_file():
+        raise DistributionError(f"не найден corpus: {corpus}")
+
+    with tempfile.TemporaryDirectory(prefix=f"arsv-{hash_name}-") as temporary:
+        root = Path(temporary)
+        adapter_source = root / "batch_counts.c"
+        adapter = root / "batch_counts"
+        adapter_source.write_text(make_adapter_source(source, spec), encoding="utf-8")
+        command = [
+            "cc",
+            "-std=c11",
+            "-O3",
+            "-Wall",
+            "-Wextra",
+            "-Wpedantic",
+            "-Werror",
+            str(adapter_source),
+            "-o",
+            str(adapter),
+        ]
+        compiled = subprocess.run(command, text=True, capture_output=True, check=False)
+        if compiled.returncode != 0:
+            details = compiled.stderr.strip() or compiled.stdout.strip()
+            raise DistributionError(f"не удалось собрать адаптер {hash_name}: {details}")
+
+        try:
+            with corpus.open("rb") as stream:
+                measured = subprocess.run(
+                    [str(adapter)],
+                    stdin=stream,
+                    text=True,
+                    capture_output=True,
+                    check=False,
+                )
+        except OSError as error:
+            raise DistributionError(f"не удалось прочитать {corpus}: {error}") from error
+        if measured.returncode != 0:
+            details = measured.stderr.strip() or measured.stdout.strip()
+            raise DistributionError(f"batch-прогон {hash_name} завершился ошибкой: {details}")
+
+    fields = measured.stdout.strip().split(",")
+    if len(fields) != spec.bits + 2:
+        raise DistributionError(
+            f"batch-прогон {hash_name} вернул {len(fields)} полей вместо {spec.bits + 2}"
+        )
+    try:
+        samples = int(fields[0])
+        bits = int(fields[1])
+        ones = tuple(int(value) for value in fields[2:])
+    except ValueError as error:
+        raise DistributionError(
+            f"batch-прогон {hash_name} вернул некорректные счётчики"
+        ) from error
+    if bits != spec.bits:
+        raise DistributionError(
+            f"batch-прогон {hash_name} сообщил ширину {bits} вместо {spec.bits}"
+        )
+    try:
+        return Distribution(hash_name, samples, bits, ones)
+    except ValueError as error:
+        raise DistributionError(f"некорректный результат {hash_name}: {error}") from error
+
+
+def write_distribution_csv(stream: TextIO, distribution: Distribution) -> None:
+    """Write bit counts and probabilities from LSB to MSB."""
+    writer = csv.writer(stream, lineterminator="\n")
+    writer.writerow(["bit", "ones", "zeros", "probability", "deviation_from_0.5"])
+    for bit, (ones, probability) in enumerate(
+        zip(distribution.ones, distribution.probabilities, strict=True)
+    ):
+        writer.writerow(
+            [
+                bit,
+                ones,
+                distribution.samples - ones,
+                f"{probability:.9f}",
+                f"{abs(probability - 0.5):.9f}",
+            ]
+        )
+
+
+def write_summary_csv(stream: TextIO, distributions: Sequence[Distribution]) -> None:
+    writer = csv.writer(stream, lineterminator="\n")
+    writer.writerow(
+        [
+            "hash",
+            "samples",
+            "bits",
+            "mean_probability",
+            "mean_abs_deviation",
+            "max_abs_deviation",
+            "min_probability",
+            "max_probability",
+        ]
+    )
+    for distribution in distributions:
+        probabilities = distribution.probabilities
+        deviations = [abs(value - 0.5) for value in probabilities]
+        writer.writerow(
+            [
+                distribution.hash_name,
+                distribution.samples,
+                distribution.bits,
+                f"{fmean(probabilities):.9f}",
+                f"{fmean(deviations):.9f}",
+                f"{max(deviations):.9f}",
+                f"{min(probabilities):.9f}",
+                f"{max(probabilities):.9f}",
+            ]
+        )
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as stream:
+        for block in iter(lambda: stream.read(1024 * 1024), b""):
+            digest.update(block)
+    return digest.hexdigest()
+
+
+def write_manifest(
+    path: Path,
+    corpus: Path,
+    distributions: Sequence[Distribution],
+) -> None:
+    samples = {distribution.samples for distribution in distributions}
+    if len(samples) != 1:
+        raise DistributionError("хэши обработали разное число строк corpus")
+    content = "\n".join(
+        [
+            f"corpus={corpus}",
+            f"corpus_bytes={corpus.stat().st_size}",
+            f"corpus_sha256={sha256_file(corpus)}",
+            f"samples={samples.pop()}",
+            f"hashes={','.join(item.hash_name for item in distributions)}",
+            "metric=P(hash_output_bit=1)",
+            "bit_order=bit_0_is_LSB",
+            "input_format=one_ASCII_symbol_per_line",
+            "",
+        ]
+    )
+    path.write_text(content, encoding="utf-8")
+
+
+def run(
+    corpus: Path,
+    output_directory: Path,
+    hash_names: Sequence[str],
+) -> list[Distribution]:
+    """Measure all requested hashes and create CSV, manifest, and one PNG."""
+    names = list(dict.fromkeys(hash_names))
+    if not names:
+        raise DistributionError("нужно указать хотя бы один хэш")
+
+    distributions = [measure_hash(name, corpus) for name in names]
+    sample_counts = {item.samples for item in distributions}
+    if len(sample_counts) != 1:
+        raise DistributionError("хэши обработали разное число строк corpus")
+
+    output_directory.mkdir(parents=True, exist_ok=True)
+    for distribution in distributions:
+        path = output_directory / f"{distribution.hash_name}.csv"
+        with path.open("w", encoding="utf-8", newline="") as stream:
+            write_distribution_csv(stream, distribution)
+
+    with (output_directory / "summary.csv").open(
+        "w", encoding="utf-8", newline=""
+    ) as stream:
+        write_summary_csv(stream, distributions)
+    write_manifest(output_directory / "manifest.txt", corpus, distributions)
+
+    rows = [
+        ProbabilityRow(
+            operation=item.hash_name,
+            pairs=item.samples,
+            probabilities=list(item.probabilities),
+        )
+        for item in distributions
+    ]
+    render_bit_probabilities(
+        rows,
+        output_directory / "bit_distribution.png",
+        "C++ Provides corpus: P(hash output bit = 1)",
+    )
+    return distributions
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Считает прямое распределение единиц по выходным битам хэша "
+            "для ASCII corpus (одна строка — одно значение). bit_0 — младший бит."
+        )
+    )
+    parser.add_argument("corpus", type=Path, help="текстовый corpus")
+    parser.add_argument(
+        "--output",
+        type=Path,
+        default=DEFAULT_OUTPUT_DIR,
+        help=f"отдельный каталог результатов (по умолчанию: {DEFAULT_OUTPUT_DIR})",
+    )
+    parser.add_argument(
+        "--hash",
+        dest="hashes",
+        action="append",
+        help="проверить указанный хэш; можно повторять (по умолчанию все)",
+    )
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    parser = build_parser()
+    arguments = parser.parse_args(argv)
+    try:
+        distributions = run(
+            corpus=arguments.corpus,
+            output_directory=arguments.output,
+            hash_names=arguments.hashes or HASHES,
+        )
+    except (DistributionError, OSError, ValueError) as error:
+        parser.error(str(error))
+
+    for distribution in distributions:
+        probabilities = distribution.probabilities
+        mean_deviation = fmean(abs(value - 0.5) for value in probabilities)
+        print(
+            f"hash={distribution.hash_name} samples={distribution.samples} "
+            f"bits={distribution.bits} mean_abs_deviation={mean_deviation:.9f}"
+        )
+    print(f"wrote {arguments.output}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

hash_testing/new_testing/extract_provided_symbols.py

@@ -0,0 +1,426 @@
+#!/usr/bin/env python3
+"""Extract ALT RPM Provides symbol names into a hash-testing corpus.
+
+Local RPM paths are processed directly.  Other positional arguments are resolved
+as binary package names through the ALT Repository Database (RDB).
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import shutil
+import stat
+import subprocess
+import sys
+import tempfile
+import urllib.parse
+import urllib.request
+from dataclasses import dataclass
+from pathlib import Path
+from typing import BinaryIO, Sequence, cast
+
+
+RDB_BASE = "https://rdb.altlinux.org/api"
+NEWC_MAGICS = {b"070701", b"070702"}
+NEWC_HEADER_SIZE = 110
+COPY_CHUNK_SIZE = 1024 * 1024
+MAX_CPIO_NAME_SIZE = 1024 * 1024
+
+
+class CorpusError(RuntimeError):
+    """A user-facing extraction error."""
+
+
+@dataclass(frozen=True)
+class PackageInput:
+    label: str
+    path: Path
+
+
+@dataclass(frozen=True)
+class PackageSymbols:
+    label: str
+    elf_files: int
+    symbols: frozenset[str]
+
+
+def parse_arguments(argv: Sequence[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Extract canonical ALT rpm-build Provided ELF symbols from multiple "
+            "RPM packages, one symbol per output line."
+        )
+    )
+    parser.add_argument(
+        "packages",
+        nargs="+",
+        help="local .rpm path or ALT binary package name",
+    )
+    parser.add_argument(
+        "-o",
+        "--output",
+        default="-",
+        help="output file (default: stdout)",
+    )
+    parser.add_argument(
+        "--cpp-only",
+        action="store_true",
+        help="keep only Itanium C++ ABI mangled names beginning with _Z",
+    )
+    parser.add_argument(
+        "--branch",
+        default="p11",
+        help="ALT branch for package-name resolution (default: p11)",
+    )
+    parser.add_argument(
+        "--arch",
+        default="x86_64",
+        help="binary package architecture (default: x86_64)",
+    )
+    parser.add_argument(
+        "--rpm2cpio",
+        default=shutil.which("rpm2cpio") or "rpm2cpio",
+        help=argparse.SUPPRESS,
+    )
+    parser.add_argument(
+        "--provided-symbols",
+        default=(
+            "/usr/lib/rpm/provided_symbols"
+            if Path("/usr/lib/rpm/provided_symbols").is_file()
+            else shutil.which("provided_symbols") or "provided_symbols"
+        ),
+        help=argparse.SUPPRESS,
+    )
+    parser.add_argument(
+        "--file-command",
+        default=shutil.which("file") or "file",
+        help=argparse.SUPPRESS,
+    )
+    return parser.parse_args(argv)
+
+
+def read_json(url: str) -> dict[str, object]:
+    try:
+        with urllib.request.urlopen(url, timeout=60) as response:
+            return json.load(response)
+    except (OSError, ValueError, json.JSONDecodeError) as error:
+        raise CorpusError(f"failed to read ALT RDB response from {url}: {error}") from error
+
+
+def download_package(name: str, branch: str, arch: str, directory: Path) -> PackageInput:
+    query = urllib.parse.urlencode({"branch": branch, "name": name, "arch": arch})
+    metadata_url = f"{RDB_BASE}/site/pkghash_by_binary_name?{query}"
+    metadata = read_json(metadata_url)
+    package_hash = metadata.get("pkghash")
+    if not isinstance(package_hash, str) or not package_hash:
+        raise CorpusError(f"ALT RDB did not return a package hash for {name!r}")
+
+    download_url = (
+        f"{RDB_BASE}/site/package_downloads_bin/{package_hash}?"
+        f"{urllib.parse.urlencode({'branch': branch, 'arch': arch})}"
+    )
+    download_data = read_json(download_url)
+    downloads = download_data.get("downloads")
+    if not isinstance(downloads, list):
+        raise CorpusError(f"ALT RDB did not return downloads for {name!r}")
+
+    package_record: dict[str, object] | None = None
+    for architecture_record in downloads:
+        if not isinstance(architecture_record, dict):
+            continue
+        if architecture_record.get("arch") != arch:
+            continue
+        packages = architecture_record.get("packages")
+        if isinstance(packages, list):
+            for candidate in packages:
+                if isinstance(candidate, dict):
+                    package_record = candidate
+                    break
+        if package_record is not None:
+            break
+
+    if package_record is None:
+        raise CorpusError(f"ALT RDB has no {arch} RPM download for {name!r}")
+
+    filename = package_record.get("name")
+    url = package_record.get("url")
+    expected_md5 = package_record.get("md5")
+    if not isinstance(filename, str) or not filename.endswith(".rpm"):
+        raise CorpusError(f"ALT RDB returned an invalid RPM filename for {name!r}")
+    if not isinstance(url, str) or not url.startswith(("https://", "http://")):
+        raise CorpusError(f"ALT RDB returned an invalid RPM URL for {name!r}")
+
+    destination = directory / filename
+    temporary = destination.with_suffix(destination.suffix + ".part")
+    digest = hashlib.md5(usedforsecurity=False)
+    try:
+        with urllib.request.urlopen(url, timeout=180) as response, temporary.open("wb") as output:
+            while chunk := response.read(COPY_CHUNK_SIZE):
+                output.write(chunk)
+                digest.update(chunk)
+    except OSError as error:
+        temporary.unlink(missing_ok=True)
+        raise CorpusError(f"failed to download {name!r} from {url}: {error}") from error
+
+    if isinstance(expected_md5, str) and digest.hexdigest().lower() != expected_md5.lower():
+        temporary.unlink(missing_ok=True)
+        raise CorpusError(f"MD5 mismatch for downloaded package {name!r}")
+    temporary.replace(destination)
+    return PackageInput(f"{name} ({filename})", destination)
+
+
+def resolve_packages(
+    specifications: Sequence[str], branch: str, arch: str, directory: Path
+) -> list[PackageInput]:
+    resolved: list[PackageInput] = []
+    seen: set[tuple[str, str]] = set()
+    for specification in specifications:
+        candidate = Path(specification).expanduser()
+        if candidate.is_file():
+            package = PackageInput(candidate.name, candidate.resolve())
+        elif candidate.exists():
+            raise CorpusError(f"package input is not a regular file: {candidate}")
+        elif "/" in specification or specification.endswith(".rpm"):
+            raise CorpusError(f"RPM file does not exist: {candidate}")
+        else:
+            package = download_package(specification, branch, arch, directory)
+
+        identity = (package.label, str(package.path))
+        if identity not in seen:
+            resolved.append(package)
+            seen.add(identity)
+    return resolved
+
+
+def read_exact(stream: BinaryIO, size: int, context: str) -> bytes:
+    chunks: list[bytes] = []
+    remaining = size
+    while remaining:
+        chunk = stream.read(remaining)
+        if not chunk:
+            raise CorpusError(f"truncated newc stream while reading {context}")
+        chunks.append(chunk)
+        remaining -= len(chunk)
+    return b"".join(chunks)
+
+
+def discard_exact(stream: BinaryIO, size: int, context: str) -> None:
+    remaining = size
+    while remaining:
+        chunk = stream.read(min(remaining, COPY_CHUNK_SIZE))
+        if not chunk:
+            raise CorpusError(f"truncated newc stream while reading {context}")
+        remaining -= len(chunk)
+
+
+def copy_exact(stream: BinaryIO, output: BinaryIO, size: int, context: str) -> None:
+    remaining = size
+    while remaining:
+        chunk = stream.read(min(remaining, COPY_CHUNK_SIZE))
+        if not chunk:
+            raise CorpusError(f"truncated newc stream while reading {context}")
+        output.write(chunk)
+        remaining -= len(chunk)
+
+
+def parse_newc_header(header: bytes) -> tuple[int, int, int]:
+    if len(header) != NEWC_HEADER_SIZE or header[:6] not in NEWC_MAGICS:
+        raise CorpusError("rpm2cpio output is not a valid newc archive")
+    try:
+        fields = [int(header[6 + index * 8 : 14 + index * 8], 16) for index in range(13)]
+    except ValueError as error:
+        raise CorpusError("newc header contains a non-hexadecimal field") from error
+    mode = fields[1]
+    file_size = fields[6]
+    name_size = fields[11]
+    if name_size < 1 or name_size > MAX_CPIO_NAME_SIZE:
+        raise CorpusError(f"invalid newc pathname size: {name_size}")
+    return mode, file_size, name_size
+
+
+def extract_elf_members(stream: BinaryIO, directory: Path) -> list[Path]:
+    elf_files: list[Path] = []
+    member_index = 0
+    while True:
+        header = read_exact(stream, NEWC_HEADER_SIZE, "header")
+        mode, file_size, name_size = parse_newc_header(header)
+        raw_name = read_exact(stream, name_size, "pathname")
+        if raw_name[-1:] != b"\0":
+            raise CorpusError("newc pathname is not NUL-terminated")
+        discard_exact(stream, -(NEWC_HEADER_SIZE + name_size) % 4, "pathname padding")
+
+        if raw_name[:-1] == b"TRAILER!!!":
+            discard_exact(stream, file_size, "trailer data")
+            discard_exact(stream, -file_size % 4, "trailer padding")
+            break
+
+        member_index += 1
+        prefix_size = min(file_size, 4)
+        prefix = read_exact(stream, prefix_size, "member data")
+        remaining = file_size - prefix_size
+        if stat.S_ISREG(mode) and prefix == b"\x7fELF":
+            destination = directory / f"elf-{member_index:06d}"
+            with destination.open("wb") as output:
+                output.write(prefix)
+                copy_exact(stream, output, remaining, "ELF member data")
+            elf_files.append(destination)
+        else:
+            discard_exact(stream, remaining, "member data")
+        discard_exact(stream, -file_size % 4, "member padding")
+    return elf_files
+
+
+def extract_package_symbols(
+    package: PackageInput,
+    rpm2cpio: str,
+    provided_symbols: str,
+    file_command: str,
+    directory: Path,
+) -> PackageSymbols:
+    package_directory = directory / f"package-{len(list(directory.iterdir())):04d}"
+    package_directory.mkdir()
+    process = subprocess.Popen(
+        [rpm2cpio, str(package.path)],
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    )
+    assert process.stdout is not None
+    assert process.stderr is not None
+    try:
+        extracted_elf_files = extract_elf_members(
+            cast(BinaryIO, process.stdout), package_directory
+        )
+    except Exception:
+        process.kill()
+        process.communicate()
+        raise
+    finally:
+        process.stdout.close()
+
+    stderr = process.stderr.read().decode(errors="replace")
+    return_code = process.wait()
+    if return_code != 0:
+        raise CorpusError(
+            f"rpm2cpio failed for {package.label} with status {return_code}: {stderr.strip()}"
+        )
+
+    elf_files: list[Path] = []
+    environment = os.environ.copy()
+    environment["LC_ALL"] = "C"
+    for path in extracted_elf_files:
+        result = subprocess.run(
+            [file_command, "--brief", "--", str(path)],
+            capture_output=True,
+            text=True,
+            env=environment,
+            check=False,
+        )
+        if result.returncode != 0:
+            raise CorpusError(
+                f"file failed for {package.label} with status "
+                f"{result.returncode}: {result.stderr.strip()}"
+            )
+        description = f" {result.stdout.strip()} "
+        if (
+            " ELF " in description
+            and " shared object, " in description
+            and " shared object, no machine, " not in description
+        ):
+            elf_files.append(path)
+
+    if not elf_files:
+        return PackageSymbols(package.label, 0, frozenset())
+
+    result = subprocess.run(
+        [provided_symbols, *(str(path) for path in elf_files)],
+        capture_output=True,
+        text=True,
+        env=environment,
+        check=False,
+    )
+    if result.returncode != 0:
+        raise CorpusError(
+            f"provided_symbols failed for {package.label} with status "
+            f"{result.returncode}: {result.stderr.strip()}"
+        )
+    symbols = frozenset(line for line in result.stdout.splitlines() if line)
+    return PackageSymbols(package.label, len(elf_files), symbols)
+
+
+def render_symbols(symbols: set[str]) -> str:
+    if not symbols:
+        return ""
+    return "\n".join(sorted(symbols)) + "\n"
+
+
+def write_output(path: str, content: str) -> None:
+    if path == "-":
+        sys.stdout.write(content)
+        return
+    destination = Path(path).expanduser()
+    destination.parent.mkdir(parents=True, exist_ok=True)
+    with tempfile.NamedTemporaryFile(
+        "w", encoding="utf-8", dir=destination.parent, delete=False
+    ) as temporary:
+        temporary.write(content)
+        temporary_path = Path(temporary.name)
+    temporary_path.replace(destination)
+
+
+def run(arguments: argparse.Namespace) -> int:
+    with tempfile.TemporaryDirectory(prefix="provided-symbols-") as temporary_name:
+        temporary = Path(temporary_name)
+        download_directory = temporary / "downloads"
+        extraction_directory = temporary / "extract"
+        download_directory.mkdir()
+        extraction_directory.mkdir()
+        packages = resolve_packages(
+            arguments.packages, arguments.branch, arguments.arch, download_directory
+        )
+
+        combined: set[str] = set()
+        total_elf_files = 0
+        for package in packages:
+            package_symbols = extract_package_symbols(
+                package,
+                arguments.rpm2cpio,
+                arguments.provided_symbols,
+                arguments.file_command,
+                extraction_directory,
+            )
+            selected = {
+                symbol
+                for symbol in package_symbols.symbols
+                if not arguments.cpp_only or symbol.startswith("_Z")
+            }
+            combined.update(selected)
+            total_elf_files += package_symbols.elf_files
+            print(
+                f"package={package_symbols.label} "
+                f"elf_files={package_symbols.elf_files} "
+                f"symbols={len(package_symbols.symbols)} selected={len(selected)}",
+                file=sys.stderr,
+            )
+
+        write_output(arguments.output, render_symbols(combined))
+        print(
+            f"packages={len(packages)} elf_files={total_elf_files} "
+            f"unique_symbols={len(combined)} output={arguments.output}",
+            file=sys.stderr,
+        )
+    return 0
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    try:
+        return run(parse_arguments(argv))
+    except (CorpusError, OSError, subprocess.SubprocessError) as error:
+        print(f"error: {error}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

hash_testing/new_testing/generate_input.py

@@ -0,0 +1,264 @@
+#!/usr/bin/env python3
+"""Generate unique words similar to a given word by applying random mutations."""
+
+from __future__ import annotations
+
+import argparse
+import random
+import string
+import sys
+from collections.abc import Callable, Sequence
+from pathlib import Path
+
+DEFAULT_ALPHABET = string.ascii_letters + string.digits + "_"
+
+OPERATION_ALIASES = {
+    "1": "replace",
+    "replace": "replace",
+    "2": "delete",
+    "delete": "delete",
+    "3": "add",
+    "add": "add",
+    "4": "swap",
+    "swap": "swap",
+    "5": "case",
+    "case": "case",
+    "6": "first",
+    "first": "first",
+    "7": "last",
+    "last": "last",
+}
+
+OPERATION_HELP = """операция изменения:
+  1, replace  заменить случайный символ
+  2, delete   удалить случайный символ
+  3, add      добавить символ в случайную позицию
+  4, swap     переставить два соседних символа
+  5, case     сменить регистр случайного символа
+  6, first    изменить первый символ
+  7, last     изменить последний символ"""
+
+
+class MutationError(ValueError):
+    """Raised when the selected mutation cannot be applied."""
+
+
+def different_character(current: str, alphabet: str, rng: random.Random) -> str:
+    choices = [character for character in alphabet if character != current]
+    if not choices:
+        raise MutationError("алфавит не содержит символа, отличного от заменяемого")
+    return rng.choice(choices)
+
+
+def replace_character(word: str, alphabet: str, rng: random.Random) -> str:
+    if not word:
+        raise MutationError("нельзя заменить символ в пустом слове")
+    index = rng.randrange(len(word))
+    replacement = different_character(word[index], alphabet, rng)
+    return word[:index] + replacement + word[index + 1 :]
+
+
+def delete_character(word: str, _alphabet: str, rng: random.Random) -> str:
+    if not word:
+        raise MutationError("нельзя удалить символ из пустого слова")
+    index = rng.randrange(len(word))
+    return word[:index] + word[index + 1 :]
+
+
+def add_character(word: str, alphabet: str, rng: random.Random) -> str:
+    index = rng.randrange(len(word) + 1)
+    return word[:index] + rng.choice(alphabet) + word[index:]
+
+
+def swap_adjacent(word: str, _alphabet: str, rng: random.Random) -> str:
+    indexes = [
+        index for index in range(len(word) - 1) if word[index] != word[index + 1]
+    ]
+    if not indexes:
+        raise MutationError(
+            "для перестановки нужны хотя бы два соседних различных символа"
+        )
+    index = rng.choice(indexes)
+    return word[:index] + word[index + 1] + word[index] + word[index + 2 :]
+
+
+def change_case(word: str, _alphabet: str, rng: random.Random) -> str:
+    indexes = []
+    replacements: dict[int, str] = {}
+    for index, character in enumerate(word):
+        swapped = character.swapcase()
+        if swapped != character and len(swapped) == 1:
+            indexes.append(index)
+            replacements[index] = swapped
+    if not indexes:
+        raise MutationError("в слове нет символов, у которых можно сменить регистр")
+    index = rng.choice(indexes)
+    return word[:index] + replacements[index] + word[index + 1 :]
+
+
+def change_first(word: str, alphabet: str, rng: random.Random) -> str:
+    if not word:
+        raise MutationError("нельзя изменить первый символ пустого слова")
+    replacement = different_character(word[0], alphabet, rng)
+    return replacement + word[1:]
+
+
+def change_last(word: str, alphabet: str, rng: random.Random) -> str:
+    if not word:
+        raise MutationError("нельзя изменить последний символ пустого слова")
+    replacement = different_character(word[-1], alphabet, rng)
+    return word[:-1] + replacement
+
+
+Mutation = Callable[[str, str, random.Random], str]
+MUTATIONS: dict[str, Mutation] = {
+    "replace": replace_character,
+    "delete": delete_character,
+    "add": add_character,
+    "swap": swap_adjacent,
+    "case": change_case,
+    "first": change_first,
+    "last": change_last,
+}
+
+
+def parse_operation(value: str) -> str:
+    try:
+        return OPERATION_ALIASES[value.lower()]
+    except KeyError as error:
+        valid = ", ".join(OPERATION_ALIASES)
+        raise argparse.ArgumentTypeError(
+            f"неизвестная операция {value!r}; допустимы: {valid}"
+        ) from error
+
+
+def generate_words(
+    source: str,
+    count: int,
+    operation: str,
+    operation_count: int,
+    alphabet: str = DEFAULT_ALPHABET,
+    seed: int | None = None,
+    max_attempts: int | None = None,
+) -> list[str]:
+    """Generate up to ``count`` unique mutations made by exactly N operations."""
+    if count < 1:
+        raise ValueError("количество выходных слов должно быть положительным")
+    if operation_count < 1:
+        raise ValueError("количество операций должно быть положительным")
+    if not alphabet:
+        raise ValueError("алфавит не должен быть пустым")
+
+    mutation = MUTATIONS[operation]
+    rng = random.Random(seed)
+    attempt_limit = max_attempts or max(10_000, count * 1_000)
+    words: list[str] = []
+    seen: set[str] = set()
+
+    for _attempt in range(attempt_limit):
+        candidate = source
+        try:
+            for _ in range(operation_count):
+                candidate = mutation(candidate, alphabet, rng)
+        except MutationError:
+            continue
+
+        if candidate != source and candidate not in seen:
+            seen.add(candidate)
+            words.append(candidate)
+            if len(words) == count:
+                return words
+
+    return words
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Создаёт список уникальных слов, похожих на исходное. "
+            "Каждое слово получается независимо от исходного ровно заданным "
+            "числом случайных операций."
+        ),
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=(
+            f"{OPERATION_HELP}\n\n"
+            "Пример:\n"
+            "  python3 nexus.py example -o swap -n 20 -k 2 --seed 42"
+        ),
+    )
+    parser.add_argument("word", help="исходное слово")
+    parser.add_argument(
+        "-o", "--operation", required=True, type=parse_operation, help=OPERATION_HELP
+    )
+    parser.add_argument(
+        "-n",
+        "--count",
+        type=int,
+        default=10,
+        help="верхняя граница числа уникальных выходных слов (по умолчанию: 10)",
+    )
+    parser.add_argument(
+        "-k",
+        "--operations",
+        type=int,
+        default=1,
+        help="число операций над каждым словом (по умолчанию: 1)",
+    )
+    parser.add_argument(
+        "--alphabet",
+        default=DEFAULT_ALPHABET,
+        help=("символы для добавления и замены " f"(по умолчанию: {DEFAULT_ALPHABET})"),
+    )
+    parser.add_argument(
+        "--seed", type=int, help="seed генератора для воспроизводимого результата"
+    )
+    parser.add_argument(
+        "--max-attempts",
+        type=int,
+        help="предельное число попыток собрать уникальные слова",
+    )
+    parser.add_argument(
+        "--output",
+        type=Path,
+        help="записать слова в файл вместо стандартного вывода",
+    )
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    if args.max_attempts is not None and args.max_attempts < 1:
+        parser.error("--max-attempts должен быть положительным")
+
+    try:
+        words = generate_words(
+            source=args.word,
+            count=args.count,
+            operation=args.operation,
+            operation_count=args.operations,
+            alphabet=args.alphabet,
+            seed=args.seed,
+            max_attempts=args.max_attempts,
+        )
+    except (MutationError, ValueError) as error:
+        parser.error(str(error))
+
+    if len(words) < args.count:
+        print(
+            f"warning: operation={args.operation}: generated {len(words)} "
+            f"of at most {args.count} unique words",
+            file=sys.stderr,
+        )
+
+    output = "".join(f"{word}\n" for word in words)
+    if args.output:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        args.output.write_text(output, encoding="utf-8")
+    else:
+        sys.stdout.write(output)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

hash_testing/new_testing/hash_funcs/jenkinsOAAT/bin_hash.c

@@ -0,0 +1,109 @@
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#define JOAAT_SEED UINT32_C(0x9e3779b9)
+
+static uint32_t jenkins_oaat(const unsigned char *data, size_t length)
+{
+    uint32_t hash = JOAAT_SEED;
+
+    for (size_t index = 0; index < length; ++index) {
+        hash += data[index];
+        hash += hash << 10;
+        hash ^= hash >> 6;
+    }
+
+    hash += hash << 3;
+    hash ^= hash >> 11;
+    hash += hash << 15;
+    return hash;
+}
+
+static int is_ascii_trailing_space(unsigned char character)
+{
+    return character == ' ' || character == '\t' || character == '\n' ||
+           character == '\r' || character == '\v' || character == '\f';
+}
+
+static int read_stdin(unsigned char **data, size_t *length)
+{
+    size_t capacity = 256;
+    unsigned char *buffer = malloc(capacity);
+
+    if (buffer == NULL) {
+        return -1;
+    }
+
+    *length = 0;
+    for (;;) {
+        size_t available = capacity - *length;
+        size_t bytes_read = fread(buffer + *length, 1, available, stdin);
+        *length += bytes_read;
+
+        if (bytes_read < available) {
+            if (ferror(stdin)) {
+                free(buffer);
+                return -1;
+            }
+            break;
+        }
+
+        if (capacity > SIZE_MAX / 2) {
+            free(buffer);
+            return -1;
+        }
+        capacity *= 2;
+
+        unsigned char *larger_buffer = realloc(buffer, capacity);
+        if (larger_buffer == NULL) {
+            free(buffer);
+            return -1;
+        }
+        buffer = larger_buffer;
+    }
+
+    *data = buffer;
+    return 0;
+}
+
+int main(int argc, char **argv)
+{
+    const unsigned char *word;
+    unsigned char *stdin_buffer = NULL;
+    size_t length;
+
+    if (argc > 2) {
+        fprintf(stderr, "usage: %s [ASCII_WORD]\n", argv[0]);
+        return EXIT_FAILURE;
+    }
+
+    if (argc == 2) {
+        word = (const unsigned char *)argv[1];
+        length = strlen(argv[1]);
+    } else {
+        if (read_stdin(&stdin_buffer, &length) != 0) {
+            fprintf(stderr, "failed to read input\n");
+            return EXIT_FAILURE;
+        }
+        word = stdin_buffer;
+    }
+
+    while (length > 0 && is_ascii_trailing_space(word[length - 1])) {
+        --length;
+    }
+
+    for (size_t index = 0; index < length; ++index) {
+        if (word[index] > 0x7f) {
+            fprintf(stderr, "input must contain ASCII characters only\n");
+            free(stdin_buffer);
+            return EXIT_FAILURE;
+        }
+    }
+
+    printf("%08" PRIx32 "\n", jenkins_oaat(word, length));
+    free(stdin_buffer);
+    return EXIT_SUCCESS;
+}

hash_testing/new_testing/hash_funcs/jenkinsOAAT/hash.c

@@ -0,0 +1,13 @@
+unsigned int hash(const char* str) {
+  unsigned int hash = 0x9e3779b9;
+  const unsigned char* p = (const unsigned char*)str;
+  while (*p) {
+    hash += *p++;
+    hash += (hash << 10);
+    hash ^= (hash >> 6);
+  }
+  hash += (hash << 3);
+  hash ^= (hash >> 11);
+  hash += (hash << 15);
+  return hash;
+}

hash_testing/new_testing/hash_funcs/t1ha2/bin_hash.c

@@ -0,0 +1,113 @@
+/*
+ * Standalone t1ha2_atonce command-line wrapper for avalanche testing.
+ *
+ * Upstream: https://gitflic.ru/project/erthink/t1ha
+ * Commit: 00eb779b6c042ccd831ec2f1ae757409c73f39f6
+ * Algorithm: t1ha2_atonce(data, length, seed=0), stable portable 64-bit mode.
+ *
+ * The vendored upstream implementation is licensed under the zlib License;
+ * see upstream/LICENSE. This wrapper is an altered integration file and is not
+ * represented as an original upstream source file.
+ */
+
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#define T1HA0_DISABLED
+#define T1HA1_DISABLED
+#define T1HA_SYS_UNALIGNED_ACCESS 0
+#define T1HA_USE_FAST_ONESHOT_READ 0
+#include "upstream/src/t1ha2.c"
+
+#define T1HA2_SEED UINT64_C(0)
+
+static int is_ascii_trailing_space(unsigned char character)
+{
+    return character == ' ' || character == '\t' || character == '\n' ||
+           character == '\r' || character == '\v' || character == '\f';
+}
+
+static int read_stdin(unsigned char **data, size_t *length)
+{
+    size_t capacity = 256;
+    unsigned char *buffer = malloc(capacity);
+
+    if (buffer == NULL) {
+        return -1;
+    }
+
+    *length = 0;
+    for (;;) {
+        size_t available = capacity - *length;
+        size_t bytes_read = fread(buffer + *length, 1, available, stdin);
+        *length += bytes_read;
+
+        if (bytes_read < available) {
+            if (ferror(stdin)) {
+                free(buffer);
+                return -1;
+            }
+            break;
+        }
+
+        if (capacity > SIZE_MAX / 2U) {
+            free(buffer);
+            return -1;
+        }
+        capacity *= 2U;
+
+        {
+            unsigned char *larger_buffer = realloc(buffer, capacity);
+            if (larger_buffer == NULL) {
+                free(buffer);
+                return -1;
+            }
+            buffer = larger_buffer;
+        }
+    }
+
+    *data = buffer;
+    return 0;
+}
+
+int main(int argc, char **argv)
+{
+    const unsigned char *word;
+    unsigned char *stdin_buffer = NULL;
+    size_t length;
+
+    if (argc > 2) {
+        fprintf(stderr, "usage: %s [ASCII_WORD]\n", argv[0]);
+        return EXIT_FAILURE;
+    }
+
+    if (argc == 2) {
+        word = (const unsigned char *)argv[1];
+        length = strlen(argv[1]);
+    } else {
+        if (read_stdin(&stdin_buffer, &length) != 0) {
+            fprintf(stderr, "failed to read input\n");
+            return EXIT_FAILURE;
+        }
+        word = stdin_buffer;
+    }
+
+    while (length > 0U && is_ascii_trailing_space(word[length - 1U])) {
+        --length;
+    }
+
+    for (size_t index = 0; index < length; ++index) {
+        if (word[index] > 0x7fU) {
+            fprintf(stderr, "input must contain ASCII characters only\n");
+            free(stdin_buffer);
+            return EXIT_FAILURE;
+        }
+    }
+
+    printf("%016" PRIx64 "\n", t1ha2_atonce(word, length, T1HA2_SEED));
+    free(stdin_buffer);
+    return EXIT_SUCCESS;
+}

hash_testing/new_testing/hash_funcs/t1ha2/upstream/LICENSE

@@ -0,0 +1,23 @@
+ zlib License, see https://en.wikipedia.org/wiki/Zlib_License
+
+ Copyright (c) 2016-2020 Positive Technologies, https://www.ptsecurity.com,
+ Fast Positive Hash.
+
+ Portions Copyright (c) 2010-2013 Leonid Yuriev <leo@yuriev.ru>,
+ The 1Hippeus project (t1h).
+
+ This software is provided 'as-is', without any express or implied
+ warranty. In no event will the authors be held liable for any damages
+ arising from the use of this software.
+
+ Permission is granted to anyone to use this software for any purpose,
+ including commercial applications, and to alter it and redistribute it
+ freely, subject to the following restrictions:
+
+ 1. The origin of this software must not be misrepresented; you must not
+    claim that you wrote the original software. If you use this software
+    in a product, an acknowledgement in the product documentation would be
+    appreciated but is not required.
+ 2. Altered source versions must be plainly marked as such, and must not be
+    misrepresented as being the original software.
+ 3. This notice may not be removed or altered from any source distribution.

hash_testing/new_testing/hash_funcs/t1ha2/upstream/src/t1ha2.c

@@ -0,0 +1,383 @@
+/*
+ *  Copyright (c) 2016-2020 Positive Technologies, https://www.ptsecurity.com,
+ *  Fast Positive Hash.
+ *
+ *  Portions Copyright (c) 2010-2020 Leonid Yuriev <leo@yuriev.ru>,
+ *  The 1Hippeus project (t1h).
+ *
+ *  This software is provided 'as-is', without any express or implied
+ *  warranty. In no event will the authors be held liable for any damages
+ *  arising from the use of this software.
+ *
+ *  Permission is granted to anyone to use this software for any purpose,
+ *  including commercial applications, and to alter it and redistribute it
+ *  freely, subject to the following restrictions:
+ *
+ *  1. The origin of this software must not be misrepresented; you must not
+ *     claim that you wrote the original software. If you use this software
+ *     in a product, an acknowledgement in the product documentation would be
+ *     appreciated but is not required.
+ *  2. Altered source versions must be plainly marked as such, and must not be
+ *     misrepresented as being the original software.
+ *  3. This notice may not be removed or altered from any source distribution.
+ */
+
+/*
+ * t1ha = { Fast Positive Hash, aka "Позитивный Хэш" }
+ * by [Positive Technologies](https://www.ptsecurity.ru)
+ *
+ * Briefly, it is a 64-bit Hash Function:
+ *  1. Created for 64-bit little-endian platforms, in predominantly for x86_64,
+ *     but portable and without penalties it can run on any 64-bit CPU.
+ *  2. In most cases up to 15% faster than City64, xxHash, mum-hash, metro-hash
+ *     and all others portable hash-functions (which do not use specific
+ *     hardware tricks).
+ *  3. Not suitable for cryptography.
+ *
+ * The Future will (be) Positive. Всё будет хорошо.
+ *
+ * ACKNOWLEDGEMENT:
+ * The t1ha was originally developed by Leonid Yuriev (Леонид Юрьев)
+ * for The 1Hippeus project - zerocopy messaging in the spirit of Sparta!
+ */
+
+#ifndef T1HA2_DISABLED
+#include "t1ha_bits.h"
+#include "t1ha_selfcheck.h"
+
+static __always_inline void init_ab(t1ha_state256_t *s, uint64_t x,
+                                    uint64_t y) {
+  s->n.a = x;
+  s->n.b = y;
+}
+
+static __always_inline void init_cd(t1ha_state256_t *s, uint64_t x,
+                                    uint64_t y) {
+  s->n.c = rot64(y, 23) + ~x;
+  s->n.d = ~y + rot64(x, 19);
+}
+
+/* TODO: C++ template in the next version */
+#define T1HA2_UPDATE(ENDIANNES, ALIGNESS, state, v)                            \
+  do {                                                                         \
+    t1ha_state256_t *const s = state;                                          \
+    const uint64_t w0 = fetch64_##ENDIANNES##_##ALIGNESS(v + 0);               \
+    const uint64_t w1 = fetch64_##ENDIANNES##_##ALIGNESS(v + 1);               \
+    const uint64_t w2 = fetch64_##ENDIANNES##_##ALIGNESS(v + 2);               \
+    const uint64_t w3 = fetch64_##ENDIANNES##_##ALIGNESS(v + 3);               \
+                                                                               \
+    const uint64_t d02 = w0 + rot64(w2 + s->n.d, 56);                          \
+    const uint64_t c13 = w1 + rot64(w3 + s->n.c, 19);                          \
+    s->n.d ^= s->n.b + rot64(w1, 38);                                          \
+    s->n.c ^= s->n.a + rot64(w0, 57);                                          \
+    s->n.b ^= prime_6 * (c13 + w2);                                            \
+    s->n.a ^= prime_5 * (d02 + w3);                                            \
+  } while (0)
+
+static __always_inline void squash(t1ha_state256_t *s) {
+  s->n.a ^= prime_6 * (s->n.c + rot64(s->n.d, 23));
+  s->n.b ^= prime_5 * (rot64(s->n.c, 19) + s->n.d);
+}
+
+/* TODO: C++ template in the next version */
+#define T1HA2_LOOP(ENDIANNES, ALIGNESS, state, data, len)                      \
+  do {                                                                         \
+    const void *detent = (const uint8_t *)data + len - 31;                     \
+    do {                                                                       \
+      const uint64_t *v = (const uint64_t *)data;                              \
+      data = (const uint64_t *)data + 4;                                       \
+      prefetch(data);                                                          \
+      T1HA2_UPDATE(le, ALIGNESS, state, v);                                    \
+    } while (likely(data < detent));                                           \
+  } while (0)
+
+/* TODO: C++ template in the next version */
+#define T1HA2_TAIL_AB(ENDIANNES, ALIGNESS, state, data, len)                   \
+  do {                                                                         \
+    t1ha_state256_t *const s = state;                                          \
+    const uint64_t *v = (const uint64_t *)data;                                \
+    switch (len) {                                                             \
+    default:                                                                   \
+      mixup64(&s->n.a, &s->n.b, fetch64_##ENDIANNES##_##ALIGNESS(v++),         \
+              prime_4);                                                        \
+    /* fall through */                                                         \
+    case 24:                                                                   \
+    case 23:                                                                   \
+    case 22:                                                                   \
+    case 21:                                                                   \
+    case 20:                                                                   \
+    case 19:                                                                   \
+    case 18:                                                                   \
+    case 17:                                                                   \
+      mixup64(&s->n.b, &s->n.a, fetch64_##ENDIANNES##_##ALIGNESS(v++),         \
+              prime_3);                                                        \
+    /* fall through */                                                         \
+    case 16:                                                                   \
+    case 15:                                                                   \
+    case 14:                                                                   \
+    case 13:                                                                   \
+    case 12:                                                                   \
+    case 11:                                                                   \
+    case 10:                                                                   \
+    case 9:                                                                    \
+      mixup64(&s->n.a, &s->n.b, fetch64_##ENDIANNES##_##ALIGNESS(v++),         \
+              prime_2);                                                        \
+    /* fall through */                                                         \
+    case 8:                                                                    \
+    case 7:                                                                    \
+    case 6:                                                                    \
+    case 5:                                                                    \
+    case 4:                                                                    \
+    case 3:                                                                    \
+    case 2:                                                                    \
+    case 1:                                                                    \
+      mixup64(&s->n.b, &s->n.a, tail64_##ENDIANNES##_##ALIGNESS(v, len),       \
+              prime_1);                                                        \
+    /* fall through */                                                         \
+    case 0:                                                                    \
+      return final64(s->n.a, s->n.b);                                          \
+    }                                                                          \
+  } while (0)
+
+/* TODO: C++ template in the next version */
+#define T1HA2_TAIL_ABCD(ENDIANNES, ALIGNESS, state, data, len)                 \
+  do {                                                                         \
+    t1ha_state256_t *const s = state;                                          \
+    const uint64_t *v = (const uint64_t *)data;                                \
+    switch (len) {                                                             \
+    default:                                                                   \
+      mixup64(&s->n.a, &s->n.d, fetch64_##ENDIANNES##_##ALIGNESS(v++),         \
+              prime_4);                                                        \
+    /* fall through */                                                         \
+    case 24:                                                                   \
+    case 23:                                                                   \
+    case 22:                                                                   \
+    case 21:                                                                   \
+    case 20:                                                                   \
+    case 19:                                                                   \
+    case 18:                                                                   \
+    case 17:                                                                   \
+      mixup64(&s->n.b, &s->n.a, fetch64_##ENDIANNES##_##ALIGNESS(v++),         \
+              prime_3);                                                        \
+    /* fall through */                                                         \
+    case 16:                                                                   \
+    case 15:                                                                   \
+    case 14:                                                                   \
+    case 13:                                                                   \
+    case 12:                                                                   \
+    case 11:                                                                   \
+    case 10:                                                                   \
+    case 9:                                                                    \
+      mixup64(&s->n.c, &s->n.b, fetch64_##ENDIANNES##_##ALIGNESS(v++),         \
+              prime_2);                                                        \
+    /* fall through */                                                         \
+    case 8:                                                                    \
+    case 7:                                                                    \
+    case 6:                                                                    \
+    case 5:                                                                    \
+    case 4:                                                                    \
+    case 3:                                                                    \
+    case 2:                                                                    \
+    case 1:                                                                    \
+      mixup64(&s->n.d, &s->n.c, tail64_##ENDIANNES##_##ALIGNESS(v, len),       \
+              prime_1);                                                        \
+    /* fall through */                                                         \
+    case 0:                                                                    \
+      return final128(s->n.a, s->n.b, s->n.c, s->n.d, extra_result);           \
+    }                                                                          \
+  } while (0)
+
+static __always_inline uint64_t final128(uint64_t a, uint64_t b, uint64_t c,
+                                         uint64_t d, uint64_t *h) {
+  mixup64(&a, &b, rot64(c, 41) ^ d, prime_0);
+  mixup64(&b, &c, rot64(d, 23) ^ a, prime_6);
+  mixup64(&c, &d, rot64(a, 19) ^ b, prime_5);
+  mixup64(&d, &a, rot64(b, 31) ^ c, prime_4);
+  *h = c + d;
+  return a ^ b;
+}
+
+//------------------------------------------------------------------------------
+
+uint64_t t1ha2_atonce(const void *data, size_t length, uint64_t seed) {
+  t1ha_state256_t state;
+  init_ab(&state, seed, length);
+
+#if T1HA_SYS_UNALIGNED_ACCESS == T1HA_UNALIGNED_ACCESS__EFFICIENT
+  if (unlikely(length > 32)) {
+    init_cd(&state, seed, length);
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+    T1HA2_LOOP(le, unaligned, &state, data, length);
+    squash(&state);
+    length &= 31;
+  }
+  T1HA2_TAIL_AB(le, unaligned, &state, data, length);
+#else
+  const bool misaligned = (((uintptr_t)data) & (ALIGNMENT_64 - 1)) != 0;
+  if (misaligned) {
+    if (unlikely(length > 32)) {
+      init_cd(&state, seed, length);
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+      T1HA2_LOOP(le, unaligned, &state, data, length);
+      squash(&state);
+      length &= 31;
+    }
+    T1HA2_TAIL_AB(le, unaligned, &state, data, length);
+  } else {
+    if (unlikely(length > 32)) {
+      init_cd(&state, seed, length);
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+      T1HA2_LOOP(le, aligned, &state, data, length);
+      squash(&state);
+      length &= 31;
+    }
+    T1HA2_TAIL_AB(le, aligned, &state, data, length);
+  }
+#endif
+}
+
+uint64_t t1ha2_atonce128(uint64_t *__restrict extra_result,
+                         const void *__restrict data, size_t length,
+                         uint64_t seed) {
+  t1ha_state256_t state;
+  init_ab(&state, seed, length);
+  init_cd(&state, seed, length);
+
+#if T1HA_SYS_UNALIGNED_ACCESS == T1HA_UNALIGNED_ACCESS__EFFICIENT
+  if (unlikely(length > 32)) {
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+    T1HA2_LOOP(le, unaligned, &state, data, length);
+    length &= 31;
+  }
+  T1HA2_TAIL_ABCD(le, unaligned, &state, data, length);
+#else
+  const bool misaligned = (((uintptr_t)data) & (ALIGNMENT_64 - 1)) != 0;
+  if (misaligned) {
+    if (unlikely(length > 32)) {
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+      T1HA2_LOOP(le, unaligned, &state, data, length);
+      length &= 31;
+    }
+    T1HA2_TAIL_ABCD(le, unaligned, &state, data, length);
+  } else {
+    if (unlikely(length > 32)) {
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+      T1HA2_LOOP(le, aligned, &state, data, length);
+      length &= 31;
+    }
+    T1HA2_TAIL_ABCD(le, aligned, &state, data, length);
+  }
+#endif
+}
+
+//------------------------------------------------------------------------------
+
+void t1ha2_init(t1ha_context_t *ctx, uint64_t seed_x, uint64_t seed_y) {
+  init_ab(&ctx->state, seed_x, seed_y);
+  init_cd(&ctx->state, seed_x, seed_y);
+  ctx->partial = 0;
+  ctx->total = 0;
+}
+
+void t1ha2_update(t1ha_context_t *__restrict ctx, const void *__restrict data,
+                  size_t length) {
+  ctx->total += length;
+
+  if (ctx->partial) {
+    const size_t left = 32 - ctx->partial;
+    const size_t chunk = (length >= left) ? left : length;
+    memcpy(ctx->buffer.bytes + ctx->partial, data, chunk);
+    ctx->partial += chunk;
+    if (ctx->partial < 32) {
+      assert(left >= length);
+      return;
+    }
+    ctx->partial = 0;
+    data = (const uint8_t *)data + chunk;
+    length -= chunk;
+    T1HA2_UPDATE(le, aligned, &ctx->state, ctx->buffer.u64);
+  }
+
+  if (length >= 32) {
+#if T1HA_SYS_UNALIGNED_ACCESS == T1HA_UNALIGNED_ACCESS__EFFICIENT
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+    T1HA2_LOOP(le, unaligned, &ctx->state, data, length);
+#else
+    const bool misaligned = (((uintptr_t)data) & (ALIGNMENT_64 - 1)) != 0;
+    if (misaligned) {
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+      T1HA2_LOOP(le, unaligned, &ctx->state, data, length);
+    } else {
+#if defined(__LCC__) && __LCC__ > 123
+/* Форсирует комбинирование пар арифметических операций в двухэтажные операции
+ * в ближайшем после объявления директивы цикле, даже если эвристики оптимизации
+ * говорят, что это нецелесообразно */
+#pragma comb_oper
+#endif /* E2K LCC > 1.23 */
+      T1HA2_LOOP(le, aligned, &ctx->state, data, length);
+    }
+#endif
+    length &= 31;
+  }
+
+  if (length)
+    memcpy(ctx->buffer.bytes, data, ctx->partial = length);
+}
+
+uint64_t t1ha2_final(t1ha_context_t *__restrict ctx,
+                     uint64_t *__restrict extra_result) {
+  uint64_t bits = (ctx->total << 3) ^ (UINT64_C(1) << 63);
+#if __BYTE_ORDER__ != __ORDER_LITTLE_ENDIAN__
+  bits = bswap64(bits);
+#endif
+  t1ha2_update(ctx, &bits, 8);
+
+  if (likely(!extra_result)) {
+    squash(&ctx->state);
+    T1HA2_TAIL_AB(le, aligned, &ctx->state, ctx->buffer.u64, ctx->partial);
+  }
+
+  T1HA2_TAIL_ABCD(le, aligned, &ctx->state, ctx->buffer.u64, ctx->partial);
+}
+
+#endif /* T1HA2_DISABLED */

hash_testing/new_testing/hash_funcs/t1ha2/upstream/src/t1ha_selfcheck.h

@@ -0,0 +1,76 @@
+/*
+ *  Copyright (c) 2016-2020 Positive Technologies, https://www.ptsecurity.com,
+ *  Fast Positive Hash.
+ *
+ *  Portions Copyright (c) 2010-2020 Leonid Yuriev <leo@yuriev.ru>,
+ *  The 1Hippeus project (t1h).
+ *
+ *  This software is provided 'as-is', without any express or implied
+ *  warranty. In no event will the authors be held liable for any damages
+ *  arising from the use of this software.
+ *
+ *  Permission is granted to anyone to use this software for any purpose,
+ *  including commercial applications, and to alter it and redistribute it
+ *  freely, subject to the following restrictions:
+ *
+ *  1. The origin of this software must not be misrepresented; you must not
+ *     claim that you wrote the original software. If you use this software
+ *     in a product, an acknowledgement in the product documentation would be
+ *     appreciated but is not required.
+ *  2. Altered source versions must be plainly marked as such, and must not be
+ *     misrepresented as being the original software.
+ *  3. This notice may not be removed or altered from any source distribution.
+ */
+
+/*
+ * t1ha = { Fast Positive Hash, aka "Позитивный Хэш" }
+ * by [Positive Technologies](https://www.ptsecurity.ru)
+ *
+ * Briefly, it is a 64-bit Hash Function:
+ *  1. Created for 64-bit little-endian platforms, in predominantly for x86_64,
+ *     but portable and without penalties it can run on any 64-bit CPU.
+ *  2. In most cases up to 15% faster than City64, xxHash, mum-hash, metro-hash
+ *     and all others portable hash-functions (which do not use specific
+ *     hardware tricks).
+ *  3. Not suitable for cryptography.
+ *
+ * The Future will (be) Positive. Всё будет хорошо.
+ *
+ * ACKNOWLEDGEMENT:
+ * The t1ha was originally developed by Leonid Yuriev (Леонид Юрьев)
+ * for The 1Hippeus project - zerocopy messaging in the spirit of Sparta!
+ */
+
+#pragma once
+#if defined(_MSC_VER) && _MSC_VER > 1800
+#pragma warning(disable : 4464) /* relative include path contains '..' */
+#endif                          /* MSVC */
+#include "../t1ha.h"
+
+/***************************************************************************/
+/* Self-checking */
+
+extern const uint8_t t1ha_test_pattern[64];
+int t1ha_selfcheck(uint64_t (*hash)(const void *, size_t, uint64_t),
+                   const uint64_t *reference_values);
+
+#ifndef T1HA2_DISABLED
+extern const uint64_t t1ha_refval_2atonce[81];
+extern const uint64_t t1ha_refval_2atonce128[81];
+extern const uint64_t t1ha_refval_2stream[81];
+extern const uint64_t t1ha_refval_2stream128[81];
+#endif /* T1HA2_DISABLED */
+
+#ifndef T1HA1_DISABLED
+extern const uint64_t t1ha_refval_64le[81];
+extern const uint64_t t1ha_refval_64be[81];
+#endif /* T1HA1_DISABLED */
+
+#ifndef T1HA0_DISABLED
+extern const uint64_t t1ha_refval_32le[81];
+extern const uint64_t t1ha_refval_32be[81];
+#if T1HA0_AESNI_AVAILABLE
+extern const uint64_t t1ha_refval_ia32aes_a[81];
+extern const uint64_t t1ha_refval_ia32aes_b[81];
+#endif /* T1HA0_AESNI_AVAILABLE */
+#endif /* T1HA0_DISABLED */

hash_testing/new_testing/hash_funcs/t1ha2/upstream/t1ha.h

@@ -0,0 +1,719 @@
+/*
+ *  Copyright (c) 2016-2020 Positive Technologies, https://www.ptsecurity.com,
+ *  Fast Positive Hash.
+ *
+ *  Portions Copyright (c) 2010-2020 Leonid Yuriev <leo@yuriev.ru>,
+ *  The 1Hippeus project (t1h).
+ *
+ *  This software is provided 'as-is', without any express or implied
+ *  warranty. In no event will the authors be held liable for any damages
+ *  arising from the use of this software.
+ *
+ *  Permission is granted to anyone to use this software for any purpose,
+ *  including commercial applications, and to alter it and redistribute it
+ *  freely, subject to the following restrictions:
+ *
+ *  1. The origin of this software must not be misrepresented; you must not
+ *     claim that you wrote the original software. If you use this software
+ *     in a product, an acknowledgement in the product documentation would be
+ *     appreciated but is not required.
+ *  2. Altered source versions must be plainly marked as such, and must not be
+ *     misrepresented as being the original software.
+ *  3. This notice may not be removed or altered from any source distribution.
+ */
+
+/*
+ * t1ha = { Fast Positive Hash, aka "Позитивный Хэш" }
+ * by [Positive Technologies](https://www.ptsecurity.ru)
+ *
+ * Briefly, it is a 64-bit Hash Function:
+ *  1. Created for 64-bit little-endian platforms, in predominantly for x86_64,
+ *     but portable and without penalties it can run on any 64-bit CPU.
+ *  2. In most cases up to 15% faster than City64, xxHash, mum-hash, metro-hash
+ *     and all others portable hash-functions (which do not use specific
+ *     hardware tricks).
+ *  3. Not suitable for cryptography.
+ *
+ * The Future will (be) Positive. Всё будет хорошо.
+ *
+ * ACKNOWLEDGEMENT:
+ * The t1ha was originally developed by Leonid Yuriev (Леонид Юрьев)
+ * for The 1Hippeus project - zerocopy messaging in the spirit of Sparta!
+ */
+
+#pragma once
+
+/*****************************************************************************
+ *
+ * PLEASE PAY ATTENTION TO THE FOLLOWING NOTES
+ * about macros definitions which controls t1ha behaviour and/or performance.
+ *
+ *
+ * 1) T1HA_SYS_UNALIGNED_ACCESS = Defines the system/platform/CPU/architecture
+ *                                abilities for unaligned data access.
+ *
+ *    By default, when the T1HA_SYS_UNALIGNED_ACCESS not defined,
+ *    it will defined on the basis hardcoded knowledge about of capabilities
+ *    of most common CPU architectures. But you could override this
+ *    default behavior when build t1ha library itself:
+ *
+ *      // To disable unaligned access at all.
+ *      #define T1HA_SYS_UNALIGNED_ACCESS 0
+ *
+ *      // To enable unaligned access, but indicate that it significantly slow.
+ *      #define T1HA_SYS_UNALIGNED_ACCESS 1
+ *
+ *      // To enable unaligned access, and indicate that it effecient.
+ *      #define T1HA_SYS_UNALIGNED_ACCESS 2
+ *
+ *
+ * 2) T1HA_USE_FAST_ONESHOT_READ = Controls the data reads at the end of buffer.
+ *
+ *    When defined to non-zero, t1ha will use 'one shot' method for reading
+ *    up to 8 bytes at the end of data. In this case just the one 64-bit read
+ *    will be performed even when the available less than 8 bytes.
+ *
+ *    This is little bit faster that switching by length of data tail.
+ *    Unfortunately this will triggering a false-positive alarms from Valgrind,
+ *    AddressSanitizer and other similar tool.
+ *
+ *    By default, t1ha defines it to 1, but you could override this
+ *    default behavior when build t1ha library itself:
+ *
+ *      // For little bit faster and small code.
+ *      #define T1HA_USE_FAST_ONESHOT_READ 1
+ *
+ *      // For calmness if doubt.
+ *      #define T1HA_USE_FAST_ONESHOT_READ 0
+ *
+ *
+ * 3) T1HA0_RUNTIME_SELECT = Controls choice fastest function in runtime.
+ *
+ *    t1ha library offers the t1ha0() function as the fastest for current CPU.
+ *    But actual CPU's features/capabilities and may be significantly different,
+ *    especially on x86 platform. Therefore, internally, t1ha0() may require
+ *    dynamic dispatching for choice best implementation.
+ *
+ *    By default, t1ha enables such runtime choice and (may be) corresponding
+ *    indirect calls if it reasonable, but you could override this default
+ *    behavior when build t1ha library itself:
+ *
+ *      // To enable runtime choice of fastest implementation.
+ *      #define T1HA0_RUNTIME_SELECT 1
+ *
+ *      // To disable runtime choice of fastest implementation.
+ *      #define T1HA0_RUNTIME_SELECT 0
+ *
+ *    When T1HA0_RUNTIME_SELECT is nonzero the t1ha0_resolve() function could
+ *    be used to get actual t1ha0() implementation address at runtime. This is
+ *    useful for two cases:
+ *      - calling by local pointer-to-function usually is little
+ *        bit faster (less overhead) than via a PLT thru the DSO boundary.
+ *      - GNU Indirect functions (see below) don't supported by environment
+ *        and calling by t1ha0_funcptr is not available and/or expensive.
+ *
+ * 4) T1HA_USE_INDIRECT_FUNCTIONS = Controls usage of GNU Indirect functions.
+ *
+ *    In continue of T1HA0_RUNTIME_SELECT the T1HA_USE_INDIRECT_FUNCTIONS
+ *    controls usage of ELF indirect functions feature. In general, when
+ *    available, this reduces overhead of indirect function's calls though
+ *    a DSO-bundary (https://sourceware.org/glibc/wiki/GNU_IFUNC).
+ *
+ *    By default, t1ha engage GNU Indirect functions when it available
+ *    and useful, but you could override this default behavior when build
+ *    t1ha library itself:
+ *
+ *      // To enable use of GNU ELF Indirect functions.
+ *      #define T1HA_USE_INDIRECT_FUNCTIONS 1
+ *
+ *      // To disable use of GNU ELF Indirect functions. This may be useful
+ *      // if the actual toolchain or the system's loader don't support ones.
+ *      #define T1HA_USE_INDIRECT_FUNCTIONS 0
+ *
+ * 5) T1HA0_AESNI_AVAILABLE = Controls AES-NI detection and dispatching on x86.
+ *
+ *    In continue of T1HA0_RUNTIME_SELECT the T1HA0_AESNI_AVAILABLE controls
+ *    detection and usage of AES-NI CPU's feature. On the other hand, this
+ *    requires compiling parts of t1ha library with certain properly options,
+ *    and could be difficult or inconvenient in some cases.
+ *
+ *    By default, t1ha engade AES-NI for t1ha0() on the x86 platform, but
+ *    you could override this default behavior when build t1ha library itself:
+ *
+ *      // To disable detection and usage of AES-NI instructions for t1ha0().
+ *      // This may be useful when you unable to build t1ha library properly
+ *      // or known that AES-NI will be unavailable at the deploy.
+ *      #define T1HA0_AESNI_AVAILABLE 0
+ *
+ *      // To force detection and usage of AES-NI instructions for t1ha0(),
+ *      // but I don't known reasons to anybody would need this.
+ *      #define T1HA0_AESNI_AVAILABLE 1
+ *
+ * 6) T1HA0_DISABLED, T1HA1_DISABLED, T1HA2_DISABLED = Controls availability of
+ *    t1ha functions.
+ *
+ *    In some cases could be useful to import/use only few of t1ha functions
+ *    or just the one. So, this definitions allows disable corresponding parts
+ *    of t1ha library.
+ *
+ *      // To disable t1ha0(), t1ha0_32le(), t1ha0_32be() and all AES-NI.
+ *      #define T1HA0_DISABLED
+ *
+ *      // To disable t1ha1_le() and t1ha1_be().
+ *      #define T1HA1_DISABLED
+ *
+ *      // To disable t1ha2_atonce(), t1ha2_atonce128() and so on.
+ *      #define T1HA2_DISABLED
+ *
+ *****************************************************************************/
+
+#define T1HA_VERSION_MAJOR 2
+#define T1HA_VERSION_MINOR 1
+#define T1HA_VERSION_RELEASE 1
+
+#ifndef __has_attribute
+#define __has_attribute(x) (0)
+#endif
+
+#ifndef __has_include
+#define __has_include(x) (0)
+#endif
+
+#ifndef __GNUC_PREREQ
+#if defined(__GNUC__) && defined(__GNUC_MINOR__)
+#define __GNUC_PREREQ(maj, min)                                                \
+  ((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))
+#else
+#define __GNUC_PREREQ(maj, min) 0
+#endif
+#endif /* __GNUC_PREREQ */
+
+#ifndef __CLANG_PREREQ
+#ifdef __clang__
+#define __CLANG_PREREQ(maj, min)                                               \
+  ((__clang_major__ << 16) + __clang_minor__ >= ((maj) << 16) + (min))
+#else
+#define __CLANG_PREREQ(maj, min) (0)
+#endif
+#endif /* __CLANG_PREREQ */
+
+#ifndef __LCC_PREREQ
+#ifdef __LCC__
+#define __LCC_PREREQ(maj, min)                                                 \
+  ((__LCC__ << 16) + __LCC_MINOR__ >= ((maj) << 16) + (min))
+#else
+#define __LCC_PREREQ(maj, min) (0)
+#endif
+#endif /* __LCC_PREREQ */
+
+/*****************************************************************************/
+
+#ifdef _MSC_VER
+/* Avoid '16' bytes padding added after data member 't1ha_context::total'
+ * and other warnings from std-headers if warning-level > 3. */
+#pragma warning(push, 3)
+#endif
+
+#if defined(__cplusplus) && __cplusplus >= 201103L
+#include <climits>
+#include <cstddef>
+#include <cstdint>
+#else
+#include <limits.h>
+#include <stddef.h>
+#include <stdint.h>
+#endif
+
+/*****************************************************************************/
+
+#if defined(i386) || defined(__386) || defined(__i386) || defined(__i386__) || \
+    defined(i486) || defined(__i486) || defined(__i486__) ||                   \
+    defined(i586) | defined(__i586) || defined(__i586__) || defined(i686) ||   \
+    defined(__i686) || defined(__i686__) || defined(_M_IX86) ||                \
+    defined(_X86_) || defined(__THW_INTEL__) || defined(__I86__) ||            \
+    defined(__INTEL__) || defined(__x86_64) || defined(__x86_64__) ||          \
+    defined(__amd64__) || defined(__amd64) || defined(_M_X64) ||               \
+    defined(_M_AMD64) || defined(__IA32__) || defined(__INTEL__)
+#ifndef __ia32__
+/* LY: define neutral __ia32__ for x86 and x86-64 archs */
+#define __ia32__ 1
+#endif /* __ia32__ */
+#if !defined(__amd64__) && (defined(__x86_64) || defined(__x86_64__) ||        \
+                            defined(__amd64) || defined(_M_X64))
+/* LY: define trusty __amd64__ for all AMD64/x86-64 arch */
+#define __amd64__ 1
+#endif /* __amd64__ */
+#endif /* all x86 */
+
+#if !defined(__BYTE_ORDER__) || !defined(__ORDER_LITTLE_ENDIAN__) ||           \
+    !defined(__ORDER_BIG_ENDIAN__)
+
+/* *INDENT-OFF* */
+/* clang-format off */
+
+#if defined(__GLIBC__) || defined(__GNU_LIBRARY__) || defined(__ANDROID__) ||  \
+    defined(HAVE_ENDIAN_H) || __has_include(<endian.h>)
+#include <endian.h>
+#elif defined(__APPLE__) || defined(__MACH__) || defined(__OpenBSD__) ||       \
+    defined(HAVE_MACHINE_ENDIAN_H) || __has_include(<machine/endian.h>)
+#include <machine/endian.h>
+#elif defined(HAVE_SYS_ISA_DEFS_H) || __has_include(<sys/isa_defs.h>)
+#include <sys/isa_defs.h>
+#elif (defined(HAVE_SYS_TYPES_H) && defined(HAVE_SYS_ENDIAN_H)) ||             \
+    (__has_include(<sys/types.h>) && __has_include(<sys/endian.h>))
+#include <sys/endian.h>
+#include <sys/types.h>
+#elif defined(__bsdi__) || defined(__DragonFly__) || defined(__FreeBSD__) ||   \
+    defined(__NETBSD__) || defined(__NetBSD__) ||                              \
+    defined(HAVE_SYS_PARAM_H) || __has_include(<sys/param.h>)
+#include <sys/param.h>
+#endif /* OS */
+
+/* *INDENT-ON* */
+/* clang-format on */
+
+#if defined(__BYTE_ORDER) && defined(__LITTLE_ENDIAN) && defined(__BIG_ENDIAN)
+#define __ORDER_LITTLE_ENDIAN__ __LITTLE_ENDIAN
+#define __ORDER_BIG_ENDIAN__ __BIG_ENDIAN
+#define __BYTE_ORDER__ __BYTE_ORDER
+#elif defined(_BYTE_ORDER) && defined(_LITTLE_ENDIAN) && defined(_BIG_ENDIAN)
+#define __ORDER_LITTLE_ENDIAN__ _LITTLE_ENDIAN
+#define __ORDER_BIG_ENDIAN__ _BIG_ENDIAN
+#define __BYTE_ORDER__ _BYTE_ORDER
+#else
+#define __ORDER_LITTLE_ENDIAN__ 1234
+#define __ORDER_BIG_ENDIAN__ 4321
+
+#if defined(__LITTLE_ENDIAN__) ||                                              \
+    (defined(_LITTLE_ENDIAN) && !defined(_BIG_ENDIAN)) ||                      \
+    defined(__ARMEL__) || defined(__THUMBEL__) || defined(__AARCH64EL__) ||    \
+    defined(__MIPSEL__) || defined(_MIPSEL) || defined(__MIPSEL) ||            \
+    defined(_M_ARM) || defined(_M_ARM64) || defined(__e2k__) ||                \
+    defined(__elbrus_4c__) || defined(__elbrus_8c__) || defined(__bfin__) ||   \
+    defined(__BFIN__) || defined(__ia64__) || defined(_IA64) ||                \
+    defined(__IA64__) || defined(__ia64) || defined(_M_IA64) ||                \
+    defined(__itanium__) || defined(__ia32__) || defined(__CYGWIN__) ||        \
+    defined(_WIN64) || defined(_WIN32) || defined(__TOS_WIN__) ||              \
+    defined(__WINDOWS__)
+#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__
+
+#elif defined(__BIG_ENDIAN__) ||                                               \
+    (defined(_BIG_ENDIAN) && !defined(_LITTLE_ENDIAN)) ||                      \
+    defined(__ARMEB__) || defined(__THUMBEB__) || defined(__AARCH64EB__) ||    \
+    defined(__MIPSEB__) || defined(_MIPSEB) || defined(__MIPSEB) ||            \
+    defined(__m68k__) || defined(M68000) || defined(__hppa__) ||               \
+    defined(__hppa) || defined(__HPPA__) || defined(__sparc__) ||              \
+    defined(__sparc) || defined(__370__) || defined(__THW_370__) ||            \
+    defined(__s390__) || defined(__s390x__) || defined(__SYSC_ZARCH__)
+#define __BYTE_ORDER__ __ORDER_BIG_ENDIAN__
+
+#else
+#error __BYTE_ORDER__ should be defined.
+#endif /* Arch */
+
+#endif
+#endif /* __BYTE_ORDER__ || __ORDER_LITTLE_ENDIAN__ || __ORDER_BIG_ENDIAN__ */
+
+/*****************************************************************************/
+
+#ifndef __dll_export
+#if defined(_WIN32) || defined(_WIN64) || defined(__CYGWIN__)
+#if defined(__GNUC__) || __has_attribute(dllexport)
+#define __dll_export __attribute__((dllexport))
+#else
+#define __dll_export __declspec(dllexport)
+#endif
+#elif defined(__GNUC__) || __has_attribute(__visibility__)
+#define __dll_export __attribute__((__visibility__("default")))
+#else
+#define __dll_export
+#endif
+#endif /* __dll_export */
+
+#ifndef __dll_import
+#if defined(_WIN32) || defined(_WIN64) || defined(__CYGWIN__)
+#if defined(__GNUC__) || __has_attribute(dllimport)
+#define __dll_import __attribute__((dllimport))
+#else
+#define __dll_import __declspec(dllimport)
+#endif
+#elif defined(__GNUC__) || __has_attribute(__visibility__)
+#define __dll_import __attribute__((__visibility__("default")))
+#else
+#define __dll_import
+#endif
+#endif /* __dll_import */
+
+#ifndef __force_inline
+#ifdef _MSC_VER
+#define __force_inline __forceinline
+#elif __GNUC_PREREQ(3, 2) || __has_attribute(__always_inline__)
+#define __force_inline __inline __attribute__((__always_inline__))
+#else
+#define __force_inline __inline
+#endif
+#endif /* __force_inline */
+
+#ifndef T1HA_API
+#if defined(t1ha_EXPORTS)
+#define T1HA_API __dll_export
+#elif defined(t1ha_IMPORTS)
+#define T1HA_API __dll_import
+#else
+#define T1HA_API
+#endif
+#endif /* T1HA_API */
+
+#if defined(_MSC_VER) && defined(__ia32__)
+#define T1HA_ALIGN_PREFIX __declspec(align(32)) /* required only for SIMD */
+#else
+#define T1HA_ALIGN_PREFIX
+#endif /* _MSC_VER */
+
+#if defined(__GNUC__) && defined(__ia32__)
+#define T1HA_ALIGN_SUFFIX                                                      \
+  __attribute__((__aligned__(32))) /* required only for SIMD */
+#else
+#define T1HA_ALIGN_SUFFIX
+#endif /* GCC x86 */
+
+#ifndef T1HA_USE_INDIRECT_FUNCTIONS
+/* GNU ELF indirect functions usage control. For more info please see
+ * https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
+ * and https://sourceware.org/glibc/wiki/GNU_IFUNC */
+#if defined(__ELF__) && defined(__amd64__) &&                                  \
+    (__has_attribute(__ifunc__) ||                                             \
+     (!defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 &&             \
+      !defined(__SANITIZE_ADDRESS__) && !defined(__SSP_ALL__)))
+/* Enable gnu_indirect_function by default if :
+ *  - ELF AND x86_64
+ *  - attribute(__ifunc__) is available OR
+ *    GCC >= 4 WITHOUT -fsanitize=address NOR -fstack-protector-all */
+#define T1HA_USE_INDIRECT_FUNCTIONS 1
+#else
+#define T1HA_USE_INDIRECT_FUNCTIONS 0
+#endif
+#endif /* T1HA_USE_INDIRECT_FUNCTIONS */
+
+#if __GNUC_PREREQ(4, 0)
+#pragma GCC visibility push(hidden)
+#endif /* __GNUC_PREREQ(4,0) */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+typedef union T1HA_ALIGN_PREFIX t1ha_state256 {
+  uint8_t bytes[32];
+  uint32_t u32[8];
+  uint64_t u64[4];
+  struct {
+    uint64_t a, b, c, d;
+  } n;
+} t1ha_state256_t T1HA_ALIGN_SUFFIX;
+
+typedef struct t1ha_context {
+  t1ha_state256_t state;
+  t1ha_state256_t buffer;
+  size_t partial;
+  uint64_t total;
+} t1ha_context_t;
+
+#ifdef _MSC_VER
+#pragma warning(pop)
+#endif
+
+/******************************************************************************
+ *
+ * Self-testing API.
+ *
+ * Unfortunately, some compilers (exactly only Microsoft Visual C/C++) has
+ * a bugs which leads t1ha-functions to produce wrong results. This API allows
+ * check the correctness of the actual code in runtime.
+ *
+ * All check-functions returns 0 on success, or -1 in case the corresponding
+ * hash-function failed verification. PLEASE, always perform such checking at
+ * initialization of your code, if you using MSVC or other troubleful compilers.
+ */
+
+T1HA_API int t1ha_selfcheck__all_enabled(void);
+
+#ifndef T1HA2_DISABLED
+T1HA_API int t1ha_selfcheck__t1ha2_atonce(void);
+T1HA_API int t1ha_selfcheck__t1ha2_atonce128(void);
+T1HA_API int t1ha_selfcheck__t1ha2_stream(void);
+T1HA_API int t1ha_selfcheck__t1ha2(void);
+#endif /* T1HA2_DISABLED */
+
+#ifndef T1HA1_DISABLED
+T1HA_API int t1ha_selfcheck__t1ha1_le(void);
+T1HA_API int t1ha_selfcheck__t1ha1_be(void);
+T1HA_API int t1ha_selfcheck__t1ha1(void);
+#endif /* T1HA1_DISABLED */
+
+#ifndef T1HA0_DISABLED
+T1HA_API int t1ha_selfcheck__t1ha0_32le(void);
+T1HA_API int t1ha_selfcheck__t1ha0_32be(void);
+T1HA_API int t1ha_selfcheck__t1ha0(void);
+
+/* Define T1HA0_AESNI_AVAILABLE to 0 for disable AES-NI support. */
+#ifndef T1HA0_AESNI_AVAILABLE
+#if defined(__e2k__) ||                                                        \
+    (defined(__ia32__) && (!defined(_M_IX86) || _MSC_VER > 1800))
+#define T1HA0_AESNI_AVAILABLE 1
+#else
+#define T1HA0_AESNI_AVAILABLE 0
+#endif
+#endif /* ifndef T1HA0_AESNI_AVAILABLE */
+
+#if T1HA0_AESNI_AVAILABLE
+T1HA_API int t1ha_selfcheck__t1ha0_ia32aes_noavx(void);
+T1HA_API int t1ha_selfcheck__t1ha0_ia32aes_avx(void);
+#ifndef __e2k__
+T1HA_API int t1ha_selfcheck__t1ha0_ia32aes_avx2(void);
+#endif
+#endif /* if T1HA0_AESNI_AVAILABLE */
+#endif /* T1HA0_DISABLED */
+
+/******************************************************************************
+ *
+ *  t1ha2 = 64 and 128-bit, SLIGHTLY MORE ATTENTION FOR QUALITY AND STRENGTH.
+ *
+ *    - The recommended version of "Fast Positive Hash" with good quality
+ *      for checksum, hash tables and fingerprinting.
+ *    - Portable and extremely efficiency on modern 64-bit CPUs.
+ *      Designed for 64-bit little-endian platforms,
+ *      in other cases will runs slowly.
+ *    - Great quality of hashing and still faster than other non-t1ha hashes.
+ *      Provides streaming mode and 128-bit result.
+ *
+ * Note: Due performance reason 64- and 128-bit results are completely
+ *       different each other, i.e. 64-bit result is NOT any part of 128-bit.
+ */
+#ifndef T1HA2_DISABLED
+
+/* The at-once variant with 64-bit result */
+T1HA_API uint64_t t1ha2_atonce(const void *data, size_t length, uint64_t seed);
+
+/* The at-once variant with 128-bit result.
+ * Argument `extra_result` is NOT optional and MUST be valid.
+ * The high 64-bit part of 128-bit hash will be always unconditionally
+ * stored to the address given by `extra_result` argument. */
+T1HA_API uint64_t t1ha2_atonce128(uint64_t *__restrict extra_result,
+                                  const void *__restrict data, size_t length,
+                                  uint64_t seed);
+
+/* The init/update/final trinity for streaming.
+ * Return 64 or 128-bit result depentently from `extra_result` argument. */
+T1HA_API void t1ha2_init(t1ha_context_t *ctx, uint64_t seed_x, uint64_t seed_y);
+T1HA_API void t1ha2_update(t1ha_context_t *__restrict ctx,
+                           const void *__restrict data, size_t length);
+
+/* Argument `extra_result` is optional and MAY be NULL.
+ *  - If `extra_result` is NOT NULL then the 128-bit hash will be calculated,
+ *    and high 64-bit part of it will be stored to the address given
+ *    by `extra_result` argument.
+ *  - Otherwise the 64-bit hash will be calculated
+ *    and returned from function directly.
+ *
+ * Note: Due performance reason 64- and 128-bit results are completely
+ *       different each other, i.e. 64-bit result is NOT any part of 128-bit. */
+T1HA_API uint64_t t1ha2_final(t1ha_context_t *__restrict ctx,
+                              uint64_t *__restrict extra_result /* optional */);
+
+#endif /* T1HA2_DISABLED */
+
+/******************************************************************************
+ *
+ *  t1ha1 = 64-bit, BASELINE FAST PORTABLE HASH:
+ *
+ *    - Runs faster on 64-bit platforms in other cases may runs slowly.
+ *    - Portable and stable, returns same 64-bit result
+ *      on all architectures and CPUs.
+ *    - Unfortunately it fails the "strict avalanche criteria",
+ *      see test results at https://github.com/demerphq/smhasher.
+ *
+ *      This flaw is insignificant for the t1ha1() purposes and imperceptible
+ *      from a practical point of view.
+ *      However, nowadays this issue has resolved in the next t1ha2(),
+ *      that was initially planned to providing a bit more quality.
+ */
+#ifndef T1HA1_DISABLED
+
+/* The little-endian variant. */
+T1HA_API uint64_t t1ha1_le(const void *data, size_t length, uint64_t seed);
+
+/* The big-endian variant. */
+T1HA_API uint64_t t1ha1_be(const void *data, size_t length, uint64_t seed);
+
+#endif /* T1HA1_DISABLED */
+
+/******************************************************************************
+ *
+ *  t1ha0 = 64-bit, JUST ONLY FASTER:
+ *
+ *    - Provides fast-as-possible hashing for current CPU, including
+ *      32-bit systems and engaging the available hardware acceleration.
+ *    - It is a facade that selects most quick-and-dirty hash
+ *      for the current processor. For instance, on IA32 (x86) actual function
+ *      will be selected in runtime, depending on current CPU capabilities
+ *
+ * BE CAREFUL!!!  THIS IS MEANS:
+ *
+ *   1. The quality of hash is a subject for tradeoffs with performance.
+ *      So, the quality and strength of t1ha0() may be lower than t1ha1(),
+ *      especially on 32-bit targets, but then much faster.
+ *      However, guaranteed that it passes all SMHasher tests.
+ *
+ *   2. No warranty that the hash result will be same for particular
+ *      key on another machine or another version of libt1ha.
+ *
+ *      Briefly, such hash-results and their derivatives, should be
+ *      used only in runtime, but should not be persist or transferred
+ *      over a network.
+ *
+ *
+ *  When T1HA0_RUNTIME_SELECT is nonzero the t1ha0_resolve() function could
+ *  be used to get actual t1ha0() implementation address at runtime. This is
+ *  useful for two cases:
+ *    - calling by local pointer-to-function usually is little
+ *      bit faster (less overhead) than via a PLT thru the DSO boundary.
+ *    - GNU Indirect functions (see below) don't supported by environment
+ *      and calling by t1ha0_funcptr is not available and/or expensive.
+ */
+
+#ifndef T1HA0_DISABLED
+
+/* The little-endian variant for 32-bit CPU. */
+uint64_t t1ha0_32le(const void *data, size_t length, uint64_t seed);
+/* The big-endian variant for 32-bit CPU. */
+uint64_t t1ha0_32be(const void *data, size_t length, uint64_t seed);
+
+/* Define T1HA0_AESNI_AVAILABLE to 0 for disable AES-NI support. */
+#ifndef T1HA0_AESNI_AVAILABLE
+#if defined(__e2k__) ||                                                        \
+    (defined(__ia32__) && (!defined(_M_IX86) || _MSC_VER > 1800))
+#define T1HA0_AESNI_AVAILABLE 1
+#else
+#define T1HA0_AESNI_AVAILABLE 0
+#endif
+#endif /* T1HA0_AESNI_AVAILABLE */
+
+/* Define T1HA0_RUNTIME_SELECT to 0 for disable dispatching t1ha0 at runtime. */
+#ifndef T1HA0_RUNTIME_SELECT
+#if T1HA0_AESNI_AVAILABLE && !defined(__e2k__)
+#define T1HA0_RUNTIME_SELECT 1
+#else
+#define T1HA0_RUNTIME_SELECT 0
+#endif
+#endif /* T1HA0_RUNTIME_SELECT */
+
+#if !T1HA0_RUNTIME_SELECT && !defined(T1HA0_USE_DEFINE)
+#if defined(__LCC__)
+#define T1HA0_USE_DEFINE 1
+#else
+#define T1HA0_USE_DEFINE 0
+#endif
+#endif /* T1HA0_USE_DEFINE */
+
+#if T1HA0_AESNI_AVAILABLE
+uint64_t t1ha0_ia32aes_noavx(const void *data, size_t length, uint64_t seed);
+uint64_t t1ha0_ia32aes_avx(const void *data, size_t length, uint64_t seed);
+#ifndef __e2k__
+uint64_t t1ha0_ia32aes_avx2(const void *data, size_t length, uint64_t seed);
+#endif
+#endif /* T1HA0_AESNI_AVAILABLE */
+
+#if T1HA0_RUNTIME_SELECT
+typedef uint64_t (*t1ha0_function_t)(const void *, size_t, uint64_t);
+T1HA_API t1ha0_function_t t1ha0_resolve(void);
+#if T1HA_USE_INDIRECT_FUNCTIONS
+T1HA_API uint64_t t1ha0(const void *data, size_t length, uint64_t seed);
+#else
+/* Otherwise function pointer will be used.
+ * Unfortunately this may cause some overhead calling. */
+T1HA_API extern uint64_t (*t1ha0_funcptr)(const void *data, size_t length,
+                                          uint64_t seed);
+static __force_inline uint64_t t1ha0(const void *data, size_t length,
+                                     uint64_t seed) {
+  return t1ha0_funcptr(data, length, seed);
+}
+#endif /* T1HA_USE_INDIRECT_FUNCTIONS */
+
+#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+
+#if T1HA0_USE_DEFINE
+
+#if (UINTPTR_MAX > 0xffffFFFFul || ULONG_MAX > 0xffffFFFFul) &&                \
+    (!defined(T1HA1_DISABLED) || !defined(T1HA2_DISABLED))
+#if defined(T1HA1_DISABLED)
+#define t1ha0 t1ha2_atonce
+#else
+#define t1ha0 t1ha1_be
+#endif /* T1HA1_DISABLED */
+#else  /* 32/64 */
+#define t1ha0 t1ha0_32be
+#endif /* 32/64 */
+
+#else /* T1HA0_USE_DEFINE */
+
+static __force_inline uint64_t t1ha0(const void *data, size_t length,
+                                     uint64_t seed) {
+#if (UINTPTR_MAX > 0xffffFFFFul || ULONG_MAX > 0xffffFFFFul) &&                \
+    (!defined(T1HA1_DISABLED) || !defined(T1HA2_DISABLED))
+#if defined(T1HA1_DISABLED)
+  return t1ha2_atonce(data, length, seed);
+#else
+  return t1ha1_be(data, length, seed);
+#endif /* T1HA1_DISABLED */
+#else  /* 32/64 */
+  return t1ha0_32be(data, length, seed);
+#endif /* 32/64 */
+}
+
+#endif /* !T1HA0_USE_DEFINE */
+
+#else /* !T1HA0_RUNTIME_SELECT && __BYTE_ORDER__ != __ORDER_BIG_ENDIAN__ */
+
+#if T1HA0_USE_DEFINE
+
+#if (UINTPTR_MAX > 0xffffFFFFul || ULONG_MAX > 0xffffFFFFul) &&                \
+    (!defined(T1HA1_DISABLED) || !defined(T1HA2_DISABLED))
+#if defined(T1HA1_DISABLED)
+#define t1ha0 t1ha2_atonce
+#else
+#define t1ha0 t1ha1_le
+#endif /* T1HA1_DISABLED */
+#else  /* 32/64 */
+#define t1ha0 t1ha0_32le
+#endif /* 32/64 */
+
+#else
+
+static __force_inline uint64_t t1ha0(const void *data, size_t length,
+                                     uint64_t seed) {
+#if (UINTPTR_MAX > 0xffffFFFFul || ULONG_MAX > 0xffffFFFFul) &&                \
+    (!defined(T1HA1_DISABLED) || !defined(T1HA2_DISABLED))
+#if defined(T1HA1_DISABLED)
+  return t1ha2_atonce(data, length, seed);
+#else
+  return t1ha1_le(data, length, seed);
+#endif /* T1HA1_DISABLED */
+#else  /* 32/64 */
+  return t1ha0_32le(data, length, seed);
+#endif /* 32/64 */
+}
+
+#endif /* !T1HA0_USE_DEFINE */
+
+#endif /* !T1HA0_RUNTIME_SELECT */
+
+#endif /* T1HA0_DISABLED */
+
+#ifdef __cplusplus
+}
+#endif
+
+#if __GNUC_PREREQ(4, 0)
+#pragma GCC visibility pop
+#endif /* __GNUC_PREREQ(4,0) */

hash_testing/new_testing/hash_funcs/xxh64/bin_hash.c

@@ -0,0 +1,224 @@
+/*
+ * Standalone XXH64 command-line wrapper for avalanche testing.
+ *
+ * XXH64 algorithm derived from xxHash by Yann Collet:
+ * https://github.com/Cyan4973/xxHash
+ *
+ * Copyright (C) 2012-2023 Yann Collet
+ *
+ * BSD 2-Clause License
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *    this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright notice,
+ *    this list of conditions and the following disclaimer in the documentation
+ *    and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED. IN NO EVENT SHALL THE
+ * COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DAMAGES ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
+ * DAMAGE.
+ */
+
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#define XXH64_SEED UINT64_C(0)
+#define XXH_PRIME64_1 UINT64_C(11400714785074694791)
+#define XXH_PRIME64_2 UINT64_C(14029467366897019727)
+#define XXH_PRIME64_3 UINT64_C(1609587929392839161)
+#define XXH_PRIME64_4 UINT64_C(9650029242287828579)
+#define XXH_PRIME64_5 UINT64_C(2870177450012600261)
+
+static uint64_t rotate_left64(uint64_t value, unsigned int count)
+{
+    return (value << count) | (value >> (64U - count));
+}
+
+static uint32_t read_little_endian32(const unsigned char *data)
+{
+    return (uint32_t)data[0] | ((uint32_t)data[1] << 8U) |
+           ((uint32_t)data[2] << 16U) | ((uint32_t)data[3] << 24U);
+}
+
+static uint64_t read_little_endian64(const unsigned char *data)
+{
+    return (uint64_t)read_little_endian32(data) |
+           ((uint64_t)read_little_endian32(data + 4) << 32U);
+}
+
+static uint64_t xxh64_round(uint64_t accumulator, uint64_t input)
+{
+    accumulator += input * XXH_PRIME64_2;
+    accumulator = rotate_left64(accumulator, 31U);
+    accumulator *= XXH_PRIME64_1;
+    return accumulator;
+}
+
+static uint64_t xxh64_merge_round(uint64_t accumulator, uint64_t value)
+{
+    value = xxh64_round(UINT64_C(0), value);
+    accumulator ^= value;
+    accumulator = accumulator * XXH_PRIME64_1 + XXH_PRIME64_4;
+    return accumulator;
+}
+
+static uint64_t xxh64(const unsigned char *data, size_t length, uint64_t seed)
+{
+    const unsigned char *position = data;
+    const unsigned char *const end = data + length;
+    uint64_t hash;
+
+    if (length >= 32U) {
+        const unsigned char *const block_end = end - 32U;
+        uint64_t accumulator1 = seed + XXH_PRIME64_1 + XXH_PRIME64_2;
+        uint64_t accumulator2 = seed + XXH_PRIME64_2;
+        uint64_t accumulator3 = seed;
+        uint64_t accumulator4 = seed - XXH_PRIME64_1;
+
+        do {
+            accumulator1 = xxh64_round(accumulator1, read_little_endian64(position));
+            position += 8;
+            accumulator2 = xxh64_round(accumulator2, read_little_endian64(position));
+            position += 8;
+            accumulator3 = xxh64_round(accumulator3, read_little_endian64(position));
+            position += 8;
+            accumulator4 = xxh64_round(accumulator4, read_little_endian64(position));
+            position += 8;
+        } while (position <= block_end);
+
+        hash = rotate_left64(accumulator1, 1U) +
+               rotate_left64(accumulator2, 7U) +
+               rotate_left64(accumulator3, 12U) +
+               rotate_left64(accumulator4, 18U);
+        hash = xxh64_merge_round(hash, accumulator1);
+        hash = xxh64_merge_round(hash, accumulator2);
+        hash = xxh64_merge_round(hash, accumulator3);
+        hash = xxh64_merge_round(hash, accumulator4);
+    } else {
+        hash = seed + XXH_PRIME64_5;
+    }
+
+    hash += (uint64_t)length;
+
+    while ((size_t)(end - position) >= 8U) {
+        uint64_t value = xxh64_round(UINT64_C(0), read_little_endian64(position));
+        hash ^= value;
+        hash = rotate_left64(hash, 27U) * XXH_PRIME64_1 + XXH_PRIME64_4;
+        position += 8;
+    }
+
+    if ((size_t)(end - position) >= 4U) {
+        hash ^= (uint64_t)read_little_endian32(position) * XXH_PRIME64_1;
+        hash = rotate_left64(hash, 23U) * XXH_PRIME64_2 + XXH_PRIME64_3;
+        position += 4;
+    }
+
+    while (position < end) {
+        hash ^= (uint64_t)(*position) * XXH_PRIME64_5;
+        hash = rotate_left64(hash, 11U) * XXH_PRIME64_1;
+        ++position;
+    }
+
+    hash ^= hash >> 33U;
+    hash *= XXH_PRIME64_2;
+    hash ^= hash >> 29U;
+    hash *= XXH_PRIME64_3;
+    hash ^= hash >> 32U;
+    return hash;
+}
+
+static int is_ascii_trailing_space(unsigned char character)
+{
+    return character == ' ' || character == '\t' || character == '\n' ||
+           character == '\r' || character == '\v' || character == '\f';
+}
+
+static int read_stdin(unsigned char **data, size_t *length)
+{
+    size_t capacity = 256;
+    unsigned char *buffer = malloc(capacity);
+
+    if (buffer == NULL) {
+        return -1;
+    }
+
+    *length = 0;
+    for (;;) {
+        size_t available = capacity - *length;
+        size_t bytes_read = fread(buffer + *length, 1, available, stdin);
+        *length += bytes_read;
+
+        if (bytes_read < available) {
+            if (ferror(stdin)) {
+                free(buffer);
+                return -1;
+            }
+            break;
+        }
+
+        if (capacity > SIZE_MAX / 2U) {
+            free(buffer);
+            return -1;
+        }
+        capacity *= 2U;
+
+        {
+            unsigned char *larger_buffer = realloc(buffer, capacity);
+            if (larger_buffer == NULL) {
+                free(buffer);
+                return -1;
+            }
+            buffer = larger_buffer;
+        }
+    }
+
+    *data = buffer;
+    return 0;
+}
+
+int main(int argc, char **argv)
+{
+    const unsigned char *word;
+    unsigned char *stdin_buffer = NULL;
+    size_t length;
+
+    if (argc > 2) {
+        fprintf(stderr, "usage: %s [ASCII_WORD]\n", argv[0]);
+        return EXIT_FAILURE;
+    }
+
+    if (argc == 2) {
+        word = (const unsigned char *)argv[1];
+        length = strlen(argv[1]);
+    } else {
+        if (read_stdin(&stdin_buffer, &length) != 0) {
+            fprintf(stderr, "failed to read input\n");
+            return EXIT_FAILURE;
+        }
+        word = stdin_buffer;
+    }
+
+    while (length > 0U && is_ascii_trailing_space(word[length - 1U])) {
+        --length;
+    }
+
+    for (size_t index = 0; index < length; ++index) {
+        if (word[index] > 0x7fU) {
+            fprintf(stderr, "input must contain ASCII characters only\n");
+            free(stdin_buffer);
+            return EXIT_FAILURE;
+        }
+    }
+
+    printf("%016" PRIx64 "\n", xxh64(word, length, XXH64_SEED));
+    free(stdin_buffer);
+    return EXIT_SUCCESS;
+}

hash_testing/new_testing/input_symbols/cpp_provides_p11.manifest.txt

@@ -0,0 +1,22 @@
+branch=p11
+arch=x86_64
+filter=Itanium C++ ABI mangled names beginning with _Z
+extractor=/usr/lib/rpm/provided_symbols
+selection=largest x86_64 C++ candidates found by Provides set-string length in apt-cache dumpavail
+candidate=slicer set_length=509743
+candidate=libmlir22.1 set_length=303303
+candidate=libmlir21.1 set_length=284616
+candidate=libmlir18.1 set_length=204305
+candidate=libmlir20.1 set_length=182470
+
+package=slicer (slicer-5.10.0-alt1.x86_64.rpm) elf_files=227 symbols=380667 selected=380530
+package=libmlir22.1 (libmlir22.1-22.1.1-alt0.1.x86_64.rpm) elf_files=9 symbols=146036 selected=145767
+package=libmlir21.1 (libmlir21.1-21.1.5-alt0.1.x86_64.rpm) elf_files=8 symbols=136102 selected=135854
+package=libmlir20.1 (libmlir20.1-20.1.8-alt0.4.x86_64.rpm) elf_files=8 symbols=90904 selected=90656
+package=libmlir18.1 (libmlir18.1-18.1.8-alt0.2.x86_64.rpm) elf_files=7 symbols=103099 selected=102835
+packages=5 elf_files=259 unique_symbols=577509 output=cpp_provides_p11.txt
+similar_lcp12=575206 share=0.996012
+similar_lcp24=548917 share=0.950491
+median_max_lcp=61
+max_lcp=1018
+sha256=bf8e760bdf4d52b42040cff4625de73988d8dd6b517566efd186c71e3382753d

hash_testing/new_testing/plot_probability_map.py

@@ -0,0 +1,445 @@
+#!/usr/bin/env python3
+"""Render avalanche bit-probability CSV files as two PNG bar charts."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from statistics import fmean
+from typing import Sequence
+
+from PIL import Image, ImageDraw, ImageFont
+
+BACKGROUND = "#f7f8fa"
+PANEL = "#ffffff"
+GRID = "#d9dee7"
+TEXT = "#172033"
+MUTED = "#637083"
+REFERENCE = "#d24b4b"
+COLORS = (
+    "#377eb8",
+    "#4daf4a",
+    "#984ea3",
+    "#ff7f00",
+    "#e41a1c",
+    "#00a6a6",
+    "#a65628",
+)
+
+
+@dataclass(frozen=True)
+class ProbabilityRow:
+    operation: str
+    pairs: int
+    probabilities: list[float]
+
+
+@dataclass(frozen=True)
+class ChartScale:
+    minimum: float
+    maximum: float
+    ticks: tuple[float, ...]
+
+
+def nice_step(value: float) -> float:
+    exponent = math.floor(math.log10(value))
+    fraction = value / 10**exponent
+    nice_fraction = min((1.0, 2.0, 2.5, 5.0, 10.0), key=lambda item: abs(item - fraction))
+    return nice_fraction * 10**exponent
+
+
+def make_scale(
+    values: Sequence[float],
+    *,
+    hard_limits: tuple[float, float],
+    reference: float | None = None,
+) -> ChartScale:
+    """Build a padded shared scale constrained to hard limits."""
+    hard_minimum, hard_maximum = hard_limits
+    finite_values = [value for value in values if math.isfinite(value)]
+    if reference is not None:
+        finite_values.append(reference)
+    if not finite_values:
+        finite_values = [hard_minimum, hard_maximum]
+
+    minimum = min(finite_values)
+    maximum = max(finite_values)
+    if minimum == maximum:
+        expansion = (hard_maximum - hard_minimum) * 0.1
+        minimum -= expansion / 2
+        maximum += expansion / 2
+
+    span = maximum - minimum
+    padded_minimum = max(hard_minimum, minimum - span * 0.1)
+    padded_maximum = min(hard_maximum, maximum + span * 0.1)
+    step = nice_step(max((padded_maximum - padded_minimum) / 5, 1e-12))
+    scaled_minimum = max(hard_minimum, math.floor(padded_minimum / step) * step)
+    scaled_maximum = min(hard_maximum, math.ceil(padded_maximum / step) * step)
+    if scaled_minimum == scaled_maximum:
+        scaled_minimum, scaled_maximum = hard_minimum, hard_maximum
+
+    tick_count = round((scaled_maximum - scaled_minimum) / step)
+    ticks = [scaled_minimum + index * step for index in range(tick_count + 1)]
+    if reference is not None and scaled_minimum <= reference <= scaled_maximum:
+        ticks.append(reference)
+    normalized_ticks = tuple(
+        sorted({round(value, 12) for value in ticks if scaled_minimum <= value <= scaled_maximum})
+    )
+    return ChartScale(scaled_minimum, scaled_maximum, normalized_ticks)
+
+
+def load_font(size: int, bold: bool = False) -> ImageFont.FreeTypeFont | ImageFont.ImageFont:
+    names = (
+        "DejaVuSans-Bold.ttf" if bold else "DejaVuSans.ttf",
+        "/usr/share/fonts/TTF/DejaVuSans-Bold.ttf"
+        if bold
+        else "/usr/share/fonts/TTF/DejaVuSans.ttf",
+        "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf"
+        if bold
+        else "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
+    )
+    for name in names:
+        try:
+            return ImageFont.truetype(name, size)
+        except OSError:
+            continue
+    return ImageFont.load_default()
+
+
+def read_probability_map(path: Path) -> list[ProbabilityRow]:
+    """Read operation rows and numerically ordered bit columns from a CSV file."""
+    try:
+        stream = path.open(encoding="utf-8", newline="")
+    except OSError as error:
+        raise ValueError(f"не удалось открыть {path}: {error}") from error
+
+    with stream:
+        reader = csv.DictReader(stream)
+        fields = reader.fieldnames
+        if not fields or "operation" not in fields or "pairs" not in fields:
+            raise ValueError("CSV должен содержать колонки operation и pairs")
+
+        bit_fields: list[tuple[int, str]] = []
+        for field in fields:
+            if not field.startswith("bit_"):
+                continue
+            try:
+                bit_fields.append((int(field.removeprefix("bit_")), field))
+            except ValueError as error:
+                raise ValueError(f"некорректная битовая колонка: {field}") from error
+        bit_fields.sort()
+        if not bit_fields:
+            raise ValueError("CSV не содержит колонок bit_N")
+        expected_bits = list(range(len(bit_fields)))
+        actual_bits = [bit for bit, _field in bit_fields]
+        if actual_bits != expected_bits:
+            raise ValueError("битовые колонки должны непрерывно идти от bit_0")
+
+        rows: list[ProbabilityRow] = []
+        for line_number, row in enumerate(reader, start=2):
+            operation = (row.get("operation") or "").strip()
+            if not operation:
+                raise ValueError(f"строка {line_number}: пустая операция")
+            try:
+                pairs = int(row["pairs"] or "")
+            except (TypeError, ValueError) as error:
+                raise ValueError(
+                    f"строка {line_number}: некорректное число пар"
+                ) from error
+            if pairs < 0:
+                raise ValueError(f"строка {line_number}: число пар меньше нуля")
+
+            probabilities: list[float] = []
+            for _bit, field in bit_fields:
+                try:
+                    value = float(row[field] or "")
+                except (TypeError, ValueError) as error:
+                    raise ValueError(
+                        f"строка {line_number}: некорректное значение {field}"
+                    ) from error
+                if not math.isnan(value) and not 0.0 <= value <= 1.0:
+                    raise ValueError(
+                        f"строка {line_number}: {field} должен быть от 0 до 1"
+                    )
+                probabilities.append(value)
+            rows.append(ProbabilityRow(operation, pairs, probabilities))
+
+    if not rows:
+        raise ValueError("CSV не содержит строк с операциями")
+    return rows
+
+
+def mean_absolute_deviation(probabilities: Sequence[float]) -> float:
+    """Return the mean |p - 0.5| over finite bit probabilities."""
+    deviations = [
+        abs(probability - 0.5)
+        for probability in probabilities
+        if math.isfinite(probability)
+    ]
+    return fmean(deviations) if deviations else float("nan")
+
+
+def text_width(
+    draw: ImageDraw.ImageDraw,
+    text: str,
+    font: ImageFont.FreeTypeFont | ImageFont.ImageFont,
+) -> int:
+    box = draw.textbbox((0, 0), text, font=font)
+    return round(box[2] - box[0])
+
+
+def draw_centered_text(
+    draw: ImageDraw.ImageDraw,
+    center_x: float,
+    y: float,
+    text: str,
+    font: ImageFont.FreeTypeFont | ImageFont.ImageFont,
+    fill: str = TEXT,
+) -> None:
+    draw.text(
+        (center_x - text_width(draw, text, font) / 2, y),
+        text,
+        font=font,
+        fill=fill,
+    )
+
+
+def draw_bit_panel(
+    draw: ImageDraw.ImageDraw,
+    bounds: tuple[int, int, int, int],
+    row: ProbabilityRow,
+    color: str,
+    scale: ChartScale,
+) -> None:
+    left, top, right, bottom = bounds
+    title_font = load_font(22, bold=True)
+    label_font = load_font(14)
+    tick_font = load_font(12)
+
+    draw.rounded_rectangle(bounds, radius=12, fill=PANEL, outline=GRID, width=1)
+    draw.text((left + 18, top + 13), row.operation, font=title_font, fill=TEXT)
+    pairs_text = f"pairs: {row.pairs}"
+    draw.text(
+        (right - 18 - text_width(draw, pairs_text, label_font), top + 17),
+        pairs_text,
+        font=label_font,
+        fill=MUTED,
+    )
+
+    plot_left = left + 54
+    plot_right = right - 18
+    plot_top = top + 55
+    plot_bottom = bottom - 42
+    plot_height = plot_bottom - plot_top
+
+    scale_span = scale.maximum - scale.minimum
+    for probability in scale.ticks:
+        y = round(plot_bottom - (probability - scale.minimum) / scale_span * plot_height)
+        line_color = REFERENCE if probability == 0.5 else GRID
+        line_width = 2 if probability == 0.5 else 1
+        draw.line((plot_left, y, plot_right, y), fill=line_color, width=line_width)
+        label = f"{probability:.3g}"
+        draw.text(
+            (plot_left - 8 - text_width(draw, label, tick_font), y - 7),
+            label,
+            font=tick_font,
+            fill=MUTED,
+        )
+
+    bit_count = len(row.probabilities)
+    slot_width = (plot_right - plot_left) / bit_count
+    bar_width = max(1, int(slot_width * 0.72))
+    for bit, probability in enumerate(row.probabilities):
+        if not math.isfinite(probability):
+            continue
+        center = plot_left + (bit + 0.5) * slot_width
+        x0 = round(center - bar_width / 2)
+        x1 = round(center + bar_width / 2)
+        y = round(
+            plot_bottom
+            - (probability - scale.minimum) / scale_span * plot_height
+        )
+        draw.rectangle((x0, y, x1, plot_bottom), fill=color)
+
+    tick_step = max(1, math.ceil(bit_count / 16))
+    for bit in range(0, bit_count, tick_step):
+        center = plot_left + (bit + 0.5) * slot_width
+        label = str(bit)
+        draw.text(
+            (center - text_width(draw, label, tick_font) / 2, plot_bottom + 7),
+            label,
+            font=tick_font,
+            fill=MUTED,
+        )
+    draw_centered_text(
+        draw,
+        (plot_left + plot_right) / 2,
+        bottom - 21,
+        "output bit (0 = LSB)",
+        tick_font,
+        MUTED,
+    )
+
+
+def render_bit_probabilities(
+    rows: Sequence[ProbabilityRow], output: Path, title: str
+) -> None:
+    columns = 2 if len(rows) > 1 else 1
+    panel_width = 760
+    panel_height = 330
+    gap = 18
+    margin = 24
+    title_height = 70
+    row_count = math.ceil(len(rows) / columns)
+    width = margin * 2 + columns * panel_width + (columns - 1) * gap
+    height = title_height + margin + row_count * panel_height + (row_count - 1) * gap
+
+    image = Image.new("RGB", (width, height), BACKGROUND)
+    draw = ImageDraw.Draw(image)
+    scale = make_scale(
+        [probability for row in rows for probability in row.probabilities],
+        reference=0.5,
+        hard_limits=(0.0, 1.0),
+    )
+    draw_centered_text(draw, width / 2, 18, title, load_font(30, bold=True))
+    draw_centered_text(
+        draw,
+        width / 2,
+        52,
+        f"Shared scale {scale.minimum:.3g}–{scale.maximum:.3g}; red line = ideal p=0.5",
+        load_font(14),
+        MUTED,
+    )
+
+    for index, row in enumerate(rows):
+        column = index % columns
+        grid_row = index // columns
+        left = margin + column * (panel_width + gap)
+        top = title_height + grid_row * (panel_height + gap)
+        draw_bit_panel(
+            draw,
+            (left, top, left + panel_width, top + panel_height),
+            row,
+            COLORS[index % len(COLORS)],
+            scale,
+        )
+
+    output.parent.mkdir(parents=True, exist_ok=True)
+    image.save(output, "PNG", optimize=True)
+
+
+def render_mean_deviations(
+    rows: Sequence[ProbabilityRow], output: Path, title: str
+) -> None:
+    width, height = 1200, 720
+    image = Image.new("RGB", (width, height), BACKGROUND)
+    draw = ImageDraw.Draw(image)
+    draw_centered_text(draw, width / 2, 22, title, load_font(30, bold=True))
+    draw_centered_text(
+        draw,
+        width / 2,
+        58,
+        "Mean absolute deviation from ideal avalanche probability: mean(|p - 0.5|)",
+        load_font(15),
+        MUTED,
+    )
+
+    plot_left, plot_right = 90, width - 40
+    plot_top, plot_bottom = 110, height - 120
+    plot_height = plot_bottom - plot_top
+    deviations = [mean_absolute_deviation(row.probabilities) for row in rows]
+    scale = make_scale(deviations, hard_limits=(0.0, 0.5))
+    scale_span = scale.maximum - scale.minimum
+    tick_font = load_font(13)
+    label_font = load_font(15)
+    value_font = load_font(14, bold=True)
+
+    for value in scale.ticks:
+        y = round(plot_bottom - (value - scale.minimum) / scale_span * plot_height)
+        draw.line((plot_left, y, plot_right, y), fill=GRID, width=1)
+        label = f"{value:.3g}"
+        draw.text(
+            (plot_left - 10 - text_width(draw, label, tick_font), y - 7),
+            label,
+            font=tick_font,
+            fill=MUTED,
+        )
+
+    slot_width = (plot_right - plot_left) / len(rows)
+    bar_width = min(105, max(20, int(slot_width * 0.62)))
+    for index, (row, deviation) in enumerate(zip(rows, deviations, strict=True)):
+        center = plot_left + (index + 0.5) * slot_width
+        x0 = round(center - bar_width / 2)
+        x1 = round(center + bar_width / 2)
+        if math.isfinite(deviation):
+            y = round(
+                plot_bottom
+                - (deviation - scale.minimum) / scale_span * plot_height
+            )
+            draw.rectangle((x0, y, x1, plot_bottom), fill=COLORS[index % len(COLORS)])
+            value_label = f"{deviation:.4f}"
+        else:
+            y = plot_bottom
+            value_label = "n/a"
+        draw_centered_text(draw, center, max(plot_top, y - 22), value_label, value_font)
+        draw_centered_text(draw, center, plot_bottom + 12, row.operation, label_font)
+        draw_centered_text(
+            draw, center, plot_bottom + 36, f"pairs: {row.pairs}", tick_font, MUTED
+        )
+
+    draw.line((plot_left, plot_top, plot_left, plot_bottom), fill=TEXT, width=2)
+    draw.line((plot_left, plot_bottom, plot_right, plot_bottom), fill=TEXT, width=2)
+    output.parent.mkdir(parents=True, exist_ok=True)
+    image.save(output, "PNG", optimize=True)
+
+
+def generate_plots(source: Path, output_directory: Path | None = None) -> tuple[Path, Path]:
+    rows = read_probability_map(source)
+    destination = output_directory or source.parent
+    bit_output = destination / f"{source.stem}_bits.png"
+    deviation_output = destination / f"{source.stem}_deviation.png"
+    display_name = source.stem
+
+    render_bit_probabilities(rows, bit_output, f"{display_name}: per-bit avalanche map")
+    render_mean_deviations(
+        rows,
+        deviation_output,
+        f"{display_name}: deviation from p=0.5",
+    )
+    return bit_output, deviation_output
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Создаёт две PNG-гистограммы из CSV вероятностной карты: "
+            "вероятности по битам для каждой операции и среднее |p-0.5|."
+        )
+    )
+    parser.add_argument("csv_file", type=Path, help="CSV из probability_map.py")
+    parser.add_argument(
+        "-o",
+        "--output-dir",
+        type=Path,
+        help="каталог PNG (по умолчанию каталог исходного CSV)",
+    )
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    try:
+        outputs = generate_plots(args.csv_file, args.output_dir)
+    except ValueError as error:
+        raise SystemExit(f"error: {error}") from error
+
+    for output in outputs:
+        print(f"wrote {output}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

hash_testing/new_testing/probability_map.py

@@ -0,0 +1,354 @@
+#!/usr/bin/env python3
+"""Build bit-probability maps for hash functions and word mutation types.
+
+For every hash listed in HASHES, the script compares each source word hash with
+hashes of generated similar words.  A table cell contains the probability that
+the corresponding output bit changed (XOR with its source word hash).
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import re
+import stat
+import subprocess
+import sys
+from collections.abc import Mapping, Sequence
+from pathlib import Path
+from typing import TextIO
+
+from generate_input import (
+    DEFAULT_ALPHABET,
+    MUTATIONS,
+    MutationError,
+    generate_words,
+    parse_operation,
+)
+
+ROOT = Path(__file__).resolve().parent
+HASH_FUNCS_DIR = ROOT / "hash_funcs"
+DEFAULT_OUTPUT_DIR = ROOT / "probability_maps"
+
+# Add directory names from hash_funcs here to include more implementations.
+HASHES = [
+    "jenkinsOAAT",
+    "xxh64",
+    "t1ha2",
+]
+
+HEX_HASH = re.compile(r"(?:0[xX])?([0-9a-fA-F]+)")
+ProbabilityRow = tuple[int, list[float]]
+
+
+class HashToolError(RuntimeError):
+    """Raised when a hash executable cannot be prepared or invoked."""
+
+
+def prepare_hash(hash_name: str, hash_funcs_dir: Path = HASH_FUNCS_DIR) -> Path:
+    """Return an executable hash tool, building or preparing it if necessary."""
+    if not hash_name or Path(hash_name).name != hash_name:
+        raise HashToolError(f"некорректное имя хэша: {hash_name!r}")
+
+    hash_directory = hash_funcs_dir / hash_name
+    if not hash_directory.is_dir():
+        raise HashToolError(f"не найдена папка хэша: {hash_directory}")
+
+    binary = hash_directory / "bin_hash"
+    if binary.is_file():
+        binary.chmod(binary.stat().st_mode | stat.S_IXUSR)
+        return binary
+
+    sources = (
+        (hash_directory / "bin_hash.c", "cc"),
+        (hash_directory / "bin_hash.cpp", "c++"),
+        (hash_directory / "bin_hash.cc", "c++"),
+        (hash_directory / "bin_hash.cxx", "c++"),
+    )
+    for source, compiler in sources:
+        if not source.is_file():
+            continue
+        command = [
+            compiler,
+            "-O2",
+            "-Wall",
+            "-Wextra",
+            "-Wpedantic",
+            "-Werror",
+            str(source),
+            "-o",
+            str(binary),
+        ]
+        if compiler == "cc":
+            command[1:1] = ["-std=c11"]
+        else:
+            command[1:1] = ["-std=c++17"]
+        result = subprocess.run(command, text=True, capture_output=True)
+        if result.returncode != 0:
+            details = result.stderr.strip() or result.stdout.strip()
+            raise HashToolError(f"не удалось скомпилировать {source}: {details}")
+        return binary
+
+    python_source = hash_directory / "bin_hash.py"
+    if python_source.is_file():
+        content = python_source.read_text(encoding="utf-8")
+        if not content.startswith("#!"):
+            python_source.write_text(
+                "#!/usr/bin/env python3\n" + content,
+                encoding="utf-8",
+            )
+        python_source.chmod(
+            python_source.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH
+        )
+        return python_source
+
+    raise HashToolError(
+        f"для {hash_name} не найден bin_hash, bin_hash.c/cpp/cc/cxx или bin_hash.py"
+    )
+
+
+def run_hash(executable: Path, word: str) -> tuple[int, int]:
+    """Run a hash tool and return its integer value and explicit output width."""
+    try:
+        result = subprocess.run(
+            [str(executable), word],
+            text=True,
+            capture_output=True,
+            check=False,
+        )
+    except OSError as error:
+        raise HashToolError(f"не удалось запустить {executable}: {error}") from error
+
+    if result.returncode != 0:
+        details = result.stderr.strip() or result.stdout.strip()
+        raise HashToolError(
+            f"{executable} завершился с кодом {result.returncode}: {details}"
+        )
+
+    output = result.stdout.strip()
+    match = HEX_HASH.fullmatch(output)
+    if match is None:
+        raise HashToolError(f"{executable} вернул не шестнадцатеричный хэш: {output!r}")
+
+    digits = match.group(1)
+    return int(digits, 16), len(digits) * 4
+
+
+def bit_probabilities(
+    source_hash: int, changed_hashes: Sequence[int], bits: int
+) -> list[float]:
+    """Calculate per-bit change probabilities, ordered from LSB to MSB."""
+    if bits < 1:
+        raise ValueError("число бит должно быть положительным")
+    if not changed_hashes:
+        raise ValueError("список изменённых хэшей не должен быть пустым")
+
+    changed_counts = [0] * bits
+    for changed_hash in changed_hashes:
+        difference = source_hash ^ changed_hash
+        for bit in range(bits):
+            changed_counts[bit] += (difference >> bit) & 1
+
+    sample_count = len(changed_hashes)
+    return [count / sample_count for count in changed_counts]
+
+
+def write_csv_table(stream: TextIO, rows: Mapping[str, ProbabilityRow]) -> None:
+    """Write one operation-by-bit probability table as CSV."""
+    if not rows:
+        raise ValueError("таблица вероятностей не должна быть пустой")
+
+    widths = {len(probabilities) for _pair_count, probabilities in rows.values()}
+    if len(widths) != 1:
+        raise ValueError("все строки таблицы должны иметь одинаковое число бит")
+    bits = widths.pop()
+
+    writer = csv.writer(stream, lineterminator="\n")
+    writer.writerow(["operation", "pairs", *(f"bit_{bit}" for bit in range(bits))])
+    for operation, (pair_count, probabilities) in rows.items():
+        writer.writerow(
+            [
+                operation,
+                pair_count,
+                *(f"{probability:.6f}" for probability in probabilities),
+            ]
+        )
+
+
+def build_probability_table(
+    executable: Path,
+    sources: Sequence[str],
+    operations: Sequence[str],
+    count: int,
+    operation_count: int,
+    alphabet: str,
+    seed: int | None,
+    max_attempts: int | None,
+) -> dict[str, ProbabilityRow]:
+    """Aggregate avalanche probabilities across all source words."""
+    if not sources:
+        raise ValueError("нужно указать хотя бы одно исходное слово")
+
+    source_hashes: list[int] = []
+    bits: int | None = None
+    for source in sources:
+        source_hash, source_bits = run_hash(executable, source)
+        if bits is None:
+            bits = source_bits
+        elif source_bits != bits:
+            raise HashToolError(
+                f"{executable} вернул хэши разной ширины: "
+                f"{bits} и {source_bits} бит"
+            )
+        source_hashes.append(source_hash)
+
+    assert bits is not None
+    table: dict[str, ProbabilityRow] = {}
+
+    for operation in operations:
+        differences: list[int] = []
+        for source, source_hash in zip(sources, source_hashes, strict=True):
+            words = generate_words(
+                source=source,
+                count=count,
+                operation=operation,
+                operation_count=operation_count,
+                alphabet=alphabet,
+                seed=seed,
+                max_attempts=max_attempts,
+            )
+            if len(words) < count:
+                print(
+                    f"warning: operation={operation} word={source!r}: "
+                    f"generated {len(words)} of at most {count} unique words",
+                    file=sys.stderr,
+                )
+            for word in words:
+                changed_hash, changed_bits = run_hash(executable, word)
+                if changed_bits != bits:
+                    raise HashToolError(
+                        f"{executable} вернул хэши разной ширины: "
+                        f"{bits} и {changed_bits} бит"
+                    )
+                differences.append(source_hash ^ changed_hash)
+
+        if differences:
+            table[operation] = (
+                len(differences),
+                bit_probabilities(0, differences, bits),
+            )
+        else:
+            table[operation] = (0, [float("nan")] * bits)
+
+    return table
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Строит для каждого хэша CSV-таблицу вероятностей изменения "
+            "выходных битов. bit_0 — младший бит."
+        )
+    )
+    parser.add_argument(
+        "words",
+        nargs="+",
+        help="одно или несколько исходных ASCII-слов",
+    )
+    parser.add_argument(
+        "-o",
+        "--operation",
+        action="append",
+        type=parse_operation,
+        help=(
+            "тип изменения (номер или имя как в generate_input.py); "
+            "можно повторять, по умолчанию используются все типы"
+        ),
+    )
+    parser.add_argument(
+        "-n",
+        "--count",
+        type=int,
+        default=10,
+        help=(
+            "верхняя граница числа уникальных изменённых слов для каждого "
+            "исходного слова и типа (по умолчанию: 10)"
+        ),
+    )
+    parser.add_argument(
+        "-k",
+        "--operations",
+        type=int,
+        default=1,
+        help="число операций над каждым словом (по умолчанию: 1)",
+    )
+    parser.add_argument(
+        "--alphabet",
+        default=DEFAULT_ALPHABET,
+        help="алфавит для добавления и замены",
+    )
+    parser.add_argument(
+        "--seed", type=int, help="seed генератора для воспроизводимого результата"
+    )
+    parser.add_argument(
+        "--max-attempts",
+        type=int,
+        help="предельное число попыток собрать уникальные слова",
+    )
+    parser.add_argument(
+        "--output",
+        type=Path,
+        default=DEFAULT_OUTPUT_DIR,
+        help=("каталог для CSV-таблиц " f"(по умолчанию: {DEFAULT_OUTPUT_DIR})"),
+    )
+    parser.add_argument(
+        "--hash",
+        dest="hashes",
+        action="append",
+        help="проверить только указанный хэш; можно повторять (по умолчанию HASHES)",
+    )
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+
+    if args.count < 1:
+        parser.error("--count должен быть положительным")
+    if args.operations < 1:
+        parser.error("--operations должен быть положительным")
+    if args.max_attempts is not None and args.max_attempts < 1:
+        parser.error("--max-attempts должен быть положительным")
+
+    operations = list(dict.fromkeys(args.operation or MUTATIONS.keys()))
+    hashes = list(dict.fromkeys(args.hashes or HASHES))
+    if not hashes:
+        parser.error("массив HASHES не должен быть пустым")
+
+    args.output.mkdir(parents=True, exist_ok=True)
+    try:
+        for hash_name in hashes:
+            executable = prepare_hash(hash_name)
+            table = build_probability_table(
+                executable=executable,
+                sources=args.words,
+                operations=operations,
+                count=args.count,
+                operation_count=args.operations,
+                alphabet=args.alphabet,
+                seed=args.seed,
+                max_attempts=args.max_attempts,
+            )
+            output_path = args.output / f"{hash_name}.csv"
+            with output_path.open("w", encoding="utf-8", newline="") as stream:
+                write_csv_table(stream, table)
+            print(f"wrote {output_path}")
+    except (HashToolError, MutationError, ValueError) as error:
+        parser.error(str(error))
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

hash_testing/new_testing/probability_maps/jenkinsOAAT.csv

@@ -0,0 +1,8 @@
+operation,pairs,bit_0,bit_1,bit_2,bit_3,bit_4,bit_5,bit_6,bit_7,bit_8,bit_9,bit_10,bit_11,bit_12,bit_13,bit_14,bit_15,bit_16,bit_17,bit_18,bit_19,bit_20,bit_21,bit_22,bit_23,bit_24,bit_25,bit_26,bit_27,bit_28,bit_29,bit_30,bit_31
+replace,1984,0.484375,0.499496,0.494960,0.497984,0.506048,0.490423,0.501512,0.511593,0.496472,0.499496,0.515121,0.504536,0.485383,0.500504,0.496976,0.477319,0.510081,0.490423,0.492440,0.501512,0.502016,0.484879,0.490927,0.490927,0.511593,0.494960,0.512097,0.523185,0.498992,0.505544,0.496472,0.500504
+delete,31,0.612903,0.451613,0.612903,0.516129,0.419355,0.548387,0.451613,0.612903,0.419355,0.548387,0.677419,0.419355,0.580645,0.516129,0.451613,0.516129,0.516129,0.645161,0.548387,0.516129,0.548387,0.419355,0.516129,0.419355,0.645161,0.612903,0.516129,0.580645,0.516129,0.419355,0.580645,0.354839
+add,2173,0.501611,0.492867,0.500690,0.505292,0.515877,0.505292,0.505292,0.505752,0.497929,0.502991,0.485044,0.503451,0.496088,0.524160,0.517257,0.507593,0.489646,0.476760,0.496549,0.521859,0.514956,0.514036,0.513116,0.508053,0.492867,0.503451,0.518638,0.506673,0.495628,0.506213,0.518178,0.489185
+swap,28,0.607143,0.500000,0.428571,0.535714,0.392857,0.535714,0.500000,0.357143,0.571429,0.428571,0.678571,0.750000,0.428571,0.464286,0.392857,0.464286,0.500000,0.428571,0.428571,0.428571,0.464286,0.428571,0.500000,0.678571,0.571429,0.464286,0.607143,0.571429,0.571429,0.535714,0.428571,0.607143
+case,29,0.344828,0.551724,0.517241,0.448276,0.448276,0.310345,0.482759,0.344828,0.379310,0.413793,0.413793,0.448276,0.620690,0.620690,0.448276,0.586207,0.517241,0.551724,0.482759,0.448276,0.413793,0.517241,0.275862,0.655172,0.551724,0.586207,0.551724,0.482759,0.448276,0.551724,0.517241,0.655172
+first,186,0.500000,0.500000,0.494624,0.467742,0.521505,0.451613,0.489247,0.478495,0.521505,0.543011,0.516129,0.510753,0.478495,0.510753,0.510753,0.510753,0.456989,0.473118,0.456989,0.543011,0.489247,0.478495,0.456989,0.467742,0.500000,0.483871,0.483871,0.489247,0.559140,0.559140,0.462366,0.483871
+last,186,0.510753,0.516129,0.510753,0.467742,0.510753,0.505376,0.473118,0.516129,0.500000,0.516129,0.516129,0.521505,0.483871,0.526882,0.510753,0.451613,0.548387,0.489247,0.505376,0.430108,0.526882,0.456989,0.478495,0.505376,0.516129,0.537634,0.526882,0.478495,0.467742,0.494624,0.446237,0.548387

hash_testing/new_testing/probability_maps/t1ha2.csv

@@ -0,0 +1,8 @@
+operation,pairs,bit_0,bit_1,bit_2,bit_3,bit_4,bit_5,bit_6,bit_7,bit_8,bit_9,bit_10,bit_11,bit_12,bit_13,bit_14,bit_15,bit_16,bit_17,bit_18,bit_19,bit_20,bit_21,bit_22,bit_23,bit_24,bit_25,bit_26,bit_27,bit_28,bit_29,bit_30,bit_31,bit_32,bit_33,bit_34,bit_35,bit_36,bit_37,bit_38,bit_39,bit_40,bit_41,bit_42,bit_43,bit_44,bit_45,bit_46,bit_47,bit_48,bit_49,bit_50,bit_51,bit_52,bit_53,bit_54,bit_55,bit_56,bit_57,bit_58,bit_59,bit_60,bit_61,bit_62,bit_63
+replace,1984,0.511593,0.515121,0.510081,0.525706,0.499496,0.504032,0.495464,0.494960,0.522681,0.521169,0.507056,0.484879,0.502016,0.482359,0.513609,0.513105,0.500504,0.503024,0.496472,0.509073,0.491431,0.488911,0.495464,0.491431,0.494960,0.491431,0.496976,0.496976,0.502016,0.511593,0.502016,0.496976,0.504536,0.499496,0.494456,0.496976,0.496976,0.490927,0.523690,0.489919,0.495968,0.478327,0.492944,0.529234,0.495968,0.496472,0.500000,0.493448,0.519657,0.510585,0.493448,0.496976,0.495464,0.501512,0.512097,0.500504,0.494960,0.494456,0.512097,0.494456,0.494960,0.495464,0.494456,0.515121
+delete,31,0.741935,0.387097,0.548387,0.354839,0.483871,0.451613,0.483871,0.612903,0.451613,0.645161,0.516129,0.419355,0.419355,0.483871,0.516129,0.483871,0.548387,0.741935,0.354839,0.483871,0.645161,0.516129,0.516129,0.548387,0.516129,0.354839,0.387097,0.387097,0.354839,0.548387,0.419355,0.516129,0.548387,0.516129,0.322581,0.419355,0.387097,0.483871,0.483871,0.419355,0.451613,0.612903,0.451613,0.419355,0.516129,0.580645,0.548387,0.483871,0.387097,0.612903,0.419355,0.516129,0.483871,0.548387,0.516129,0.516129,0.677419,0.451613,0.483871,0.451613,0.516129,0.612903,0.516129,0.483871
+add,2173,0.496088,0.508514,0.489646,0.488725,0.500230,0.514496,0.491486,0.479521,0.499770,0.484123,0.488725,0.495168,0.499310,0.484584,0.498850,0.530143,0.488265,0.484123,0.519098,0.494248,0.497469,0.505752,0.501611,0.520018,0.497929,0.512195,0.498389,0.474919,0.498389,0.504372,0.491026,0.514496,0.496549,0.479521,0.492867,0.499310,0.499770,0.498389,0.480902,0.493787,0.492407,0.505292,0.492867,0.499310,0.494708,0.511735,0.524620,0.506213,0.518638,0.518178,0.492867,0.498850,0.498389,0.502531,0.509434,0.498850,0.492867,0.499310,0.501611,0.506213,0.482743,0.503451,0.511275,0.493787
+swap,28,0.571429,0.571429,0.607143,0.535714,0.428571,0.392857,0.428571,0.535714,0.571429,0.535714,0.571429,0.464286,0.285714,0.714286,0.607143,0.500000,0.500000,0.357143,0.392857,0.571429,0.678571,0.464286,0.428571,0.392857,0.357143,0.392857,0.607143,0.571429,0.392857,0.642857,0.464286,0.678571,0.750000,0.428571,0.607143,0.428571,0.535714,0.500000,0.571429,0.642857,0.428571,0.357143,0.535714,0.571429,0.535714,0.571429,0.535714,0.464286,0.607143,0.642857,0.642857,0.678571,0.285714,0.500000,0.642857,0.571429,0.500000,0.357143,0.464286,0.428571,0.571429,0.571429,0.678571,0.535714
+case,29,0.413793,0.413793,0.620690,0.586207,0.379310,0.586207,0.482759,0.448276,0.551724,0.448276,0.655172,0.586207,0.655172,0.586207,0.379310,0.551724,0.517241,0.310345,0.517241,0.517241,0.310345,0.620690,0.482759,0.655172,0.551724,0.379310,0.413793,0.482759,0.517241,0.517241,0.482759,0.448276,0.586207,0.448276,0.620690,0.448276,0.448276,0.448276,0.551724,0.586207,0.551724,0.413793,0.413793,0.586207,0.517241,0.482759,0.551724,0.482759,0.586207,0.586207,0.379310,0.482759,0.586207,0.379310,0.482759,0.344828,0.620690,0.482759,0.448276,0.551724,0.482759,0.551724,0.344828,0.482759
+first,186,0.500000,0.500000,0.526882,0.526882,0.489247,0.483871,0.456989,0.532258,0.510753,0.483871,0.537634,0.456989,0.500000,0.424731,0.500000,0.435484,0.532258,0.456989,0.478495,0.532258,0.430108,0.489247,0.521505,0.478495,0.478495,0.532258,0.478495,0.532258,0.500000,0.473118,0.473118,0.537634,0.575269,0.526882,0.505376,0.526882,0.424731,0.462366,0.548387,0.478495,0.462366,0.532258,0.500000,0.473118,0.473118,0.559140,0.510753,0.526882,0.564516,0.521505,0.532258,0.521505,0.564516,0.537634,0.467742,0.473118,0.500000,0.424731,0.424731,0.462366,0.494624,0.489247,0.537634,0.559140
+last,186,0.510753,0.543011,0.580645,0.516129,0.516129,0.564516,0.500000,0.478495,0.489247,0.494624,0.494624,0.478495,0.521505,0.462366,0.526882,0.521505,0.564516,0.451613,0.489247,0.543011,0.537634,0.467742,0.467742,0.526882,0.500000,0.526882,0.478495,0.532258,0.462366,0.473118,0.569892,0.505376,0.516129,0.494624,0.456989,0.478495,0.537634,0.473118,0.516129,0.494624,0.440860,0.505376,0.537634,0.548387,0.408602,0.569892,0.537634,0.559140,0.494624,0.559140,0.526882,0.559140,0.489247,0.569892,0.537634,0.543011,0.543011,0.494624,0.537634,0.505376,0.473118,0.494624,0.532258,0.483871

hash_testing/new_testing/probability_maps/xxh64.csv

@@ -0,0 +1,8 @@
+operation,pairs,bit_0,bit_1,bit_2,bit_3,bit_4,bit_5,bit_6,bit_7,bit_8,bit_9,bit_10,bit_11,bit_12,bit_13,bit_14,bit_15,bit_16,bit_17,bit_18,bit_19,bit_20,bit_21,bit_22,bit_23,bit_24,bit_25,bit_26,bit_27,bit_28,bit_29,bit_30,bit_31,bit_32,bit_33,bit_34,bit_35,bit_36,bit_37,bit_38,bit_39,bit_40,bit_41,bit_42,bit_43,bit_44,bit_45,bit_46,bit_47,bit_48,bit_49,bit_50,bit_51,bit_52,bit_53,bit_54,bit_55,bit_56,bit_57,bit_58,bit_59,bit_60,bit_61,bit_62,bit_63
+replace,1984,0.491431,0.497984,0.532258,0.504536,0.494456,0.519657,0.516129,0.493448,0.501008,0.519657,0.490423,0.495968,0.516129,0.498488,0.491935,0.509073,0.520665,0.487903,0.497984,0.487903,0.502016,0.504536,0.502520,0.510081,0.512097,0.492440,0.505544,0.492944,0.491431,0.500000,0.509577,0.511593,0.491431,0.485383,0.481351,0.493448,0.498992,0.498992,0.499496,0.504032,0.495968,0.483367,0.495464,0.498488,0.512097,0.489415,0.502520,0.496472,0.501008,0.495464,0.503528,0.507056,0.510081,0.504032,0.487399,0.523690,0.511089,0.472278,0.496976,0.501008,0.503528,0.504032,0.494456,0.491935
+delete,31,0.548387,0.516129,0.451613,0.580645,0.548387,0.483871,0.548387,0.387097,0.451613,0.451613,0.451613,0.387097,0.677419,0.419355,0.354839,0.612903,0.387097,0.419355,0.548387,0.645161,0.451613,0.548387,0.451613,0.322581,0.516129,0.483871,0.516129,0.387097,0.483871,0.483871,0.483871,0.451613,0.483871,0.516129,0.645161,0.483871,0.580645,0.612903,0.677419,0.516129,0.516129,0.516129,0.516129,0.516129,0.677419,0.419355,0.516129,0.612903,0.612903,0.451613,0.483871,0.419355,0.516129,0.419355,0.580645,0.483871,0.645161,0.419355,0.419355,0.677419,0.451613,0.483871,0.419355,0.516129
+add,2173,0.506213,0.502531,0.484584,0.506213,0.505292,0.469397,0.497929,0.499310,0.487345,0.516797,0.502071,0.504832,0.515416,0.498389,0.486884,0.492867,0.505752,0.484123,0.513116,0.503451,0.508974,0.494248,0.501150,0.509894,0.485964,0.482743,0.490106,0.501150,0.487345,0.491486,0.515416,0.502071,0.497009,0.480902,0.499310,0.510815,0.509894,0.499310,0.511735,0.510354,0.498389,0.485504,0.497009,0.511275,0.504832,0.490566,0.495628,0.495628,0.498850,0.497009,0.485504,0.498850,0.506673,0.488265,0.492407,0.509434,0.495168,0.489185,0.500230,0.489646,0.498850,0.493327,0.493327,0.504832
+swap,28,0.535714,0.607143,0.428571,0.321429,0.535714,0.464286,0.500000,0.357143,0.464286,0.500000,0.464286,0.607143,0.535714,0.357143,0.607143,0.321429,0.428571,0.571429,0.464286,0.357143,0.357143,0.464286,0.464286,0.607143,0.500000,0.642857,0.642857,0.535714,0.428571,0.464286,0.535714,0.535714,0.285714,0.607143,0.500000,0.571429,0.428571,0.642857,0.500000,0.607143,0.571429,0.392857,0.500000,0.535714,0.535714,0.392857,0.571429,0.285714,0.642857,0.500000,0.535714,0.321429,0.428571,0.428571,0.464286,0.678571,0.642857,0.571429,0.607143,0.535714,0.535714,0.571429,0.285714,0.357143
+case,29,0.448276,0.551724,0.586207,0.620690,0.482759,0.655172,0.655172,0.586207,0.517241,0.379310,0.551724,0.448276,0.482759,0.482759,0.586207,0.448276,0.517241,0.448276,0.448276,0.517241,0.551724,0.620690,0.655172,0.655172,0.551724,0.586207,0.655172,0.586207,0.482759,0.379310,0.586207,0.517241,0.517241,0.551724,0.517241,0.655172,0.413793,0.551724,0.551724,0.448276,0.517241,0.379310,0.517241,0.517241,0.586207,0.310345,0.620690,0.379310,0.413793,0.448276,0.586207,0.689655,0.482759,0.310345,0.551724,0.413793,0.448276,0.517241,0.586207,0.551724,0.517241,0.448276,0.448276,0.448276
+first,186,0.521505,0.456989,0.543011,0.516129,0.505376,0.494624,0.569892,0.516129,0.564516,0.500000,0.500000,0.500000,0.516129,0.446237,0.580645,0.532258,0.521505,0.521505,0.473118,0.462366,0.462366,0.473118,0.526882,0.500000,0.500000,0.516129,0.532258,0.532258,0.489247,0.559140,0.462366,0.564516,0.521505,0.467742,0.456989,0.516129,0.537634,0.483871,0.548387,0.500000,0.532258,0.516129,0.478495,0.478495,0.462366,0.483871,0.494624,0.526882,0.537634,0.478495,0.446237,0.473118,0.478495,0.564516,0.435484,0.467742,0.526882,0.494624,0.494624,0.494624,0.543011,0.478495,0.430108,0.494624
+last,186,0.526882,0.446237,0.526882,0.521505,0.494624,0.521505,0.521505,0.483871,0.516129,0.500000,0.478495,0.500000,0.564516,0.500000,0.381720,0.500000,0.537634,0.473118,0.532258,0.543011,0.505376,0.510753,0.483871,0.456989,0.564516,0.489247,0.596774,0.526882,0.510753,0.462366,0.494624,0.494624,0.408602,0.548387,0.505376,0.543011,0.494624,0.467742,0.462366,0.440860,0.548387,0.505376,0.516129,0.510753,0.543011,0.462366,0.478495,0.505376,0.478495,0.532258,0.510753,0.586022,0.526882,0.478495,0.467742,0.516129,0.543011,0.478495,0.510753,0.478495,0.494624,0.505376,0.548387,0.478495

hash_testing/new_testing/results/cpp_provides_p11_bit_distribution/jenkinsOAAT.csv

@@ -0,0 +1,33 @@
+bit,ones,zeros,probability,deviation_from_0.5
+0,287790,289719,0.498329896,0.001670104
+1,288092,289417,0.498852832,0.001147168
+2,288750,288759,0.499992208,0.000007792
+3,288546,288963,0.499638967,0.000361033
+4,288778,288731,0.500040692,0.000040692
+5,289060,288449,0.500528996,0.000528996
+6,288814,288695,0.500103029,0.000103029
+7,288788,288721,0.500058008,0.000058008
+8,288712,288797,0.499926408,0.000073592
+9,288209,289300,0.499055426,0.000944574
+10,289080,288429,0.500563628,0.000563628
+11,288964,288545,0.500362765,0.000362765
+12,289040,288469,0.500494365,0.000494365
+13,289175,288334,0.500728127,0.000728127
+14,288883,288626,0.500222507,0.000222507
+15,288908,288601,0.500265797,0.000265797
+16,288259,289250,0.499142005,0.000857995
+17,288493,289016,0.499547193,0.000452807
+18,288101,289408,0.498868416,0.001131584
+19,289164,288345,0.500709080,0.000709080
+20,288817,288692,0.500108223,0.000108223
+21,288723,288786,0.499945455,0.000054545
+22,288577,288932,0.499692645,0.000307355
+23,288523,288986,0.499599140,0.000400860
+24,288625,288884,0.499775761,0.000224239
+25,288128,289381,0.498915168,0.001084832
+26,288821,288688,0.500115150,0.000115150
+27,288762,288747,0.500012987,0.000012987
+28,289539,287970,0.501358420,0.001358420
+29,288493,289016,0.499547193,0.000452807
+30,288614,288895,0.499756714,0.000243286
+31,288681,288828,0.499872729,0.000127271

hash_testing/new_testing/results/cpp_provides_p11_bit_distribution/manifest.txt

@@ -0,0 +1,8 @@
+corpus=input_symbols/cpp_provides_p11.txt
+corpus_bytes=54941449
+corpus_sha256=bf8e760bdf4d52b42040cff4625de73988d8dd6b517566efd186c71e3382753d
+samples=577509
+hashes=jenkinsOAAT,xxh64,t1ha2
+metric=P(hash_output_bit=1)
+bit_order=bit_0_is_LSB
+input_format=one_ASCII_symbol_per_line

hash_testing/new_testing/results/cpp_provides_p11_bit_distribution/summary.csv

@@ -0,0 +1,4 @@
+hash,samples,bits,mean_probability,mean_abs_deviation,max_abs_deviation,min_probability,max_probability
+jenkinsOAAT,577509,32,0.499879060,0.000475425,0.001670104,0.498329896,0.501358420
+xxh64,577509,64,0.499871404,0.000497665,0.001704735,0.498295265,0.501221626
+t1ha2,577509,64,0.500008550,0.000532513,0.001876161,0.498123839,0.501256257

hash_testing/new_testing/results/cpp_provides_p11_bit_distribution/t1ha2.csv

@@ -0,0 +1,65 @@
+bit,ones,zeros,probability,deviation_from_0.5
+0,288524,288985,0.499600872,0.000399128
+1,289273,288236,0.500897822,0.000897822
+2,289241,288268,0.500842411,0.000842411
+3,289050,288459,0.500511680,0.000511680
+4,288827,288682,0.500125539,0.000125539
+5,289341,288168,0.501015569,0.001015569
+6,288570,288939,0.499680524,0.000319476
+7,289480,288029,0.501256257,0.001256257
+8,289469,288040,0.501237210,0.001237210
+9,289359,288150,0.501046737,0.001046737
+10,288340,289169,0.499282262,0.000717738
+11,288850,288659,0.500165365,0.000165365
+12,288401,289108,0.499387888,0.000612112
+13,288828,288681,0.500127271,0.000127271
+14,288809,288700,0.500094371,0.000094371
+15,288437,289072,0.499450225,0.000549775
+16,287671,289838,0.498123839,0.001876161
+17,288776,288733,0.500037229,0.000037229
+18,288058,289451,0.498793958,0.001206042
+19,289300,288209,0.500944574,0.000944574
+20,288721,288788,0.499941992,0.000058008
+21,288403,289106,0.499391351,0.000608649
+22,288749,288760,0.499990476,0.000009524
+23,289382,288127,0.501086563,0.001086563
+24,288503,289006,0.499564509,0.000435491
+25,289237,288272,0.500835485,0.000835485
+26,288918,288591,0.500283112,0.000283112
+27,288889,288620,0.500232897,0.000232897
+28,288205,289304,0.499048500,0.000951500
+29,288637,288872,0.499796540,0.000203460
+30,289047,288462,0.500506486,0.000506486
+31,288479,289030,0.499522951,0.000477049
+32,289147,288362,0.500679643,0.000679643
+33,288429,289080,0.499436372,0.000563628
+34,288675,288834,0.499862340,0.000137660
+35,288817,288692,0.500108223,0.000108223
+36,288302,289207,0.499216462,0.000783538
+37,288197,289312,0.499034647,0.000965353
+38,289472,288037,0.501242405,0.001242405
+39,289181,288328,0.500738517,0.000738517
+40,289046,288463,0.500504754,0.000504754
+41,288910,288599,0.500269260,0.000269260
+42,288608,288901,0.499746324,0.000253676
+43,288745,288764,0.499983550,0.000016450
+44,288429,289080,0.499436372,0.000563628
+45,288738,288771,0.499971429,0.000028571
+46,288425,289084,0.499429446,0.000570554
+47,288857,288652,0.500177486,0.000177486
+48,288795,288714,0.500070129,0.000070129
+49,288155,289354,0.498961921,0.001038079
+50,288639,288870,0.499800003,0.000199997
+51,289013,288496,0.500447612,0.000447612
+52,288717,288792,0.499935066,0.000064934
+53,289130,288379,0.500650206,0.000650206
+54,288936,288573,0.500314281,0.000314281
+55,288904,288605,0.500258870,0.000258870
+56,288616,288893,0.499760177,0.000239823
+57,288873,288636,0.500205192,0.000205192
+58,288340,289169,0.499282262,0.000717738
+59,288507,289002,0.499571435,0.000428565
+60,288986,288523,0.500400860,0.000400860
+61,288162,289347,0.498974042,0.001025958
+62,288354,289155,0.499306504,0.000693496
+63,288725,288784,0.499948919,0.000051081

hash_testing/new_testing/results/cpp_provides_p11_bit_distribution/xxh64.csv

@@ -0,0 +1,65 @@
+bit,ones,zeros,probability,deviation_from_0.5
+0,288778,288731,0.500040692,0.000040692
+1,288503,289006,0.499564509,0.000435491
+2,288868,288641,0.500196534,0.000196534
+3,288638,288871,0.499798272,0.000201728
+4,288821,288688,0.500115150,0.000115150
+5,288588,288921,0.499711693,0.000288307
+6,288882,288627,0.500220776,0.000220776
+7,289460,288049,0.501221626,0.001221626
+8,288866,288643,0.500193071,0.000193071
+9,288162,289347,0.498974042,0.001025958
+10,289120,288389,0.500632891,0.000632891
+11,288842,288667,0.500151513,0.000151513
+12,288456,289053,0.499483125,0.000516875
+13,288410,289099,0.499403472,0.000596528
+14,289202,288307,0.500774880,0.000774880
+15,288850,288659,0.500165365,0.000165365
+16,288499,289010,0.499557583,0.000442417
+17,288163,289346,0.498975774,0.001024226
+18,288220,289289,0.499074473,0.000925527
+19,288892,288617,0.500238092,0.000238092
+20,288695,288814,0.499896971,0.000103029
+21,288797,288712,0.500073592,0.000073592
+22,288795,288714,0.500070129,0.000070129
+23,288188,289321,0.499019063,0.000980937
+24,288864,288645,0.500189607,0.000189607
+25,288653,288856,0.499824245,0.000175755
+26,289025,288484,0.500468391,0.000468391
+27,288750,288759,0.499992208,0.000007792
+28,289267,288242,0.500887432,0.000887432
+29,288448,289061,0.499469272,0.000530728
+30,288742,288767,0.499978355,0.000021645
+31,288247,289262,0.499121226,0.000878774
+32,288629,288880,0.499782687,0.000217313
+33,289449,288060,0.501202579,0.001202579
+34,288761,288748,0.500011255,0.000011255
+35,288208,289301,0.499053694,0.000946306
+36,288308,289201,0.499226852,0.000773148
+37,289073,288436,0.500551507,0.000551507
+38,288394,289115,0.499375767,0.000624233
+39,289142,288367,0.500670985,0.000670985
+40,288574,288935,0.499687451,0.000312549
+41,288209,289300,0.499055426,0.000944574
+42,288292,289217,0.499199147,0.000800853
+43,288781,288728,0.500045887,0.000045887
+44,287770,289739,0.498295265,0.001704735
+45,289014,288495,0.500449344,0.000449344
+46,289209,288300,0.500787001,0.000787001
+47,289366,288143,0.501058858,0.001058858
+48,288445,289064,0.499464078,0.000535922
+49,288384,289125,0.499358452,0.000641548
+50,288509,289000,0.499574898,0.000425102
+51,288599,288910,0.499730740,0.000269260
+52,288771,288738,0.500028571,0.000028571
+53,287887,289622,0.498497859,0.001502141
+54,288871,288638,0.500201728,0.000201728
+55,288294,289215,0.499202610,0.000797390
+56,288557,288952,0.499658014,0.000341986
+57,288416,289093,0.499413862,0.000586138
+58,288674,288835,0.499860608,0.000139392
+59,288644,288865,0.499808661,0.000191339
+60,288679,288830,0.499869266,0.000130734
+61,288828,288681,0.500127271,0.000127271
+62,288905,288604,0.500260602,0.000260602
+63,289202,288307,0.500774880,0.000774880

hash_testing/new_testing/tests/test_corpus_bit_distribution.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python3
+"""Behavior tests for corpus_bit_distribution.py."""
+
+from __future__ import annotations
+
+import csv
+import io
+import tempfile
+import unittest
+from pathlib import Path
+
+from PIL import Image
+
+import corpus_bit_distribution
+import probability_map
+
+
+class CorpusBitDistributionTests(unittest.TestCase):
+    def test_batch_counts_match_scalar_cli_for_every_hash(self) -> None:
+        words = ["a", "abc", "HashWord", "_ZN4llvm3fooEv"]
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            corpus = Path(temporary_directory) / "corpus.txt"
+            corpus.write_text("".join(f"{word}\n" for word in words), encoding="ascii")
+
+            for hash_name in probability_map.HASHES:
+                with self.subTest(hash_name=hash_name):
+                    distribution = corpus_bit_distribution.measure_hash(
+                        hash_name,
+                        corpus,
+                    )
+                    executable = probability_map.prepare_hash(hash_name)
+                    scalar = [
+                        probability_map.run_hash(executable, word)[0] for word in words
+                    ]
+                    expected_ones = tuple(
+                        sum((value >> bit) & 1 for value in scalar)
+                        for bit in range(distribution.bits)
+                    )
+
+                    self.assertEqual(distribution.samples, len(words))
+                    self.assertEqual(distribution.ones, expected_ones)
+
+    def test_writes_one_csv_row_per_lsb_first_bit(self) -> None:
+        distribution = corpus_bit_distribution.Distribution(
+            hash_name="tiny",
+            samples=4,
+            bits=3,
+            ones=(1, 2, 4),
+        )
+        stream = io.StringIO()
+
+        corpus_bit_distribution.write_distribution_csv(stream, distribution)
+
+        rows = list(csv.reader(io.StringIO(stream.getvalue())))
+        self.assertEqual(
+            rows[0],
+            ["bit", "ones", "zeros", "probability", "deviation_from_0.5"],
+        )
+        self.assertEqual(rows[1], ["0", "1", "3", "0.250000000", "0.250000000"])
+        self.assertEqual(rows[2], ["1", "2", "2", "0.500000000", "0.000000000"])
+        self.assertEqual(rows[3], ["2", "4", "0", "1.000000000", "0.500000000"])
+
+    def test_run_creates_separate_csv_summary_manifest_and_png(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            root = Path(temporary_directory)
+            corpus = root / "corpus.txt"
+            corpus.write_text("a\nabc\nHashWord\n", encoding="ascii")
+            output = root / "results"
+
+            result = corpus_bit_distribution.run(
+                corpus=corpus,
+                output_directory=output,
+                hash_names=["jenkinsOAAT"],
+            )
+
+            self.assertEqual(result[0].samples, 3)
+            self.assertTrue((output / "jenkinsOAAT.csv").is_file())
+            self.assertTrue((output / "summary.csv").is_file())
+            self.assertTrue((output / "manifest.txt").is_file())
+            image_path = output / "bit_distribution.png"
+            self.assertTrue(image_path.is_file())
+            with Image.open(image_path) as image:
+                self.assertEqual(image.format, "PNG")
+                self.assertGreater(image.width, 300)
+                self.assertGreater(image.height, 200)
+
+
+if __name__ == "__main__":
+    unittest.main()

hash_testing/new_testing/tests/test_extract_provided_symbols.py

@@ -0,0 +1,195 @@
+#!/usr/bin/env python3
+"""Behavior tests for extract_provided_symbols.py."""
+
+from __future__ import annotations
+
+import stat
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+SCRIPT = ROOT / "extract_provided_symbols.py"
+
+
+def newc_entry(name: str, data: bytes, mode: int = stat.S_IFREG | 0o644) -> bytes:
+    encoded_name = name.encode("utf-8") + b"\0"
+    fields = (
+        1,
+        mode,
+        0,
+        0,
+        1,
+        0,
+        len(data),
+        0,
+        0,
+        0,
+        0,
+        len(encoded_name),
+        0,
+    )
+    header = b"070701" + b"".join(f"{value:08x}".encode() for value in fields)
+    record = header + encoded_name
+    record += b"\0" * (-len(record) % 4)
+    record += data
+    record += b"\0" * (-len(data) % 4)
+    return record
+
+
+def newc_archive(entries: list[tuple[str, bytes, int]]) -> bytes:
+    payload = b"".join(newc_entry(*entry) for entry in entries)
+    return payload + newc_entry("TRAILER!!!", b"", 0)
+
+
+class ExtractProvidedSymbolsTests(unittest.TestCase):
+    def setUp(self) -> None:
+        self.temporary_directory = tempfile.TemporaryDirectory()
+        self.work = Path(self.temporary_directory.name)
+        self.rpm2cpio = self.work / "rpm2cpio"
+        self.provided_symbols = self.work / "provided_symbols"
+        self.file_command = self.work / "file"
+
+        self.rpm2cpio.write_text(
+            "#!/usr/bin/env python3\n"
+            "import pathlib, sys\n"
+            "sys.stdout.buffer.write(pathlib.Path(sys.argv[1]).read_bytes())\n"
+        )
+        self.provided_symbols.write_text(
+            "#!/usr/bin/env python3\n"
+            "import pathlib, sys\n"
+            "symbols = set()\n"
+            "for name in sys.argv[1:]:\n"
+            "    symbols.update(pathlib.Path(name).read_bytes()[4:].decode().splitlines())\n"
+            "print(*sorted(symbols), sep='\\n')\n"
+        )
+        self.file_command.write_text(
+            "#!/usr/bin/env python3\n"
+            "import pathlib, sys\n"
+            "data = pathlib.Path(sys.argv[-1]).read_bytes()\n"
+            "kind = 'pie executable' if b'EXECUTABLE' in data else 'shared object'\n"
+            "print(f'ELF 64-bit LSB {kind}, x86-64, version 1 (SYSV)')\n"
+        )
+        self.rpm2cpio.chmod(0o755)
+        self.provided_symbols.chmod(0o755)
+        self.file_command.chmod(0o755)
+
+    def tearDown(self) -> None:
+        self.temporary_directory.cleanup()
+
+    def run_script(self, *arguments: str) -> subprocess.CompletedProcess[str]:
+        return subprocess.run(
+            [
+                sys.executable,
+                str(SCRIPT),
+                "--rpm2cpio",
+                str(self.rpm2cpio),
+                "--provided-symbols",
+                str(self.provided_symbols),
+                "--file-command",
+                str(self.file_command),
+                *arguments,
+            ],
+            capture_output=True,
+            text=True,
+            check=False,
+        )
+
+    def test_combines_unique_cpp_symbols_from_multiple_packages(self) -> None:
+        first = self.work / "first.rpm"
+        second = self.work / "second.rpm"
+        output = self.work / "symbols.txt"
+        first.write_bytes(
+            newc_archive(
+                [
+                    ("./usr/lib64/libfirst.so", b"\x7fELF_ZN3Foo3barEv\nplain_c_symbol\n", stat.S_IFREG | 0o755),
+                    ("./usr/share/doc/readme", b"not ELF", stat.S_IFREG | 0o644),
+                    ("./usr/lib64/libalias.so", b"libfirst.so", stat.S_IFLNK | 0o777),
+                ]
+            )
+        )
+        second.write_bytes(
+            newc_archive(
+                [
+                    (
+                        "./usr/lib64/libsecond.so",
+                        b"\x7fELF_ZN3Foo3barEv\n_ZN3Foo3bazEv\nanother_c_symbol\n",
+                        stat.S_IFREG | 0o755,
+                    ),
+                ]
+            )
+        )
+
+        result = self.run_script(
+            "--cpp-only", "--output", str(output), str(first), str(second)
+        )
+
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual(
+            output.read_text().splitlines(),
+            ["_ZN3Foo3barEv", "_ZN3Foo3bazEv"],
+        )
+        self.assertIn("package=first.rpm elf_files=1 symbols=2 selected=1", result.stderr)
+        self.assertIn("package=second.rpm elf_files=1 symbols=3 selected=2", result.stderr)
+        self.assertIn("unique_symbols=2", result.stderr)
+        self.assertEqual(result.stdout, "")
+
+    def test_without_cpp_filter_preserves_all_provided_symbols(self) -> None:
+        package = self.work / "all.rpm"
+        package.write_bytes(
+            newc_archive(
+                [
+                    (
+                        "./usr/lib64/liball.so",
+                        b"\x7fELF_ZN3Foo3barEv\nplain_c_symbol\n",
+                        stat.S_IFREG | 0o755,
+                    )
+                ]
+            )
+        )
+
+        result = self.run_script(str(package))
+
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual(result.stdout.splitlines(), ["_ZN3Foo3barEv", "plain_c_symbol"])
+
+    def test_excludes_elf_executables_that_do_not_generate_library_provides(self) -> None:
+        package = self.work / "mixed.rpm"
+        package.write_bytes(
+            newc_archive(
+                [
+                    (
+                        "./usr/lib64/libmixed.so",
+                        b"\x7fELF_ZN3Lib3runEv\n",
+                        stat.S_IFREG | 0o755,
+                    ),
+                    (
+                        "./usr/bin/mixed",
+                        b"\x7fELFEXECUTABLE\n_ZN4Exec3runEv\n",
+                        stat.S_IFREG | 0o755,
+                    ),
+                ]
+            )
+        )
+
+        result = self.run_script("--cpp-only", str(package))
+
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual(result.stdout.splitlines(), ["_ZN3Lib3runEv"])
+        self.assertIn("elf_files=1", result.stderr)
+
+    def test_rejects_malformed_cpio_stream(self) -> None:
+        package = self.work / "broken.rpm"
+        package.write_bytes(b"not a newc archive")
+
+        result = self.run_script(str(package))
+
+        self.assertNotEqual(result.returncode, 0)
+        self.assertIn("newc", result.stderr)
+
+
+if __name__ == "__main__":
+    unittest.main()

hash_testing/new_testing/tests/test_plot_probability_map.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""Behavior tests for plot_probability_map.py."""
+
+from __future__ import annotations
+
+import tempfile
+import unittest
+from pathlib import Path
+
+from PIL import Image
+
+import plot_probability_map
+
+
+class ProbabilityMapPlotTests(unittest.TestCase):
+    def test_reads_bit_columns_and_ignores_operation_and_pairs(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            path = Path(temporary_directory) / "hash.csv"
+            path.write_text(
+                "operation,pairs,bit_0,bit_1\n"
+                "replace,4,0.25,0.75\n"
+                "delete,2,0.5,nan\n",
+                encoding="utf-8",
+            )
+
+            rows = plot_probability_map.read_probability_map(path)
+
+            self.assertEqual(rows[0].operation, "replace")
+            self.assertEqual(rows[0].pairs, 4)
+            self.assertEqual(rows[0].probabilities, [0.25, 0.75])
+            self.assertEqual(rows[1].operation, "delete")
+            self.assertEqual(rows[1].pairs, 2)
+            self.assertEqual(len(rows[1].probabilities), 2)
+
+    def test_mean_absolute_deviation_from_half_ignores_nan(self) -> None:
+        result = plot_probability_map.mean_absolute_deviation(
+            [0.25, 0.5, 0.75, float("nan")]
+        )
+
+        self.assertEqual(result, 1 / 6)
+
+    def test_shared_scale_zooms_to_all_values_and_reference(self) -> None:
+        scale = plot_probability_map.make_scale(
+            [0.45, 0.48, 0.52, 0.55],
+            reference=0.5,
+            hard_limits=(0.0, 1.0),
+        )
+
+        self.assertGreater(scale.minimum, 0.0)
+        self.assertLess(scale.maximum, 1.0)
+        self.assertLessEqual(scale.minimum, 0.45)
+        self.assertGreaterEqual(scale.maximum, 0.55)
+        self.assertIn(0.5, scale.ticks)
+
+    def test_deviation_scale_uses_data_range_instead_of_fixed_half(self) -> None:
+        scale = plot_probability_map.make_scale(
+            [0.05, 0.08], hard_limits=(0.0, 0.5)
+        )
+
+        self.assertGreater(scale.minimum, 0.0)
+        self.assertLess(scale.maximum, 0.5)
+        self.assertLessEqual(scale.minimum, 0.05)
+        self.assertGreaterEqual(scale.maximum, 0.08)
+
+    def test_generates_two_nonempty_png_files(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            root = Path(temporary_directory)
+            source = root / "sample.csv"
+            source.write_text(
+                "operation,pairs,bit_0,bit_1,bit_2,bit_3\n"
+                "replace,4,0.25,0.50,0.75,1.0\n"
+                "delete,2,0.10,0.20,0.30,0.40\n",
+                encoding="utf-8",
+            )
+
+            outputs = plot_probability_map.generate_plots(source, root / "plots")
+
+            self.assertEqual(len(outputs), 2)
+            for output in outputs:
+                self.assertTrue(output.is_file())
+                with Image.open(output) as image:
+                    self.assertEqual(image.format, "PNG")
+                    self.assertGreater(image.width, 300)
+                    self.assertGreater(image.height, 200)
+                    colors = image.convert("RGB").getcolors(maxcolors=1_000_000)
+                    self.assertIsNotNone(colors)
+                    self.assertGreater(len(colors or []), 2)
+
+
+if __name__ == "__main__":
+    unittest.main()

hash_testing/new_testing/tests/test_probability_map.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""Behavior tests for probability_map.py."""
+
+from __future__ import annotations
+
+import csv
+import io
+import os
+import subprocess
+import tempfile
+import unittest
+from contextlib import redirect_stderr
+from pathlib import Path
+
+import generate_input
+import probability_map
+
+
+class PrepareHashTests(unittest.TestCase):
+    def test_compiles_c_source_when_binary_is_missing(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            root = Path(temporary_directory)
+            hash_directory = root / "constant"
+            hash_directory.mkdir()
+            (hash_directory / "bin_hash.c").write_text(
+                "#include <stdio.h>\n"
+                "int main(void) { puts(\"00000001\"); return 0; }\n",
+                encoding="utf-8",
+            )
+
+            executable = probability_map.prepare_hash("constant", root)
+
+            self.assertEqual(executable, hash_directory / "bin_hash")
+            self.assertEqual(
+                subprocess.check_output([executable, "word"], text=True).strip(),
+                "00000001",
+            )
+
+    def test_adds_python_shebang_and_execute_permission(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            root = Path(temporary_directory)
+            hash_directory = root / "python_hash"
+            hash_directory.mkdir()
+            source = hash_directory / "bin_hash.py"
+            source.write_text("print('00000002')\n", encoding="utf-8")
+
+            executable = probability_map.prepare_hash("python_hash", root)
+
+            self.assertEqual(executable, source)
+            self.assertTrue(os.access(source, os.X_OK))
+            self.assertTrue(
+                source.read_text(encoding="utf-8").startswith(
+                    "#!/usr/bin/env python3\n"
+                )
+            )
+            self.assertEqual(
+                subprocess.check_output([executable, "word"], text=True).strip(),
+                "00000002",
+            )
+
+
+class ProbabilityTests(unittest.TestCase):
+    def test_counts_changed_hash_bits_relative_to_source(self) -> None:
+        probabilities = probability_map.bit_probabilities(
+            source_hash=0b0000,
+            changed_hashes=[0b0001, 0b0011, 0b0010, 0b0000],
+            bits=4,
+        )
+
+        self.assertEqual(probabilities, [0.5, 0.5, 0.0, 0.0])
+
+    def test_csv_table_has_operation_rows_and_bit_columns(self) -> None:
+        stream = io.StringIO()
+
+        probability_map.write_csv_table(
+            stream,
+            {
+                "replace": (4, [0.25, 0.75]),
+                "delete": (2, [0.5, 0.0]),
+            },
+        )
+
+        rows = list(csv.reader(io.StringIO(stream.getvalue())))
+        self.assertEqual(rows[0], ["operation", "pairs", "bit_0", "bit_1"])
+        self.assertEqual(rows[1], ["replace", "4", "0.250000", "0.750000"])
+        self.assertEqual(rows[2], ["delete", "2", "0.500000", "0.000000"])
+
+    def test_parser_accepts_multiple_source_words(self) -> None:
+        arguments = probability_map.build_parser().parse_args(["first", "second"])
+
+        self.assertEqual(arguments.words, ["first", "second"])
+
+    def test_aggregates_samples_from_multiple_words_and_warns_on_shortfall(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            executable = Path(temporary_directory) / "hash.py"
+            executable.write_text(
+                "#!/usr/bin/env python3\n"
+                "import sys\n"
+                "print(f'{sum(sys.argv[1].encode()):08x}')\n",
+                encoding="utf-8",
+            )
+            executable.chmod(0o755)
+            warnings = io.StringIO()
+
+            with redirect_stderr(warnings):
+                table = probability_map.build_probability_table(
+                    executable=executable,
+                    sources=["A", "B"],
+                    operations=["delete"],
+                    count=10,
+                    operation_count=1,
+                    alphabet=generate_input.DEFAULT_ALPHABET,
+                    seed=1,
+                    max_attempts=None,
+                )
+
+            pair_count, probabilities = table["delete"]
+            self.assertEqual(pair_count, 2)
+            self.assertEqual(
+                probabilities[:8],
+                [0.5, 0.5, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
+            )
+            self.assertEqual(warnings.getvalue().count("delete"), 2)
+
+
+class GenerateWordsTests(unittest.TestCase):
+    def test_count_is_an_upper_bound_when_unique_results_are_exhausted(self) -> None:
+        words = generate_input.generate_words(
+            source="abc",
+            count=100,
+            operation="delete",
+            operation_count=1,
+            seed=42,
+        )
+
+        self.assertEqual(set(words), {"ab", "ac", "bc"})
+
+
+if __name__ == "__main__":
+    unittest.main()

hash_testing/new_testing/tests/test_t1ha2_bin_hash.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+"""Behavior tests for the standalone t1ha2_atonce hash CLI."""
+
+from __future__ import annotations
+
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+SOURCE = ROOT / "hash_funcs" / "t1ha2" / "bin_hash.c"
+
+
+class T1ha2BinHashTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls) -> None:
+        cls.temporary_directory = tempfile.TemporaryDirectory()
+        cls.binary = Path(cls.temporary_directory.name) / "bin_hash"
+        subprocess.run(
+            [
+                "cc",
+                "-std=c11",
+                "-O2",
+                "-Wall",
+                "-Wextra",
+                "-Wpedantic",
+                "-Werror",
+                str(SOURCE),
+                "-o",
+                str(cls.binary),
+            ],
+            check=True,
+            text=True,
+            capture_output=True,
+        )
+
+    @classmethod
+    def tearDownClass(cls) -> None:
+        cls.temporary_directory.cleanup()
+
+    def run_hash(
+        self, word: str | None = None, stdin: bytes | None = None
+    ) -> subprocess.CompletedProcess[bytes]:
+        command = [str(self.binary)]
+        if word is not None:
+            command.append(word)
+        return subprocess.run(command, input=stdin, capture_output=True, check=False)
+
+    def test_matches_upstream_t1ha2_atonce_seed_zero_vectors(self) -> None:
+        vectors = {
+            "": b"0000000000000000\n",
+            "hello": b"2a5f2abd74df73b4\n",
+            "HashWord": b"3885d16135ce64f0\n",
+            "abc": b"16bae0f716c45f2e\n",
+            "12345678901234567890123456789012": b"75ed8a8aa66a4602\n",
+        }
+
+        for word, expected in vectors.items():
+            with self.subTest(word=word):
+                result = self.run_hash(word)
+                self.assertEqual(result.returncode, 0)
+                self.assertEqual(result.stdout, expected)
+                self.assertRegex(result.stdout.decode(), r"^[0-9a-f]{16}\n$")
+
+    def test_argv_and_stdin_are_equivalent_and_strip_trailing_ascii_space(self) -> None:
+        argv = self.run_hash("hello")
+        stdin = self.run_hash(stdin=b"hello \t\r\n")
+
+        self.assertEqual(stdin.returncode, 0)
+        self.assertEqual(stdin.stdout, argv.stdout)
+
+    def test_handles_long_ascii_input(self) -> None:
+        payload = b"a" * 100_000
+        result = self.run_hash(stdin=payload)
+
+        self.assertEqual(result.returncode, 0)
+        self.assertEqual(result.stdout, b"bdf3f8539f0504ea\n")
+
+    def test_rejects_non_ascii_input(self) -> None:
+        result = self.run_hash(stdin="ёж".encode())
+
+        self.assertNotEqual(result.returncode, 0)
+        self.assertIn(b"ASCII", result.stderr)
+        self.assertEqual(result.stdout, b"")
+
+    def test_unaligned_argv_is_clean_under_undefined_behavior_sanitizer(self) -> None:
+        sanitized = Path(self.temporary_directory.name) / "bin_hash_ubsan"
+        subprocess.run(
+            [
+                "cc",
+                "-std=c11",
+                "-O1",
+                "-g",
+                "-fsanitize=undefined",
+                "-fno-sanitize-recover=undefined",
+                str(SOURCE),
+                "-o",
+                str(sanitized),
+            ],
+            check=True,
+            capture_output=True,
+        )
+
+        for word in ("hello", "12345678", "a" * 33):
+            with self.subTest(word=word):
+                result = subprocess.run(
+                    [str(sanitized), word], capture_output=True, check=False
+                )
+                self.assertEqual(result.returncode, 0, result.stderr.decode())
+
+
+if __name__ == "__main__":
+    unittest.main()

hash_testing/new_testing/tests/test_xxh64_bin_hash.py

@@ -0,0 +1,90 @@
+#!/usr/bin/env python3
+"""Behavior tests for the standalone XXH64 hash CLI."""
+
+from __future__ import annotations
+
+import re
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+SOURCE = ROOT / "hash_funcs" / "xxh64" / "bin_hash.c"
+
+
+class Xxh64BinHashTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls) -> None:
+        cls.temporary_directory = tempfile.TemporaryDirectory()
+        cls.binary = Path(cls.temporary_directory.name) / "bin_hash"
+        subprocess.run(
+            [
+                "cc",
+                "-std=c11",
+                "-O2",
+                "-Wall",
+                "-Wextra",
+                "-Wpedantic",
+                "-Werror",
+                str(SOURCE),
+                "-o",
+                str(cls.binary),
+            ],
+            check=True,
+            text=True,
+            capture_output=True,
+        )
+
+    @classmethod
+    def tearDownClass(cls) -> None:
+        cls.temporary_directory.cleanup()
+
+    def run_hash(self, word: str | None = None, stdin: bytes | None = None) -> subprocess.CompletedProcess[bytes]:
+        command = [str(self.binary)]
+        if word is not None:
+            command.append(word)
+        return subprocess.run(command, input=stdin, capture_output=True, check=False)
+
+    def test_matches_official_xxh64_seed_zero_vectors(self) -> None:
+        vectors = {
+            "": b"ef46db3751d8e999\n",
+            "hello": b"26c7827d889f6da3\n",
+            "HashWord": b"3e26fc2935163fbe\n",
+        }
+
+        for word, expected in vectors.items():
+            with self.subTest(word=word):
+                result = self.run_hash(word)
+                self.assertEqual(result.returncode, 0)
+                self.assertEqual(result.stdout, expected)
+                self.assertRegex(result.stdout.decode(), r"^[0-9a-f]{16}\n$")
+
+    def test_argv_and_stdin_are_equivalent_and_strip_trailing_ascii_space(self) -> None:
+        argv = self.run_hash("hello")
+        stdin = self.run_hash(stdin=b"hello \t\r\n")
+
+        self.assertEqual(stdin.returncode, 0)
+        self.assertEqual(stdin.stdout, argv.stdout)
+
+    def test_handles_long_ascii_input(self) -> None:
+        payload = b"a" * 100_000
+        result = self.run_hash(stdin=payload)
+        reference = subprocess.run(
+            ["xxhsum", "-H64"], input=payload, capture_output=True, check=True
+        ).stdout.split()[0]
+
+        self.assertEqual(result.returncode, 0)
+        self.assertEqual(result.stdout.strip(), reference)
+
+    def test_rejects_non_ascii_input(self) -> None:
+        result = self.run_hash(stdin="ёж".encode())
+
+        self.assertNotEqual(result.returncode, 0)
+        self.assertIn(b"ASCII", result.stderr)
+        self.assertEqual(result.stdout, b"")
+
+
+if __name__ == "__main__":
+    unittest.main()

new_version/direct_hash/about.md

@@ -0,0 +1,31 @@
+В этой реализации хэши хранятся напрямую, без Golomb-Rice-кодирования.
+
+Set-строка целиком декодируется и проверяется при первом обращении. Как и в
+`set9.c`, полностью декодированные массивы хэшей сохраняются в двух LRU-кэшах
+по 512 записей — отдельно для первого и второго операнда.
+
+Для 32-битного формата Base64
+декодируется сразу в три `unsigned` за блок с одновременной проверкой порядка.
+Большое понижение BPP выполняется radix-сортировкой, а не отдельным проходом на
+каждый бит.
+
+Последний запуск `taskset -c 2 python3 benchmark.py`:
+
+```text
+symbols=1000 required=500 bpp=32
+implementation  set_chars  format
+set9                3994  golomb/base62
+direct              5340  D1/base64
+
+operation                 set9       direct   direct/set9
+set_fini only           147.45 us  105.09 us         0.71x
+new+add (ctypes)        611.28 us  608.24 us         1.00x
+new+add+fini (ctypes)   769.66 us  693.60 us         0.90x
+rpmsetcmp cold          129.67 us  113.04 us         0.87x
+rpmsetcmp warm            2.99 us    1.04 us         0.35x
+```
+
+Для разреженного сравнения
+(`taskset -c 2 python3 benchmark.py --required 1`) получено
+`83.22 us` на холодном кэше и `0.86 us` на прогретом: соответственно `0.88x`
+и `1.02x` от времени `set9`.

new_version/direct_hash/apt_benchmark/rewrite_sisyphus_pkglist.c

@@ -0,0 +1,296 @@
+#include <errno.h>
+#include <limits.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#ifdef ARSV_WITH_RPM
+#include <rpm/header.h>
+#include <rpm/rpmio.h>
+#include <rpm/rpmlib.h>
+#include <rpm/rpmtag.h>
+#include <rpm/rpmtd.h>
+#endif
+
+int arsv_set9_decode(const char* source, unsigned** hashes, size_t* count, unsigned* bpp);
+
+#define D1_PREFIX "set:D1"
+#define D1_HEADER_LEN 8
+
+static const char base64_alphabet[] =
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+
+static size_t base64_encoded_size(size_t byte_count) {
+  if (byte_count > SIZE_MAX - 2) return SIZE_MAX;
+  size_t groups = (byte_count + 2) / 3;
+  if (groups > SIZE_MAX / 4) return SIZE_MAX;
+  size_t size = groups * 4;
+  size_t remainder = byte_count % 3;
+  if (remainder) size -= 3 - remainder;
+  return size;
+}
+
+static void base64_encode(const unsigned char* input, size_t input_len, char* output) {
+  while (input_len >= 3) {
+    uint32_t value = ((uint32_t)input[0] << 16) | ((uint32_t)input[1] << 8) | input[2];
+    output[0] = base64_alphabet[(value >> 18) & 0x3f];
+    output[1] = base64_alphabet[(value >> 12) & 0x3f];
+    output[2] = base64_alphabet[(value >> 6) & 0x3f];
+    output[3] = base64_alphabet[value & 0x3f];
+    input += 3;
+    input_len -= 3;
+    output += 4;
+  }
+  if (input_len == 1) {
+    uint32_t value = (uint32_t)input[0] << 16;
+    output[0] = base64_alphabet[(value >> 18) & 0x3f];
+    output[1] = base64_alphabet[(value >> 12) & 0x3f];
+  } else if (input_len == 2) {
+    uint32_t value = ((uint32_t)input[0] << 16) | ((uint32_t)input[1] << 8);
+    output[0] = base64_alphabet[(value >> 18) & 0x3f];
+    output[1] = base64_alphabet[(value >> 12) & 0x3f];
+    output[2] = base64_alphabet[(value >> 6) & 0x3f];
+  }
+}
+
+static int encode_d1(const unsigned* hashes, size_t count, unsigned bpp, char** result) {
+  if (!hashes || !count || !result || bpp < 10 || bpp > 32) return -EINVAL;
+  if (count > (SIZE_MAX - 7) / bpp) return -EOVERFLOW;
+
+  size_t bit_count = count * bpp;
+  size_t byte_count = (bit_count + 7) / 8;
+  unsigned char* bytes = calloc(byte_count, 1);
+  if (!bytes) return -ENOMEM;
+
+  unsigned char* output = bytes;
+  uint64_t bits = 0;
+  unsigned filled = 0;
+  for (size_t i = 0; i < count; ++i) {
+    bits |= (uint64_t)hashes[i] << filled;
+    filled += bpp;
+    while (filled >= 8) {
+      *output++ = (unsigned char)bits;
+      bits >>= 8;
+      filled -= 8;
+    }
+  }
+  if (filled) *output++ = (unsigned char)bits;
+  if ((size_t)(output - bytes) != byte_count) {
+    free(bytes);
+    return -EIO;
+  }
+
+  size_t payload_len = base64_encoded_size(byte_count);
+  if (payload_len == SIZE_MAX || payload_len > SIZE_MAX - D1_HEADER_LEN - 1) {
+    free(bytes);
+    return -EOVERFLOW;
+  }
+  char* encoded = malloc(D1_HEADER_LEN + payload_len + 1);
+  if (!encoded) {
+    free(bytes);
+    return -ENOMEM;
+  }
+  memcpy(encoded, D1_PREFIX, sizeof(D1_PREFIX) - 1);
+  encoded[6] = (char)('0' + bpp / 10);
+  encoded[7] = (char)('0' + bpp % 10);
+  base64_encode(bytes, byte_count, encoded + D1_HEADER_LEN);
+  encoded[D1_HEADER_LEN + payload_len] = '\0';
+  free(bytes);
+  *result = encoded;
+  return 0;
+}
+
+static int convert_set(const char* source, char** result, size_t* hash_count, unsigned* bpp) {
+  if (!source || strncmp(source, "set:", 4) != 0 || strncmp(source, D1_PREFIX, 6) == 0)
+    return -EINVAL;
+
+  unsigned* hashes = NULL;
+  size_t count = 0;
+  unsigned precision = 0;
+  int rc = arsv_set9_decode(source, &hashes, &count, &precision);
+  if (rc == 0) rc = encode_d1(hashes, count, precision, result);
+  free(hashes);
+  if (rc < 0) return rc;
+  if (hash_count) *hash_count = count;
+  if (bpp) *bpp = precision;
+  return 0;
+}
+
+#ifdef ARSV_WITH_RPM
+struct statistics {
+  unsigned long headers;
+  unsigned long set_occurrences;
+  unsigned long set_bytes_old;
+  unsigned long set_bytes_new;
+  unsigned long hashes;
+};
+
+static int rewrite_version_tag(Header header, rpmTagVal tag, const char* package,
+                               struct statistics* stats) {
+  struct rpmtd_s values;
+  memset(&values, 0, sizeof(values));
+  if (headerGet(header, tag, &values, HEADERGET_MINMEM) != 1) return 0;
+  if (rpmtdType(&values) != RPM_STRING_ARRAY_TYPE) {
+    rpmtdFreeData(&values);
+    fprintf(stderr, "%s: tag %d is not a string array\n", package, (int)tag);
+    return -EINVAL;
+  }
+
+  rpm_count_t count = rpmtdCount(&values);
+  const char** rewritten = calloc((size_t)count, sizeof(*rewritten));
+  if (!rewritten) {
+    rpmtdFreeData(&values);
+    return -ENOMEM;
+  }
+
+  int rc = 0;
+  int changed = 0;
+  rpmtdInit(&values);
+  for (rpm_count_t i = 0; i < count; ++i) {
+    const char* value = rpmtdNextString(&values);
+    if (!value) {
+      rc = -EINVAL;
+      break;
+    }
+    if (strncmp(value, "set:", 4) == 0) {
+      char* converted = NULL;
+      size_t hashes = 0;
+      unsigned bpp = 0;
+      rc = convert_set(value, &converted, &hashes, &bpp);
+      if (rc < 0) {
+        fprintf(stderr, "%s: cannot convert tag %d index %u (rc=%d)\n", package,
+                (int)tag, (unsigned)i, rc);
+        break;
+      }
+      rewritten[i] = converted;
+      changed = 1;
+      ++stats->set_occurrences;
+      stats->set_bytes_old += strlen(value);
+      stats->set_bytes_new += strlen(converted);
+      stats->hashes += hashes;
+    } else {
+      rewritten[i] = strdup(value);
+      if (!rewritten[i]) {
+        rc = -ENOMEM;
+        break;
+      }
+    }
+  }
+
+  if (rc == 0 && changed) {
+    headerDel(header, tag);
+    if (!headerPutStringArray(header, tag, rewritten, count)) rc = -EIO;
+  }
+  for (rpm_count_t i = 0; i < count; ++i) free((void*)rewritten[i]);
+  free(rewritten);
+  rpmtdFreeData(&values);
+  return rc;
+}
+
+static int rewrite_pkglist(const char* input_path, const char* output_path, unsigned long limit) {
+  FD_t input = Fopen(input_path, "r.ufdio");
+  if (!input || Ferror(input)) {
+    fprintf(stderr, "%s: %s\n", input_path, input ? Fstrerror(input) : "cannot open");
+    return 1;
+  }
+  FD_t output = Fopen(output_path, "w.ufdio");
+  if (!output || Ferror(output)) {
+    fprintf(stderr, "%s: %s\n", output_path, output ? Fstrerror(output) : "cannot open");
+    Fclose(input);
+    return 1;
+  }
+
+  struct statistics stats = {0};
+  Header header;
+  int failed = 0;
+  const rpmTagVal tags[] = {
+      RPMTAG_REQUIREVERSION,
+      RPMTAG_PROVIDEVERSION,
+      RPMTAG_CONFLICTVERSION,
+      RPMTAG_OBSOLETEVERSION,
+      RPMTAG_RECOMMENDVERSION,
+      RPMTAG_SUGGESTVERSION,
+      RPMTAG_SUPPLEMENTVERSION,
+      RPMTAG_ENHANCEVERSION,
+  };
+  while ((!limit || stats.headers < limit) &&
+         (header = headerRead(input, HEADER_MAGIC_YES)) != NULL) {
+    const char* package = headerGetString(header, RPMTAG_NAME);
+    if (!package) package = "<unknown>";
+    for (size_t i = 0; i < sizeof(tags) / sizeof(tags[0]); ++i) {
+      int rc = rewrite_version_tag(header, tags[i], package, &stats);
+      if (rc < 0) {
+        failed = 1;
+        break;
+      }
+    }
+    if (!failed && headerWrite(output, header, HEADER_MAGIC_YES) != 0) {
+      fprintf(stderr, "%s: failed to write header for %s\n", output_path, package);
+      failed = 1;
+    }
+    headerFree(header);
+    if (failed) break;
+    ++stats.headers;
+  }
+  if (!failed && Ferror(input)) {
+    fprintf(stderr, "%s: read error: %s\n", input_path, Fstrerror(input));
+    failed = 1;
+  }
+  if (Fclose(output) != 0) failed = 1;
+  Fclose(input);
+
+  if (failed) {
+    remove(output_path);
+    return 1;
+  }
+  fprintf(stderr,
+          "headers=%lu set_occurrences=%lu hashes=%lu old_set_bytes=%lu "
+          "new_set_bytes=%lu\n",
+          stats.headers, stats.set_occurrences, stats.hashes, stats.set_bytes_old,
+          stats.set_bytes_new);
+  return 0;
+}
+#endif
+
+static void usage(const char* program) {
+  fprintf(stderr, "usage: %s --convert-set set:VALUE\n", program);
+#ifdef ARSV_WITH_RPM
+  fprintf(stderr, "       %s --rewrite INPUT OUTPUT [--max-headers N]\n", program);
+#endif
+}
+
+int main(int argc, char** argv) {
+  if (argc == 3 && strcmp(argv[1], "--convert-set") == 0) {
+    char* converted = NULL;
+    int rc = convert_set(argv[2], &converted, NULL, NULL);
+    if (rc < 0) {
+      fprintf(stderr, "cannot convert set value (rc=%d)\n", rc);
+      return 1;
+    }
+    puts(converted);
+    free(converted);
+    return 0;
+  }
+#ifdef ARSV_WITH_RPM
+  if ((argc == 4 || argc == 6) && strcmp(argv[1], "--rewrite") == 0) {
+    unsigned long limit = 0;
+    if (argc == 6) {
+      if (strcmp(argv[4], "--max-headers") != 0) {
+        usage(argv[0]);
+        return 2;
+      }
+      char* end = NULL;
+      errno = 0;
+      limit = strtoul(argv[5], &end, 10);
+      if (errno || !end || *end || !limit) {
+        fprintf(stderr, "invalid --max-headers value: %s\n", argv[5]);
+        return 2;
+      }
+    }
+    return rewrite_pkglist(argv[2], argv[3], limit);
+  }
+#endif
+  usage(argv[0]);
+  return 2;
+}

new_version/direct_hash/apt_benchmark/run_sisyphus_pkglist.py

@@ -0,0 +1,335 @@
+#!/usr/bin/env python3
+"""Create D1 copies of the real Sisyphus x86_64 and noarch pkglist files."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+from datetime import datetime, timezone
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+ROOT = HERE.parents[2]
+ARCHITECTURES = ("x86_64", "noarch")
+SUMMARY_RE = re.compile(
+    r"headers=(?P<headers>\d+) set_occurrences=(?P<set_occurrences>\d+) "
+    r"hashes=(?P<hashes>\d+) old_set_bytes=(?P<old_set_bytes>\d+) "
+    r"new_set_bytes=(?P<new_set_bytes>\d+)"
+)
+VERSION_FORMAT = (
+    "[%{REQUIREVERSION}\\n]"
+    "[%{PROVIDEVERSION}\\n]"
+    "[%{CONFLICTVERSION}\\n]"
+    "[%{OBSOLETEVERSION}\\n]"
+    "[%{RECOMMENDVERSION}\\n]"
+    "[%{SUGGESTVERSION}\\n]"
+    "[%{SUPPLEMENTVERSION}\\n]"
+    "[%{ENHANCEVERSION}\\n]"
+)
+
+
+def classify_pkglist(path: Path) -> str | None:
+    name = path.name
+    if not name.endswith("_base_pkglist.classic"):
+        return None
+    if "_Sisyphus_x86%5f64_" in name or "_Sisyphus_x86_64_" in name:
+        return "x86_64"
+    if "_Sisyphus_noarch_" in name:
+        return "noarch"
+    return None
+
+
+def validate_output(path: Path) -> Path:
+    resolved = path.expanduser().resolve()
+    home = Path.home().resolve()
+    source = ROOT.resolve()
+    if resolved == home or resolved == source or source in resolved.parents:
+        raise ValueError(f"refusing protected output path: {resolved}")
+    if resolved == Path("/"):
+        raise ValueError("refusing filesystem root as output")
+    return resolved
+
+
+def run(
+    command: list[str],
+    *,
+    check: bool = True,
+    cwd: Path | None = None,
+    stdout: int | None = None,
+) -> subprocess.CompletedProcess[bytes]:
+    result = subprocess.run(
+        command,
+        cwd=cwd,
+        stdout=stdout if stdout is not None else subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        check=False,
+    )
+    if check and result.returncode != 0:
+        stderr = result.stderr.decode(errors="replace")
+        raise RuntimeError(f"command failed ({result.returncode}): {' '.join(command)}\n{stderr}")
+    return result
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as stream:
+        for block in iter(lambda: stream.read(1024 * 1024), b""):
+            digest.update(block)
+    return digest.hexdigest()
+
+
+def compile_converter(
+    destination: Path,
+    rpm_include_dir: Path | None = None,
+    rpm_libraries: tuple[Path, ...] = (),
+) -> None:
+    include = destination.parent / "compat-include"
+    include.mkdir()
+    for name in ("rpmlib.h", "system.h", "set.h"):
+        (include / name).touch()
+    command = [
+            "cc",
+            "-O2",
+            "-std=gnu11",
+            "-Wall",
+            "-Wextra",
+            "-Werror",
+            "-D_GNU_SOURCE",
+            "-DARSV_SET9_EXPORT",
+            "-DARSV_WITH_RPM",
+            "-I",
+            str(include),
+    ]
+    if rpm_include_dir is not None:
+        command.extend(["-I", str(rpm_include_dir)])
+    command.extend(
+        [
+            "-include",
+            str(ROOT / "scripts/rpmsetcmp/newset_compat.h"),
+            str(ROOT / "reimplement/set9.c"),
+            str(HERE / "rewrite_sisyphus_pkglist.c"),
+        ]
+    )
+    if rpm_libraries:
+        command.extend(str(path) for path in rpm_libraries)
+    else:
+        command.extend(["-lrpm", "-lrpmio"])
+    command.extend(["-o", str(destination)])
+    run(command)
+
+
+def find_pkglists(lists_dir: Path) -> dict[str, Path]:
+    found: dict[str, Path] = {}
+    for path in lists_dir.glob("*_base_pkglist.classic"):
+        architecture = classify_pkglist(path)
+        if architecture is None:
+            continue
+        if architecture in found:
+            raise RuntimeError(f"multiple Sisyphus {architecture} pkglist files")
+        found[architecture] = path
+    missing = set(ARCHITECTURES) - set(found)
+    if missing:
+        raise RuntimeError(f"missing Sisyphus pkglist for: {', '.join(sorted(missing))}")
+    return found
+
+
+def query_header_count(path: Path) -> int:
+    process = subprocess.Popen(
+        ["pkglist-query", "%{NAME}\\n", str(path)],
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    )
+    assert process.stdout is not None
+    count = sum(1 for _ in process.stdout)
+    stderr = process.stderr.read() if process.stderr else b""
+    status = process.wait()
+    if status != 0:
+        raise RuntimeError(f"pkglist-query failed for {path}: {stderr.decode(errors='replace')}")
+    return count
+
+
+def query_set_counts(path: Path) -> dict[str, int]:
+    process = subprocess.Popen(
+        ["pkglist-query", VERSION_FORMAT, str(path)],
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    )
+    assert process.stdout is not None
+    old = direct = 0
+    for raw_line in process.stdout:
+        value = raw_line.strip()
+        if value.startswith(b"set:D1"):
+            direct += 1
+        elif value.startswith(b"set:"):
+            old += 1
+    stderr = process.stderr.read() if process.stderr else b""
+    status = process.wait()
+    if status != 0:
+        raise RuntimeError(f"pkglist-query failed for {path}: {stderr.decode(errors='replace')}")
+    return {"set9": old, "d1": direct}
+
+
+def conversion_summary(stderr: bytes) -> dict[str, int]:
+    text = stderr.decode(errors="replace")
+    match = SUMMARY_RE.search(text)
+    if not match:
+        raise RuntimeError(f"converter did not report summary:\n{text}")
+    return {name: int(value) for name, value in match.groupdict().items()}
+
+
+def convert_one(converter: Path, source: Path, destination: Path) -> dict[str, object]:
+    result = run([str(converter), "--rewrite", str(source), str(destination)])
+    summary = conversion_summary(result.stderr)
+    source_headers = query_header_count(source)
+    output_headers = query_header_count(destination)
+    source_sets = query_set_counts(source)
+    output_sets = query_set_counts(destination)
+    if source_headers != output_headers or output_headers != summary["headers"]:
+        raise RuntimeError(
+            f"header count mismatch: source={source_headers} output={output_headers} "
+            f"converter={summary['headers']}"
+        )
+    if source_sets["d1"] != 0:
+        raise RuntimeError(f"source already contains {source_sets['d1']} D1 values")
+    if output_sets["set9"] != 0:
+        raise RuntimeError(f"output still contains {output_sets['set9']} set9 values")
+    if source_sets["set9"] != output_sets["d1"] or output_sets["d1"] != summary["set_occurrences"]:
+        raise RuntimeError(
+            "set occurrence mismatch: "
+            f"source={source_sets['set9']} output={output_sets['d1']} "
+            f"converter={summary['set_occurrences']}"
+        )
+    if summary["set_occurrences"] == 0:
+        raise RuntimeError("source pkglist contains no set versions")
+    return {
+        **summary,
+        "source": {
+            "path": source.name,
+            "size": source.stat().st_size,
+            "sha256": sha256_file(source),
+        },
+        "output": {
+            "path": destination.name,
+            "size": destination.stat().st_size,
+            "sha256": sha256_file(destination),
+        },
+    }
+
+
+def convert_local(
+    output: Path,
+    lists_dir: Path,
+    rpm_include_dir: Path | None = None,
+    rpm_libraries: tuple[Path, ...] = (),
+) -> None:
+    output.mkdir(parents=True, exist_ok=True)
+    if any(output.iterdir()):
+        raise RuntimeError(f"output directory is not empty: {output}")
+
+    lists_dir = lists_dir.expanduser().resolve()
+    if not lists_dir.is_dir():
+        raise RuntimeError(f"APT lists directory does not exist: {lists_dir}")
+    pkglists = find_pkglists(lists_dir)
+    staging = Path(tempfile.mkdtemp(prefix=".d1-staging-", dir=output))
+    try:
+        converter = staging / "rewrite-sisyphus-pkglist"
+        compile_converter(converter, rpm_include_dir, rpm_libraries)
+        architectures: dict[str, object] = {}
+        for architecture in ARCHITECTURES:
+            destination = staging / f"Sisyphus.{architecture}.pkglist.classic"
+            architectures[architecture] = convert_one(
+                converter, pkglists[architecture], destination
+            )
+        converter.unlink()
+        shutil.rmtree(staging / "compat-include")
+        manifest = {
+            "format": 1,
+            "created_at": datetime.now(timezone.utc).isoformat(),
+            "input": {"lists_dir": str(lists_dir)},
+            "rpm": run(["rpm", "--version"]).stdout.decode().strip(),
+            "apt": run(["rpmquery", "--qf", "%{VERSION}-%{RELEASE}", "apt"])
+            .stdout.decode()
+            .strip(),
+            "converter_source": {
+                "set9_sha256": sha256_file(ROOT / "reimplement/set9.c"),
+                "rewrite_sha256": sha256_file(HERE / "rewrite_sisyphus_pkglist.c"),
+            },
+            "architectures": architectures,
+        }
+        (staging / "manifest.json").write_text(
+            json.dumps(manifest, indent=2, sort_keys=True) + "\n"
+        )
+        final = output / "d1-pkglists"
+        staging.replace(final)
+        print(final)
+        for architecture in ARCHITECTURES:
+            data = architectures[architecture]
+            assert isinstance(data, dict)
+            print(
+                f"{architecture}: headers={data['headers']} "
+                f"set_occurrences={data['set_occurrences']} "
+                f"output={final / f'Sisyphus.{architecture}.pkglist.classic'}"
+            )
+    except BaseException:
+        shutil.rmtree(staging, ignore_errors=True)
+        raise
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Create D1 copies of local Sisyphus x86_64/noarch pkglist files"
+    )
+    parser.add_argument("output", type=Path, help="new or empty output directory")
+    parser.add_argument(
+        "--lists-dir",
+        type=Path,
+        default=Path("/var/lib/apt/lists"),
+        help="APT lists snapshot to convert (default: /var/lib/apt/lists)",
+    )
+    parser.add_argument(
+        "--rpm-include-dir",
+        type=Path,
+        help="directory containing rpm/header.h (for an unpacked librpm-devel)",
+    )
+    parser.add_argument(
+        "--rpm-library",
+        type=Path,
+        action="append",
+        default=[],
+        help="versioned RPM library to link; repeat for librpm and librpmio",
+    )
+    return parser.parse_args()
+
+
+def main() -> int:
+    args = parse_args()
+    try:
+        output = validate_output(args.output)
+        rpm_include_dir = (
+            args.rpm_include_dir.expanduser().resolve()
+            if args.rpm_include_dir is not None
+            else None
+        )
+        rpm_libraries = tuple(path.expanduser().resolve() for path in args.rpm_library)
+        if rpm_include_dir is not None and not (rpm_include_dir / "rpm/header.h").is_file():
+            raise ValueError(f"rpm/header.h not found below: {rpm_include_dir}")
+        if rpm_libraries and len(rpm_libraries) != 2:
+            raise ValueError("pass exactly two --rpm-library values: librpm and librpmio")
+        for library in rpm_libraries:
+            if not library.is_file():
+                raise ValueError(f"RPM library not found: {library}")
+        convert_local(output, args.lists_dir, rpm_include_dir, rpm_libraries)
+    except (OSError, RuntimeError, ValueError) as error:
+        print(f"error: {error}", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

new_version/direct_hash/benchmark.py

@@ -0,0 +1,271 @@
+#!/usr/bin/env python3
+import argparse
+import ctypes
+import gc
+import os
+import statistics
+import subprocess
+import time
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+BUILD = HERE / "build"
+LIBC = ctypes.CDLL(None)
+LIBC.free.argtypes = [ctypes.c_void_p]
+
+
+class SetAPI:
+    def __init__(self, path: Path):
+        self.lib = ctypes.CDLL(str(path))
+        self.lib.set_new.restype = ctypes.c_void_p
+        self.lib.set_add.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
+        self.lib.set_fini.argtypes = [ctypes.c_void_p, ctypes.c_int]
+        self.lib.set_fini.restype = ctypes.c_void_p
+        self.lib.set_free.argtypes = [ctypes.c_void_p]
+        self.lib.set_free.restype = ctypes.c_void_p
+        self.lib.rpmsetcmp.argtypes = [ctypes.c_char_p, ctypes.c_char_p]
+        self.lib.rpmsetcmp.restype = ctypes.c_int
+
+    def new_with_symbols(self, symbols):
+        value = self.lib.set_new()
+        if not value:
+            raise RuntimeError("set_new returned NULL")
+        for symbol in symbols:
+            self.lib.set_add(value, symbol)
+        return value
+
+    def release(self, value, result):
+        if result:
+            LIBC.free(result)
+        self.lib.set_free(value)
+
+    def encode(self, symbols, bpp):
+        value = self.new_with_symbols(symbols)
+        result = self.lib.set_fini(value, bpp)
+        if not result:
+            raise RuntimeError("set_fini returned NULL")
+        encoded = ctypes.string_at(result)
+        self.release(value, result)
+        return encoded
+
+
+def median_fini(api, symbols, bpp, calls, rounds):
+    samples = []
+    for _ in range(rounds):
+        sets = [api.new_with_symbols(symbols) for _ in range(calls)]
+        results = []
+        start = time.perf_counter_ns()
+        for value in sets:
+            results.append(api.lib.set_fini(value, bpp))
+        samples.append((time.perf_counter_ns() - start) / calls)
+        if not all(results):
+            raise RuntimeError("set_fini returned NULL")
+        encoded = [ctypes.string_at(result) for result in results]
+        if len(set(encoded)) != 1:
+            raise RuntimeError("set_fini is not deterministic")
+        for value, result in zip(sets, results):
+            api.release(value, result)
+    return statistics.median(samples)
+
+
+def median_build(api, symbols, bpp, calls, rounds):
+    samples = []
+    for _ in range(rounds):
+        sets = []
+        results = []
+        start = time.perf_counter_ns()
+        for _ in range(calls):
+            value = api.new_with_symbols(symbols)
+            result = api.lib.set_fini(value, bpp)
+            sets.append(value)
+            results.append(result)
+        samples.append((time.perf_counter_ns() - start) / calls)
+        if not all(results):
+            raise RuntimeError("set_fini returned NULL")
+        for value, result in zip(sets, results):
+            api.release(value, result)
+    return statistics.median(samples)
+
+
+def median_add(api, symbols, calls, rounds):
+    samples = []
+    for _ in range(rounds):
+        values = []
+        start = time.perf_counter_ns()
+        for _ in range(calls):
+            values.append(api.new_with_symbols(symbols))
+        samples.append((time.perf_counter_ns() - start) / calls)
+        for value in values:
+            api.lib.set_free(value)
+    return statistics.median(samples)
+
+
+def median_cmp(api, provider, requirement, calls, rounds):
+    expected = api.lib.rpmsetcmp(provider, requirement)
+    if expected != 1 or api.lib.rpmsetcmp(provider, provider) != 0:
+        raise RuntimeError(f"unexpected rpmsetcmp result: {expected}")
+    for _ in range(100):
+        api.lib.rpmsetcmp(provider, requirement)
+
+    samples = []
+    for _ in range(rounds):
+        checksum = 0
+        start = time.perf_counter_ns()
+        for _ in range(calls):
+            checksum += api.lib.rpmsetcmp(provider, requirement)
+        samples.append((time.perf_counter_ns() - start) / calls)
+        if checksum != calls:
+            raise RuntimeError("rpmsetcmp result changed during benchmark")
+    return statistics.median(samples)
+
+
+def cold_cmp_once(api, provider, requirement):
+    read_fd, write_fd = os.pipe()
+    pid = os.fork()
+    if pid == 0:
+        os.close(read_fd)
+        start = time.perf_counter_ns()
+        result = api.lib.rpmsetcmp(provider, requirement)
+        elapsed = time.perf_counter_ns() - start
+        os.write(write_fd, f"{elapsed} {result}".encode())
+        os.close(write_fd)
+        os._exit(0)
+
+    os.close(write_fd)
+    payload = b""
+    while chunk := os.read(read_fd, 128):
+        payload += chunk
+    os.close(read_fd)
+    _, status = os.waitpid(pid, 0)
+    if status != 0:
+        raise RuntimeError(f"cold rpmsetcmp child failed: status={status}")
+    elapsed, result = map(int, payload.split())
+    if result != 1:
+        raise RuntimeError(f"unexpected cold rpmsetcmp result: {result}")
+    return elapsed
+
+
+def median_cmp_cold(api, provider, requirement, calls, rounds):
+    samples = []
+    for _ in range(rounds):
+        total = sum(cold_cmp_once(api, provider, requirement) for _ in range(calls))
+        samples.append(total / calls)
+    return statistics.median(samples)
+
+
+def verify_complete_decoding(api, provider, requirement):
+    if api.lib.rpmsetcmp(provider, requirement) != 1:
+        raise RuntimeError("provider must contain requirement")
+    if api.lib.rpmsetcmp(requirement, provider) != -1:
+        raise RuntimeError("requirement must be contained by provider")
+
+    payload_position = 4 + (len(provider) - 5) * 3 // 4
+    corrupted = provider[:payload_position] + b"!" + provider[payload_position + 1 :]
+    if api.lib.rpmsetcmp(corrupted, requirement) != -3:
+        raise RuntimeError("first operand was not decoded and validated completely")
+    if api.lib.rpmsetcmp(requirement, corrupted) != -4:
+        raise RuntimeError("second operand was not decoded and validated completely")
+
+
+def format_time(ns):
+    return f"{ns / 1000:.2f} us"
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Compare set9 and direct-hash set APIs")
+    parser.add_argument("--symbols", type=int, default=1000)
+    parser.add_argument(
+        "--required",
+        type=int,
+        help="number of evenly distributed required symbols (default: every second symbol)",
+    )
+    parser.add_argument("--bpp", type=int, default=32)
+    parser.add_argument("--rounds", type=int, default=7)
+    parser.add_argument("--fini-calls", type=int, default=5)
+    parser.add_argument("--cmp-calls", type=int, default=2000)
+    parser.add_argument("--cold-calls", type=int, default=20)
+    parser.add_argument("--skip-build", action="store_true")
+    args = parser.parse_args()
+    if args.symbols < 2 or not 10 <= args.bpp <= 32:
+        parser.error("symbols must be >= 2 and bpp must be in 10..32")
+    if args.required is not None and not 1 <= args.required < args.symbols:
+        parser.error("required must be in 1..symbols-1")
+    if min(args.rounds, args.fini_calls, args.cmp_calls, args.cold_calls) < 1:
+        parser.error("rounds and call counts must be positive")
+
+    if not args.skip_build:
+        subprocess.run([str(HERE / "build.sh")], check=True)
+
+    symbols = tuple(
+        f"symbol_{i:08d}_version_ALT_{i % 97}".encode() for i in range(args.symbols)
+    )
+    required = (
+        symbols[::2]
+        if args.required is None
+        else tuple(symbols[i * args.symbols // args.required] for i in range(args.required))
+    )
+    apis = {
+        "set9": SetAPI(BUILD / "libset9.so"),
+        "direct": SetAPI(BUILD / "libdirect-hash.so"),
+    }
+
+    gc.disable()
+    try:
+        timings = {name: [[] for _ in range(5)] for name in apis}
+        lengths = {}
+        encoded = {}
+        for name, api in apis.items():
+            provider = api.encode(symbols, args.bpp)
+            requirement = api.encode(required, args.bpp)
+            encoded[name] = (provider, requirement)
+            wire_format = "D1/base64" if provider.startswith(b"D1") else "golomb/base62"
+            lengths[name] = (len(provider), wire_format)
+
+        operations = (
+            lambda name, api: median_fini(api, symbols, args.bpp, args.fini_calls, 1),
+            lambda name, api: median_add(api, symbols, args.fini_calls, 1),
+            lambda name, api: median_build(api, symbols, args.bpp, args.fini_calls, 1),
+            lambda name, api: median_cmp_cold(
+                api, encoded[name][0], encoded[name][1], args.cold_calls, 1
+            ),
+            lambda name, api: median_cmp(
+                api, encoded[name][0], encoded[name][1], args.cmp_calls, 1
+            ),
+        )
+        names = tuple(apis)
+        for operation_index, operation in enumerate(operations):
+            for round_index in range(args.rounds):
+                order = names if round_index % 2 == 0 else tuple(reversed(names))
+                for name in order:
+                    timings[name][operation_index].append(operation(name, apis[name]))
+        timings = {
+            name: tuple(statistics.median(samples) for samples in operation_samples)
+            for name, operation_samples in timings.items()
+        }
+        # Run validation after timing: forked cold samples must inherit an empty
+        # decoded-set cache from the parent process.
+        for name, api in apis.items():
+            verify_complete_decoding(api, *encoded[name])
+    finally:
+        gc.enable()
+
+    print(f"symbols={args.symbols} required={len(required)} bpp={args.bpp}")
+    print("implementation  set_chars  format")
+    for name in apis:
+        print(f"{name:<14} {lengths[name][0]:>9}  {lengths[name][1]}")
+    print("\noperation                 set9       direct   direct/set9")
+    labels = (
+        "set_fini only",
+        "new+add (ctypes)",
+        "new+add+fini (ctypes)",
+        "rpmsetcmp cold",
+        "rpmsetcmp warm",
+    )
+    for index, label in enumerate(labels):
+        old = timings["set9"][index]
+        new = timings["direct"][index]
+        print(f"{label:<22} {format_time(old):>10} {format_time(new):>10} {new / old:>12.2f}x")
+
+
+if __name__ == "__main__":
+    main()

new_version/direct_hash/build.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+HERE=$(cd "$(dirname "$0")" && pwd)
+ROOT=$(cd "$HERE/../.." && pwd)
+BUILD="$HERE/build"
+mkdir -p "$BUILD"
+touch "$BUILD/rpmlib.h" "$BUILD/system.h"
+cp "$HERE/../roaring_bitmap/set.h" "$BUILD/set.h"
+
+CFLAGS=(-O2 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -Werror -I"$BUILD")
+COMPAT=(-include "$ROOT/scripts/rpmsetcmp/newset_compat.h")
+
+for tool in mkset setcmp; do
+  cc "${CFLAGS[@]}" "${COMPAT[@]}" \
+    "$ROOT/reimplement/set9.c" "$ROOT/scripts/rpmsetcmp/$tool.c" \
+    -o "$BUILD/$tool-set9"
+  cc "${CFLAGS[@]}" "${COMPAT[@]}" \
+    "$HERE/hash_set.c" "$ROOT/scripts/rpmsetcmp/$tool.c" \
+    -o "$BUILD/$tool-direct"
+done
+
+cc "${CFLAGS[@]}" -fPIC -shared "${COMPAT[@]}" \
+  "$ROOT/reimplement/set9.c" -o "$BUILD/libset9.so"
+cc "${CFLAGS[@]}" -fPIC -shared "${COMPAT[@]}" \
+  "$HERE/hash_set.c" -o "$BUILD/libdirect-hash.so"
+
+printf 'Built tools and benchmark libraries in %s\n' "$BUILD"

new_version/direct_hash/hash_set.c

@@ -0,0 +1,959 @@
+#include <assert.h>
+#include <limits.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/types.h>
+
+#include "rpmlib.h"
+#include "set.h"
+#include "system.h"
+
+/*
+ * This is intentionally a new set-string format. It is not compatible with
+ * the Golomb-Rice/base62 strings produced by the original lib/set.c.
+ *
+ *   D1<two decimal bpp digits><unpadded RFC 4648 base64 of packed hashes>
+ *
+ * Sorted unique hashes are packed least-significant bit first, using exactly
+ * bpp bits per hash. Base64 is only a textual representation of those bytes;
+ * there is no delta or Golomb-Rice coding.
+ */
+#define FORMAT_PREFIX "D1"
+#define FORMAT_HEADER_LEN 4
+
+_Static_assert(CHAR_BIT == 8, "direct-hash format requires 8-bit bytes");
+
+struct set {
+  size_t cnt;
+  size_t symbols_cap;
+  size_t strings_len;
+  size_t strings_cap;
+  char* strings;
+  struct symbols {
+    size_t offset;
+    unsigned hash;
+  }* symbols_v;
+};
+
+struct decoded_set {
+  unsigned* hashes;
+  size_t count;
+  unsigned bpp;
+};
+
+enum {
+  DECODED_CACHE_SIZE = 512,
+  DECODED_CACHE_BUCKETS = 1024,
+  PAIR_CACHE_SIZE = 4,
+};
+
+struct decoded_cache_entry;
+
+struct pair_cache_entry {
+  uint64_t other_identity;
+  int result;
+};
+
+struct decoded_cache_entry {
+  struct decoded_cache_entry* bucket_next;
+  struct decoded_cache_entry* newer;
+  struct decoded_cache_entry* older;
+  char* str;
+  unsigned* hashes;
+  size_t len;
+  size_t count;
+  uint32_t fingerprint;
+  uint64_t identity;
+  unsigned bucket;
+  unsigned target_bpp;
+  unsigned pair_next;
+  struct pair_cache_entry pairs[PAIR_CACHE_SIZE];
+};
+
+struct set_meta {
+  const char* str;
+  size_t len;
+  unsigned bpp;
+};
+
+static unsigned decoded_cache_count[2];
+static struct decoded_cache_entry* decoded_cache_buckets[2][DECODED_CACHE_BUCKETS];
+static struct decoded_cache_entry* decoded_cache_newest[2];
+static struct decoded_cache_entry* decoded_cache_oldest[2];
+static uint64_t decoded_cache_next_identity = 1;
+/* Cached arrays remain in use until comparison completes, so lookup, eviction,
+ * and comparison share one lock. */
+static atomic_flag decoded_cache_lock = ATOMIC_FLAG_INIT;
+
+struct set* set_new(void) {
+  struct set* set = xmalloc(sizeof(*set));
+  set->cnt = 0;
+  set->symbols_cap = 0;
+  set->strings_len = 0;
+  set->strings_cap = 0;
+  set->strings = NULL;
+  set->symbols_v = NULL;
+
+  return set;
+}
+
+void set_add(struct set* set, const char* sym) {
+  if (set->cnt == set->symbols_cap) {
+    set->symbols_cap += 1024;
+    set->symbols_v = xrealloc(set->symbols_v, sizeof(*set->symbols_v) * set->symbols_cap);
+  }
+
+  size_t length = strlen(sym) + 1;
+  size_t required = set->strings_len + length;
+  if (required > set->strings_cap) {
+    size_t capacity = set->strings_cap ? set->strings_cap : 4096;
+    while (capacity < required) capacity *= 2;
+    set->strings = xrealloc(set->strings, capacity);
+    set->strings_cap = capacity;
+  }
+
+  set->symbols_v[set->cnt].offset = set->strings_len;
+  set->symbols_v[set->cnt].hash = 0;
+  memcpy(set->strings + set->strings_len, sym, length);
+  set->strings_len = required;
+  ++set->cnt;
+
+  return;
+}
+
+struct set* set_free(struct set* set) {
+  if (set) {
+    _free(set->strings);
+    _free(set->symbols_v);
+    set = _free(set);
+  }
+
+  return NULL;
+}
+
+static unsigned hash(const char* str) {
+  unsigned hash = UINT32_C(0x9e3779b9);
+  const unsigned char* p = (const unsigned char*)str;
+
+  while (*p) {
+    hash += *p++;
+    hash += hash << 10;
+    hash ^= hash >> 6;
+  }
+
+  hash += hash << 3;
+  hash ^= hash >> 11;
+  hash += hash << 15;
+
+  return hash;
+}
+
+static int compare_symbols(const void* arg1, const void* arg2) {
+  const struct symbols* s1 = arg1;
+  const struct symbols* s2 = arg2;
+
+  if (s1->hash > s2->hash) return 1;
+  if (s1->hash < s2->hash) return -1;
+
+  return 0;
+}
+
+static void sort_symbols(struct symbols* values, size_t count, unsigned bpp) {
+  if (count < 128) {
+    qsort(values, count, sizeof(*values), compare_symbols);
+
+    return;
+  }
+
+  struct symbols* temporary = xmalloc(count * sizeof(*temporary));
+  struct symbols* source = values;
+  struct symbols* destination = temporary;
+  unsigned passes = (bpp + 7) / 8;
+
+  for (unsigned pass = 0; pass < passes; ++pass) {
+    size_t offsets[256] = {0};
+    unsigned shift = pass * 8;
+    for (size_t i = 0; i < count; ++i) ++offsets[(source[i].hash >> shift) & 0xffu];
+
+    size_t position = 0;
+    for (size_t i = 0; i < 256; ++i) {
+      size_t bucket_count = offsets[i];
+      offsets[i] = position;
+      position += bucket_count;
+    }
+
+    for (size_t i = 0; i < count; ++i) {
+      unsigned bucket = (source[i].hash >> shift) & 0xffu;
+      destination[offsets[bucket]++] = source[i];
+    }
+
+    struct symbols* swap = source;
+    source = destination;
+    destination = swap;
+  }
+
+  if (source != values) memcpy(values, source, count * sizeof(*values));
+  _free(temporary);
+
+  return;
+}
+
+static const char base64_alphabet[] =
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+
+static size_t base64_encoded_size(size_t byte_count) {
+  if (byte_count > SIZE_MAX - 2) abort();
+  size_t groups = (byte_count + 2) / 3;
+  if (groups > (SIZE_MAX - FORMAT_HEADER_LEN - 1) / 4) abort();
+
+  size_t size = groups * 4;
+  size_t remainder = byte_count % 3;
+  if (remainder != 0) size -= 3 - remainder;
+  return size;
+}
+
+static void base64_encode(const unsigned char* input, size_t input_len, char* output) {
+  while (input_len >= 3) {
+    uint32_t value = ((uint32_t)input[0] << 16) | ((uint32_t)input[1] << 8) | input[2];
+    output[0] = base64_alphabet[(value >> 18) & 0x3f];
+    output[1] = base64_alphabet[(value >> 12) & 0x3f];
+    output[2] = base64_alphabet[(value >> 6) & 0x3f];
+    output[3] = base64_alphabet[value & 0x3f];
+    input += 3;
+    input_len -= 3;
+    output += 4;
+  }
+
+  if (input_len == 1) {
+    uint32_t value = (uint32_t)input[0] << 16;
+    output[0] = base64_alphabet[(value >> 18) & 0x3f];
+    output[1] = base64_alphabet[(value >> 12) & 0x3f];
+  } else if (input_len == 2) {
+    uint32_t value = ((uint32_t)input[0] << 16) | ((uint32_t)input[1] << 8);
+    output[0] = base64_alphabet[(value >> 18) & 0x3f];
+    output[1] = base64_alphabet[(value >> 12) & 0x3f];
+    output[2] = base64_alphabet[(value >> 6) & 0x3f];
+  }
+
+  return;
+}
+
+static unsigned char* pack_hashes(const unsigned* hashes, size_t count, unsigned bpp,
+                                  size_t* byte_count) {
+  if (count > (SIZE_MAX - 7) / bpp) abort();
+  size_t bit_count = count * bpp;
+  *byte_count = (bit_count + 7) / 8;
+  unsigned char* bytes = xmalloc(*byte_count);
+  unsigned char* output = bytes;
+  uint64_t bits = 0;
+  unsigned filled = 0;
+
+  for (size_t i = 0; i < count; ++i) {
+    bits |= (uint64_t)hashes[i] << filled;
+    filled += bpp;
+
+    while (filled >= 8) {
+      *output++ = (unsigned char)bits;
+      bits >>= 8;
+      filled -= 8;
+    }
+  }
+
+  if (filled) *output++ = (unsigned char)bits;
+  assert((size_t)(output - bytes) == *byte_count);
+
+  return bytes;
+}
+
+const char* set_fini(struct set* set, int bpp) {
+  assert(set != NULL);
+  assert(set->cnt > 0);
+  assert(bpp >= 10 && bpp <= 32);
+
+  unsigned mask = bpp < 32 ? (UINT32_C(1) << bpp) - 1 : UINT32_MAX;
+  for (size_t i = 0; i < set->cnt; ++i) {
+    set->symbols_v[i].hash = hash(set->strings + set->symbols_v[i].offset) & mask;
+  }
+  sort_symbols(set->symbols_v, set->cnt, (unsigned)bpp);
+
+  for (size_t i = 0; i + 1 < set->cnt; ++i) {
+    if (set->symbols_v[i].hash != set->symbols_v[i + 1].hash) continue;
+    const char* left = set->strings + set->symbols_v[i].offset;
+    const char* right = set->strings + set->symbols_v[i + 1].offset;
+    if (strcmp(left, right) != 0) fprintf(stderr, "warning: hash collision: %s %s\n", left, right);
+  }
+
+  unsigned* unique_hashes = xmalloc(set->cnt * sizeof(*unique_hashes));
+  size_t unique_count = 0;
+  for (size_t i = 0; i < set->cnt; ++i) {
+    while (i + 1 < set->cnt && set->symbols_v[i].hash == set->symbols_v[i + 1].hash) ++i;
+    unique_hashes[unique_count++] = set->symbols_v[i].hash;
+  }
+
+  size_t byte_count;
+  unsigned char* allocated_bytes = NULL;
+  const unsigned char* bytes;
+#if UINT_MAX == UINT32_MAX && defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && \
+    __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+  if (bpp == 32 && sizeof(unsigned) == 4) {
+    byte_count = unique_count * sizeof(*unique_hashes);
+    bytes = (const unsigned char*)unique_hashes;
+  } else
+#endif
+  {
+    allocated_bytes = pack_hashes(unique_hashes, unique_count, (unsigned)bpp, &byte_count);
+    bytes = allocated_bytes;
+  }
+  size_t payload_len = base64_encoded_size(byte_count);
+  char* output = xmalloc(FORMAT_HEADER_LEN + payload_len + 1);
+  memcpy(output, FORMAT_PREFIX, sizeof(FORMAT_PREFIX) - 1);
+  output[2] = (char)('0' + bpp / 10);
+  output[3] = (char)('0' + bpp % 10);
+  base64_encode(bytes, byte_count, output + FORMAT_HEADER_LEN);
+  output[FORMAT_HEADER_LEN + payload_len] = '\0';
+
+  _free(allocated_bytes);
+  _free(unique_hashes);
+
+  return output;
+}
+
+static const unsigned char base64_values[256] = {
+    ['A'] = 1,  ['B'] = 2,  ['C'] = 3,  ['D'] = 4,  ['E'] = 5,  ['F'] = 6,  ['G'] = 7,  ['H'] = 8,
+    ['I'] = 9,  ['J'] = 10, ['K'] = 11, ['L'] = 12, ['M'] = 13, ['N'] = 14, ['O'] = 15, ['P'] = 16,
+    ['Q'] = 17, ['R'] = 18, ['S'] = 19, ['T'] = 20, ['U'] = 21, ['V'] = 22, ['W'] = 23, ['X'] = 24,
+    ['Y'] = 25, ['Z'] = 26, ['a'] = 27, ['b'] = 28, ['c'] = 29, ['d'] = 30, ['e'] = 31, ['f'] = 32,
+    ['g'] = 33, ['h'] = 34, ['i'] = 35, ['j'] = 36, ['k'] = 37, ['l'] = 38, ['m'] = 39, ['n'] = 40,
+    ['o'] = 41, ['p'] = 42, ['q'] = 43, ['r'] = 44, ['s'] = 45, ['t'] = 46, ['u'] = 47, ['v'] = 48,
+    ['w'] = 49, ['x'] = 50, ['y'] = 51, ['z'] = 52, ['0'] = 53, ['1'] = 54, ['2'] = 55, ['3'] = 56,
+    ['4'] = 57, ['5'] = 58, ['6'] = 59, ['7'] = 60, ['8'] = 61, ['9'] = 62, ['+'] = 63, ['/'] = 64,
+};
+
+static inline int base64_value(unsigned char c) { return (int)base64_values[c] - 1; }
+
+static int base64_decoded_size(size_t input_len, size_t* byte_count) {
+  size_t remainder = input_len % 4;
+  if (input_len == 0 || remainder == 1) return -1;
+
+  size_t groups = input_len / 4;
+  if (groups > SIZE_MAX / 3) return -1;
+  size_t size = groups * 3;
+  size_t tail_size = remainder == 0 ? 0 : remainder - 1;
+  if (size > SIZE_MAX - tail_size) return -1;
+
+  *byte_count = size + tail_size;
+  return 0;
+}
+
+static inline int has_set_prefix(const char* str) {
+  return str[0] == 's' && str[1] == 'e' && str[2] == 't' && str[3] == ':';
+}
+
+static int set_meta_init(const char* source, struct set_meta* meta) {
+  const char* str = source;
+  if (has_set_prefix(str)) str += 4;
+  if (has_set_prefix(str)) return -1;
+
+  /* With bpp >= 10, a valid direct set has at least three Base64 characters.
+   * Checking this fixed prefix makes cache hits independent of total key length. */
+  if (str[0] != FORMAT_PREFIX[0] || str[1] != FORMAT_PREFIX[1]) return -1;
+  if (str[2] < '0' || str[2] > '9' || str[3] < '0' || str[3] > '9') return -1;
+  if (str[4] == '\0' || str[5] == '\0' || str[6] == '\0') return -1;
+
+  unsigned bpp = (unsigned)(str[2] - '0') * 10 + (unsigned)(str[3] - '0');
+  if (bpp < 10 || bpp > 32) return -1;
+
+  meta->str = str;
+  meta->len = 0;
+  meta->bpp = bpp;
+  return 0;
+}
+
+struct decode_writer {
+  unsigned* hashes;
+  size_t capacity;
+  size_t written;
+  uint64_t bits;
+  uint64_t mask;
+  unsigned previous;
+  unsigned filled;
+  unsigned bpp;
+  int has_previous;
+};
+
+static inline int decode_writer_put(struct decode_writer* writer, uint32_t bytes,
+                                    unsigned byte_count) {
+  writer->bits |= (uint64_t)bytes << writer->filled;
+  writer->filled += byte_count * 8;
+
+  while (writer->filled >= writer->bpp) {
+    if (writer->written == writer->capacity) return -1;
+
+    unsigned current = (unsigned)(writer->bits & writer->mask);
+    writer->bits >>= writer->bpp;
+    writer->filled -= writer->bpp;
+
+    if (writer->has_previous && writer->previous >= current) return -1;
+    if (writer->hashes) writer->hashes[writer->written] = current;
+    writer->previous = current;
+    writer->has_previous = 1;
+    ++writer->written;
+  }
+
+  return 0;
+}
+
+static int decode_base64_bytes(const char* input, size_t input_len, unsigned char* output,
+                               size_t output_len) {
+  unsigned char* const output_end = output + output_len;
+  size_t offset = 0;
+  while (input_len - offset >= 4) {
+    int v0 = base64_value((unsigned char)input[offset]);
+    int v1 = base64_value((unsigned char)input[offset + 1]);
+    int v2 = base64_value((unsigned char)input[offset + 2]);
+    int v3 = base64_value((unsigned char)input[offset + 3]);
+    if ((v0 | v1 | v2 | v3) < 0 || output_end - output < 3) return -1;
+    output[0] = (unsigned char)((v0 << 2) | (v1 >> 4));
+    output[1] = (unsigned char)(((v1 & 0x0f) << 4) | (v2 >> 2));
+    output[2] = (unsigned char)(((v2 & 0x03) << 6) | v3);
+    output += 3;
+    offset += 4;
+  }
+
+  size_t remainder = input_len - offset;
+  if (remainder == 0) return output == output_end ? 0 : -1;
+  if (remainder == 1) return -1;
+
+  int v0 = base64_value((unsigned char)input[offset]);
+  int v1 = base64_value((unsigned char)input[offset + 1]);
+  if ((v0 | v1) < 0 || output == output_end) return -1;
+  *output++ = (unsigned char)((v0 << 2) | (v1 >> 4));
+
+  if (remainder == 2) return (v1 & 0x0f) == 0 && output == output_end ? 0 : -1;
+
+  int v2 = base64_value((unsigned char)input[offset + 2]);
+  if (v2 < 0 || output == output_end) return -1;
+  *output++ = (unsigned char)(((v1 & 0x0f) << 4) | (v2 >> 2));
+  return (v2 & 0x03) == 0 && output == output_end ? 0 : -1;
+}
+
+static inline int decode_base64_triplet(const char* input, uint32_t* triplet) {
+  int v0 = base64_value((unsigned char)input[0]);
+  int v1 = base64_value((unsigned char)input[1]);
+  int v2 = base64_value((unsigned char)input[2]);
+  int v3 = base64_value((unsigned char)input[3]);
+  if ((v0 | v1 | v2 | v3) < 0) return -1;
+
+  *triplet = (uint32_t)((v0 << 2) | (v1 >> 4)) |
+             (uint32_t)(((v1 & 0x0f) << 4) | (v2 >> 2)) << 8 |
+             (uint32_t)(((v2 & 0x03) << 6) | v3) << 16;
+  return 0;
+}
+
+static int decode_base64_u32(const char* input, size_t input_len, unsigned* hashes,
+                             size_t count) {
+  size_t blocks = count / 3;
+  size_t written = 0;
+  unsigned previous = 0;
+  int has_previous = 0;
+
+  for (size_t block = 0; block < blocks; ++block) {
+    uint32_t t0, t1, t2, t3;
+    if (decode_base64_triplet(input, &t0) < 0 ||
+        decode_base64_triplet(input + 4, &t1) < 0 ||
+        decode_base64_triplet(input + 8, &t2) < 0 ||
+        decode_base64_triplet(input + 12, &t3) < 0)
+      return -1;
+
+    unsigned value0 = (unsigned)(t0 | ((t1 & UINT32_C(0xff)) << 24));
+    unsigned value1 = (unsigned)((t1 >> 8) | ((t2 & UINT32_C(0xffff)) << 16));
+    unsigned value2 = (unsigned)((t2 >> 16) | (t3 << 8));
+    if ((has_previous && previous >= value0) || value0 >= value1 || value1 >= value2) return -1;
+
+    hashes[written++] = value0;
+    hashes[written++] = value1;
+    hashes[written++] = value2;
+    previous = value2;
+    has_previous = 1;
+    input += 16;
+    input_len -= 16;
+  }
+
+  size_t remaining = count - written;
+  if (remaining != 0) {
+    size_t byte_count = remaining * sizeof(*hashes);
+    unsigned char tail[2 * sizeof(*hashes)];
+    if (decode_base64_bytes(input, input_len, tail, byte_count) < 0) return -1;
+    for (size_t i = 0; i < remaining; ++i, ++written) {
+      const unsigned char* bytes = tail + i * 4;
+      unsigned current = (unsigned)bytes[0] | ((unsigned)bytes[1] << 8) |
+                         ((unsigned)bytes[2] << 16) | ((unsigned)bytes[3] << 24);
+      if (has_previous && previous >= current) return -1;
+      hashes[written] = current;
+      previous = current;
+      has_previous = 1;
+    }
+  } else if (input_len != 0) {
+    return -1;
+  }
+
+  return 0;
+}
+
+static int decode_set_sized(const char* str, size_t str_len, struct decoded_set* decoded) {
+  if (str_len == 0) str_len = strlen(str);
+  if (str_len <= FORMAT_HEADER_LEN) return -1;
+  if (strncmp(str, FORMAT_PREFIX, sizeof(FORMAT_PREFIX) - 1) != 0) return -1;
+  if (str[2] < '0' || str[2] > '9' || str[3] < '0' || str[3] > '9') return -1;
+
+  unsigned bpp = (unsigned)(str[2] - '0') * 10 + (unsigned)(str[3] - '0');
+  if (bpp < 10 || bpp > 32) return -1;
+
+  const char* input = str + FORMAT_HEADER_LEN;
+  size_t input_len = str_len - FORMAT_HEADER_LEN;
+  size_t byte_count;
+  if (base64_decoded_size(input_len, &byte_count) < 0) return -1;
+  if (byte_count > SIZE_MAX / 8) return -1;
+
+  size_t count = byte_count * 8 / bpp;
+  if (count == 0 || count > (SIZE_MAX - 7) / bpp || count > SIZE_MAX / sizeof(unsigned) ||
+      (count * bpp + 7) / 8 != byte_count) {
+    return -1;
+  }
+
+  unsigned* hashes = xmalloc(count * sizeof(*hashes));
+#if UINT_MAX == UINT32_MAX
+  if (bpp == 32 && sizeof(unsigned) == 4) {
+    if (decode_base64_u32(input, input_len, hashes, count) < 0) goto invalid;
+    if (decoded) {
+      decoded->hashes = hashes;
+      decoded->count = count;
+      decoded->bpp = bpp;
+    }
+    return 0;
+  }
+
+  /* Byte-aligned widths can bypass the generic bit reservoir. Decode the
+   * Base64 payload into the front of the final allocation, then expand 16-
+   * and 24-bit values backwards so unread packed bytes are never overwritten. */
+  if ((bpp == 16 || bpp == 24) && sizeof(unsigned) == 4) {
+    if (decode_base64_bytes(input, input_len, (unsigned char*)hashes, byte_count) < 0) goto invalid;
+    if (bpp == 16) {
+      for (size_t i = count; i > 0; --i) {
+        const unsigned char* bytes = (const unsigned char*)hashes + (i - 1) * 2;
+        hashes[i - 1] = (unsigned)bytes[0] | ((unsigned)bytes[1] << 8);
+      }
+    } else if (bpp == 24) {
+      for (size_t i = count; i > 0; --i) {
+        const unsigned char* bytes = (const unsigned char*)hashes + (i - 1) * 3;
+        hashes[i - 1] = (unsigned)bytes[0] | ((unsigned)bytes[1] << 8) |
+                        ((unsigned)bytes[2] << 16);
+      }
+    }
+    for (size_t i = 1; i < count; ++i) {
+      if (hashes[i - 1] >= hashes[i]) goto invalid;
+    }
+    if (decoded) {
+      decoded->hashes = hashes;
+      decoded->count = count;
+      decoded->bpp = bpp;
+    }
+    return 0;
+  }
+#endif
+
+  struct decode_writer writer = {
+      .hashes = hashes,
+      .capacity = count,
+      .mask = bpp < 32 ? (UINT64_C(1) << bpp) - 1 : UINT32_MAX,
+      .bpp = bpp,
+  };
+
+  size_t full_len = input_len - input_len % 4;
+  for (size_t offset = 0; offset < full_len; offset += 4) {
+    int v0 = base64_value((unsigned char)input[offset]);
+    int v1 = base64_value((unsigned char)input[offset + 1]);
+    int v2 = base64_value((unsigned char)input[offset + 2]);
+    int v3 = base64_value((unsigned char)input[offset + 3]);
+    if ((v0 | v1 | v2 | v3) < 0) goto invalid;
+
+    uint32_t bytes = (uint32_t)((v0 << 2) | (v1 >> 4)) |
+                     (uint32_t)(((v1 & 0x0f) << 4) | (v2 >> 2)) << 8 |
+                     (uint32_t)(((v2 & 0x03) << 6) | v3) << 16;
+    if (decode_writer_put(&writer, bytes, 3) < 0) goto invalid;
+  }
+
+  size_t remainder = input_len - full_len;
+  if (remainder != 0) {
+    int v0 = base64_value((unsigned char)input[full_len]);
+    int v1 = base64_value((unsigned char)input[full_len + 1]);
+    if ((v0 | v1) < 0) goto invalid;
+
+    uint32_t bytes = (uint32_t)((v0 << 2) | (v1 >> 4));
+    if (remainder == 2) {
+      if ((v1 & 0x0f) != 0 || decode_writer_put(&writer, bytes, 1) < 0) goto invalid;
+    } else {
+      int v2 = base64_value((unsigned char)input[full_len + 2]);
+      if (v2 < 0 || (v2 & 0x03) != 0) goto invalid;
+      bytes |= (uint32_t)(((v1 & 0x0f) << 4) | (v2 >> 2)) << 8;
+      if (decode_writer_put(&writer, bytes, 2) < 0) goto invalid;
+    }
+  }
+
+  if (writer.written != count || writer.bits != 0) goto invalid;
+
+  if (decoded) {
+    decoded->hashes = hashes;
+    decoded->count = count;
+    decoded->bpp = bpp;
+  }
+
+  return 0;
+
+invalid:
+  _free(hashes);
+
+  return -1;
+}
+
+/* Reduce a sorted set of (bpp + 1)-bit values to a sorted set of bpp-bit values. */
+static size_t downsample_set(size_t count, const unsigned* hashes, unsigned* result, unsigned bpp) {
+  unsigned mask = (UINT32_C(1) << bpp) - 1;
+  size_t lower = 0;
+  size_t upper = count;
+
+  while (lower < upper) {
+    size_t middle = lower + (upper - lower) / 2;
+    if (hashes[middle] <= mask)
+      lower = middle + 1;
+    else
+      upper = middle;
+  }
+
+  unsigned* output = result;
+  const unsigned* low = hashes;
+  const unsigned* low_end = hashes + lower;
+  const unsigned* high = hashes + lower;
+  const unsigned* high_end = hashes + count;
+
+  while (low < low_end && high < high_end) {
+    unsigned low_value = *low;
+    unsigned high_value = *high & mask;
+    if (low_value < high_value) {
+      *output++ = low_value;
+      ++low;
+    } else if (high_value < low_value) {
+      *output++ = high_value;
+      ++high;
+    } else {
+      *output++ = low_value;
+      ++low;
+      ++high;
+    }
+  }
+  while (low < low_end) *output++ = *low++;
+  while (high < high_end) *output++ = *high++ & mask;
+
+  return (size_t)(output - result);
+}
+
+static void downsample_radix_to(struct decoded_set* set, unsigned target_bpp) {
+  unsigned* original = set->hashes;
+  unsigned* scratch = xmalloc(set->count * sizeof(*scratch));
+  unsigned mask = (UINT32_C(1) << target_bpp) - 1;
+  for (size_t i = 0; i < set->count; ++i) original[i] &= mask;
+
+  unsigned* source = original;
+  unsigned* destination = scratch;
+  unsigned passes = (target_bpp + 7) / 8;
+  for (unsigned pass = 0; pass < passes; ++pass) {
+    size_t offsets[256] = {0};
+    unsigned shift = pass * 8;
+    for (size_t i = 0; i < set->count; ++i) ++offsets[(source[i] >> shift) & 0xffu];
+
+    size_t position = 0;
+    for (size_t i = 0; i < 256; ++i) {
+      size_t bucket_count = offsets[i];
+      offsets[i] = position;
+      position += bucket_count;
+    }
+    for (size_t i = 0; i < set->count; ++i) {
+      unsigned value = source[i];
+      destination[offsets[(value >> shift) & 0xffu]++] = value;
+    }
+
+    unsigned* swap = source;
+    source = destination;
+    destination = swap;
+  }
+
+  size_t unique_count = 1;
+  for (size_t i = 1; i < set->count; ++i) {
+    if (source[i] != source[unique_count - 1]) source[unique_count++] = source[i];
+  }
+
+  if (source == original) {
+    _free(scratch);
+  } else {
+    _free(original);
+    set->hashes = scratch;
+  }
+  set->count = unique_count;
+  set->bpp = target_bpp;
+}
+
+static void downsample_to(struct decoded_set* set, unsigned target_bpp) {
+  if (set->bpp == target_bpp) return;
+
+  unsigned passes = (target_bpp + 7) / 8;
+  if (set->bpp - target_bpp > passes) {
+    downsample_radix_to(set, target_bpp);
+    return;
+  }
+
+  unsigned* original = set->hashes;
+  unsigned* scratch = xmalloc(set->count * sizeof(*scratch));
+  unsigned* source = original;
+  unsigned* destination = scratch;
+
+  while (set->bpp > target_bpp) {
+    --set->bpp;
+    set->count = downsample_set(set->count, source, destination, set->bpp);
+    unsigned* swap = source;
+    source = destination;
+    destination = swap;
+  }
+
+  if (source == original) {
+    _free(scratch);
+  } else {
+    _free(original);
+    set->hashes = scratch;
+  }
+
+  return;
+}
+
+static uint32_t decoded_cache_fingerprint(const struct set_meta* meta, unsigned target_bpp) {
+  const unsigned char* str = (const unsigned char*)meta->str;
+  uint32_t fingerprint = (uint32_t)str[4] | ((uint32_t)str[5] << 8) |
+                         ((uint32_t)str[6] << 16) | ((uint32_t)str[7] << 24);
+  fingerprint ^= meta->bpp * UINT32_C(0x27d4eb2d);
+  fingerprint ^= target_bpp * UINT32_C(0x85ebca6b);
+  fingerprint ^= fingerprint >> 11;
+  fingerprint *= UINT32_C(0x9e3779b1);
+  fingerprint ^= fingerprint >> 16;
+  return fingerprint;
+}
+
+static void decoded_cache_touch(struct decoded_cache_entry* entry, unsigned cache_id) {
+  if (entry == decoded_cache_newest[cache_id]) return;
+
+  if (entry->newer) entry->newer->older = entry->older;
+  if (entry->older) entry->older->newer = entry->newer;
+  if (entry == decoded_cache_oldest[cache_id]) decoded_cache_oldest[cache_id] = entry->newer;
+
+  entry->newer = NULL;
+  entry->older = decoded_cache_newest[cache_id];
+  decoded_cache_newest[cache_id]->newer = entry;
+  decoded_cache_newest[cache_id] = entry;
+}
+
+static void decoded_cache_remove(struct decoded_cache_entry* victim, unsigned cache_id) {
+  if (victim->newer)
+    victim->newer->older = victim->older;
+  else
+    decoded_cache_newest[cache_id] = victim->older;
+  if (victim->older)
+    victim->older->newer = victim->newer;
+  else
+    decoded_cache_oldest[cache_id] = victim->newer;
+
+  struct decoded_cache_entry** link = &decoded_cache_buckets[cache_id][victim->bucket];
+  while (*link && *link != victim) link = &(*link)->bucket_next;
+  assert(*link == victim);
+  *link = victim->bucket_next;
+  assert(decoded_cache_count[cache_id] > 0);
+  --decoded_cache_count[cache_id];
+
+  _free(victim->hashes);
+  _free(victim);
+}
+
+static void decoded_cache_reset_pair_identities(void) {
+  for (unsigned bucket = 0; bucket < DECODED_CACHE_BUCKETS; ++bucket) {
+    for (struct decoded_cache_entry* provider = decoded_cache_buckets[0][bucket]; provider;
+         provider = provider->bucket_next) {
+      for (unsigned i = 0; i < PAIR_CACHE_SIZE; ++i) provider->pairs[i].other_identity = 0;
+    }
+  }
+  decoded_cache_next_identity = 1;
+}
+
+static int cache_decode_set(const struct set_meta* meta, unsigned target_bpp, unsigned cache_id,
+                            const unsigned** hashes, size_t* count,
+                            struct decoded_cache_entry** cache_entry) {
+  assert(cache_id < 2);
+  assert(target_bpp <= meta->bpp);
+
+  uint32_t fingerprint = decoded_cache_fingerprint(meta, target_bpp);
+  unsigned bucket = fingerprint & (DECODED_CACHE_BUCKETS - 1);
+  for (struct decoded_cache_entry* entry = decoded_cache_buckets[cache_id][bucket]; entry;
+       entry = entry->bucket_next) {
+    if (entry->fingerprint != fingerprint || entry->target_bpp != target_bpp ||
+        strcmp(entry->str, meta->str) != 0)
+      continue;
+
+    decoded_cache_touch(entry, cache_id);
+    *hashes = entry->hashes;
+    *count = entry->count;
+    *cache_entry = entry;
+    return 0;
+  }
+
+  size_t len = strlen(meta->str);
+
+  struct decoded_set decoded;
+  if (decode_set_sized(meta->str, len, &decoded) < 0) return -1;
+  if (decoded.bpp != meta->bpp) {
+    _free(decoded.hashes);
+    return -1;
+  }
+  downsample_to(&decoded, target_bpp);
+
+  if (len > SIZE_MAX - sizeof(struct decoded_cache_entry) - 1) {
+    _free(decoded.hashes);
+    return -1;
+  }
+  struct decoded_cache_entry* entry = xmalloc(sizeof(*entry) + len + 1);
+  memset(entry, 0, sizeof(*entry));
+  entry->str = (char*)(entry + 1);
+  memcpy(entry->str, meta->str, len + 1);
+  entry->hashes = decoded.hashes;
+  entry->len = len;
+  entry->count = decoded.count;
+  entry->fingerprint = fingerprint;
+  if (decoded_cache_next_identity == 0) decoded_cache_reset_pair_identities();
+  entry->identity = decoded_cache_next_identity++;
+  entry->bucket = bucket;
+  entry->target_bpp = target_bpp;
+
+  if (decoded_cache_count[cache_id] == DECODED_CACHE_SIZE) {
+    decoded_cache_remove(decoded_cache_oldest[cache_id], cache_id);
+  }
+
+  entry->bucket_next = decoded_cache_buckets[cache_id][bucket];
+  decoded_cache_buckets[cache_id][bucket] = entry;
+  entry->older = decoded_cache_newest[cache_id];
+  if (decoded_cache_newest[cache_id]) {
+    decoded_cache_newest[cache_id]->newer = entry;
+  } else {
+    decoded_cache_oldest[cache_id] = entry;
+  }
+  decoded_cache_newest[cache_id] = entry;
+  ++decoded_cache_count[cache_id];
+
+  *hashes = entry->hashes;
+  *count = entry->count;
+  *cache_entry = entry;
+  return 0;
+}
+
+static const unsigned* step_lower_bound(const unsigned* first, const unsigned* last, unsigned value,
+                                        size_t jump) {
+  size_t count = (size_t)(last - first);
+  if (count == 0 || first[0] >= value) return first;
+  if (jump == 0) jump = 1;
+
+  size_t position = 0;
+  size_t step = jump;
+  while (step != 0) {
+    if (step > count - position - 1) {
+      step /= 2;
+      continue;
+    }
+    size_t next = position + step;
+    if (first[next] < value)
+      position = next;
+    else
+      step /= 2;
+  }
+
+  return first + position + 1;
+}
+
+static int sorted_subset(const unsigned* small, size_t small_count, const unsigned* large,
+                         size_t large_count) {
+  const unsigned* small_end = small + small_count;
+  const unsigned* large_end = large + large_count;
+  size_t jump = large_count / small_count;
+
+  if (jump < 4) {
+    while (small < small_end) {
+      unsigned value = *small++;
+      while (large < large_end && *large < value) ++large;
+      if (large == large_end || *large != value) return 0;
+      ++large;
+    }
+
+    return 1;
+  }
+
+  while (small < small_end) {
+    unsigned value = *small++;
+    large = step_lower_bound(large, large_end, value, jump);
+    if (large == large_end || *large != value) return 0;
+    ++large;
+  }
+
+  return 1;
+}
+
+static int rpmsetcmp_locked(const char* str1, const char* str2) {
+  struct set_meta meta1;
+  if (set_meta_init(str1, &meta1) < 0) return -3;
+
+  struct set_meta meta2;
+  int meta2_status = set_meta_init(str2, &meta2);
+  unsigned target_bpp =
+      meta2_status == 0 && meta2.bpp < meta1.bpp ? meta2.bpp : meta1.bpp;
+
+  const unsigned* hashes1;
+  size_t count1;
+  struct decoded_cache_entry* entry1;
+  if (cache_decode_set(&meta1, target_bpp, 0, &hashes1, &count1, &entry1) < 0) return -3;
+  if (meta2_status < 0) return -4;
+
+  const unsigned* hashes2;
+  size_t count2;
+  struct decoded_cache_entry* entry2;
+  if (cache_decode_set(&meta2, target_bpp, 1, &hashes2, &count2, &entry2) < 0) return -4;
+
+  for (unsigned i = 0; i < PAIR_CACHE_SIZE; ++i) {
+    if (entry1->pairs[i].other_identity == entry2->identity) return entry1->pairs[i].result;
+  }
+
+  int result;
+  if (count1 == count2)
+    result = memcmp(hashes1, hashes2, count1 * sizeof(*hashes1)) == 0 ? 0 : -2;
+  else if (count1 > count2)
+    result = sorted_subset(hashes2, count2, hashes1, count1) ? 1 : -2;
+  else
+    result = sorted_subset(hashes1, count1, hashes2, count2) ? -1 : -2;
+
+  struct pair_cache_entry* pair = &entry1->pairs[entry1->pair_next++ % PAIR_CACHE_SIZE];
+  pair->other_identity = entry2->identity;
+  pair->result = result;
+  return result;
+}
+
+int rpmsetcmp(const char* str1, const char* str2) {
+  while (atomic_flag_test_and_set_explicit(&decoded_cache_lock, memory_order_acquire)) {
+  }
+  int result = rpmsetcmp_locked(str1, str2);
+  atomic_flag_clear_explicit(&decoded_cache_lock, memory_order_release);
+  return result;
+}

new_version/main_ideas.md

@@ -0,0 +1,40 @@
+# Другой дизайн
+
+## Roaring map
+
+- описание [здесь](new_version/roaring_bitmap/about.md)
+
+## хранить сразу расшифрованный set
+
+- WIP [здесь](new_version/direct_hash/about.md)
+- огромная часть ресурсов уходит не на просмотр включения множеств, а на декодирование set-строк
+  - при возможности хранить больше данных за более дешёвое сравнение - прекрасно
+  - тупо условный формат:
+    - `<bpp> <последовательно упакованные bpp-битные хеши>`
+- Shannon–Fano–Elias coding как одна из идей, но надо глубже копать
+
+## Группировать, а не хэшировать
+
+- если была бы возможность точно делать соответствия между label и id, то provides стал бы в большинстве последовательным, а required было бы легко искать в P.
+
+# Доработки на текущий дизайн
+
+## битовый prefilter
+
+- Позволяет отбросить заведомо ложные варианты
+  - но так ли часто это будет срабатывать, но вызывает доп расходы при высчитывании
+- фильтр Блума как пример
+  - В теории хэш можно заменить на него
+    - необходимо знать, насколько больше будет ложноположительных срабатываний
+
+## улучшить проход по массивам
+
+- Можно provides хранить не в виде массива, а сразу как хэш-структурку
+  - если provides строка кэшируется не так часто, смысла не будет
+
+# Прочие улучшения
+
+## улучшение работы с хэшем
+
+- если условно "отсортировать" массив provides/requires, можно получить лучшую работу с кэшом
+  - (надеюсь, что под капотом оно уже и так это делает, но проверить стоит)

new_version/roaring_bitmap/about.md

@@ -0,0 +1,39 @@
+попытка воссоздания алгоритма с roaring bitmap
+
+Даже эффективней в создании, но проигрывает в дешифровке (сравнении) и занимаемому месту
+
+Кратно выше становится сама длина строки
+
+```text
+symbols=1000 required=500 bpp=32
+implementation  set_chars  format
+set9                3994  golomb
+bitmap             19924  R1
+```
+
+```text
+operation                 set9       bitmap   bitmap/set9
+set_fini only           266.24 us  202.89 us         0.76x
+new+add+fini           1696.05 us 1600.44 us         0.94x
+rpmsetcmp cold          293.87 us 1321.17 us         4.50x
+rpmsetcmp warm            5.24 us  399.20 us        76.14x
+```
+
+### Использование zstd
+
+```text
+symbols=1000 required=500 bpp=32
+implementation  set_chars  format
+set9                3994  golomb
+bitmap             14126  R2
+```
+
+без zstd строка становится ~40% длинее
+
+```text
+operation                 set9       bitmap   bitmap/set9
+set_fini only           171.84 us  314.20 us         1.83x
+new+add+fini           1185.33 us 1726.32 us         1.46x
+rpmsetcmp cold          285.04 us 1412.84 us         4.96x
+rpmsetcmp warm            4.82 us  490.94 us       101.86x
+```

new_version/roaring_bitmap/benchmark.py

@@ -0,0 +1,206 @@
+#!/usr/bin/env python3
+import argparse
+import ctypes
+import gc
+import os
+import statistics
+import subprocess
+import time
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+BUILD = HERE / "build"
+LIBC = ctypes.CDLL(None)
+LIBC.free.argtypes = [ctypes.c_void_p]
+
+
+class SetAPI:
+    def __init__(self, path: Path, result_needs_free: bool):
+        self.lib = ctypes.CDLL(str(path))
+        self.result_needs_free = result_needs_free
+        self.lib.set_new.restype = ctypes.c_void_p
+        self.lib.set_add.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
+        self.lib.set_fini.argtypes = [ctypes.c_void_p, ctypes.c_int]
+        self.lib.set_fini.restype = ctypes.c_void_p
+        self.lib.set_free.argtypes = [ctypes.c_void_p]
+        self.lib.set_free.restype = ctypes.c_void_p
+        self.lib.rpmsetcmp.argtypes = [ctypes.c_char_p, ctypes.c_char_p]
+        self.lib.rpmsetcmp.restype = ctypes.c_int
+
+    def new_with_symbols(self, symbols):
+        value = self.lib.set_new()
+        if not value:
+            raise RuntimeError("set_new returned NULL")
+        for symbol in symbols:
+            self.lib.set_add(value, symbol)
+        return value
+
+    def release(self, value, result):
+        if self.result_needs_free and result:
+            LIBC.free(result)
+        self.lib.set_free(value)
+
+    def encode(self, symbols, bpp):
+        value = self.new_with_symbols(symbols)
+        result = self.lib.set_fini(value, bpp)
+        if not result:
+            raise RuntimeError("set_fini returned NULL")
+        encoded = ctypes.string_at(result)
+        self.release(value, result)
+        return encoded
+
+
+def median_fini(api, symbols, bpp, calls, rounds):
+    samples = []
+    for _ in range(rounds):
+        sets = [api.new_with_symbols(symbols) for _ in range(calls)]
+        results = []
+        start = time.perf_counter_ns()
+        for value in sets:
+            results.append(api.lib.set_fini(value, bpp))
+        samples.append((time.perf_counter_ns() - start) / calls)
+        if not all(results):
+            raise RuntimeError("set_fini returned NULL")
+        encoded = [ctypes.string_at(result) for result in results]
+        if len(set(encoded)) != 1:
+            raise RuntimeError("set_fini is not deterministic")
+        for value, result in zip(sets, results):
+            api.release(value, result)
+    return statistics.median(samples)
+
+
+def median_build(api, symbols, bpp, calls, rounds):
+    samples = []
+    for _ in range(rounds):
+        sets = []
+        results = []
+        start = time.perf_counter_ns()
+        for _ in range(calls):
+            value = api.new_with_symbols(symbols)
+            result = api.lib.set_fini(value, bpp)
+            sets.append(value)
+            results.append(result)
+        samples.append((time.perf_counter_ns() - start) / calls)
+        if not all(results):
+            raise RuntimeError("set_fini returned NULL")
+        for value, result in zip(sets, results):
+            api.release(value, result)
+    return statistics.median(samples)
+
+
+def median_cmp(api, provider, requirement, calls, rounds):
+    expected = api.lib.rpmsetcmp(provider, requirement)
+    if expected != 1 or api.lib.rpmsetcmp(provider, provider) != 0:
+        raise RuntimeError(f"unexpected rpmsetcmp result: {expected}")
+    for _ in range(100):
+        api.lib.rpmsetcmp(provider, requirement)
+
+    samples = []
+    for _ in range(rounds):
+        checksum = 0
+        start = time.perf_counter_ns()
+        for _ in range(calls):
+            checksum += api.lib.rpmsetcmp(provider, requirement)
+        samples.append((time.perf_counter_ns() - start) / calls)
+        if checksum != calls:
+            raise RuntimeError("rpmsetcmp result changed during benchmark")
+    return statistics.median(samples)
+
+
+def cold_cmp_once(api, provider, requirement):
+    read_fd, write_fd = os.pipe()
+    pid = os.fork()
+    if pid == 0:
+        os.close(read_fd)
+        start = time.perf_counter_ns()
+        result = api.lib.rpmsetcmp(provider, requirement)
+        elapsed = time.perf_counter_ns() - start
+        os.write(write_fd, f"{elapsed} {result}".encode())
+        os.close(write_fd)
+        os._exit(0)
+
+    os.close(write_fd)
+    payload = b""
+    while chunk := os.read(read_fd, 128):
+        payload += chunk
+    os.close(read_fd)
+    _, status = os.waitpid(pid, 0)
+    if status != 0:
+        raise RuntimeError(f"cold rpmsetcmp child failed: status={status}")
+    elapsed, result = map(int, payload.split())
+    if result != 1:
+        raise RuntimeError(f"unexpected cold rpmsetcmp result: {result}")
+    return elapsed
+
+
+def median_cmp_cold(api, provider, requirement, calls, rounds):
+    samples = []
+    for _ in range(rounds):
+        total = sum(cold_cmp_once(api, provider, requirement) for _ in range(calls))
+        samples.append(total / calls)
+    return statistics.median(samples)
+
+
+def format_time(ns):
+    return f"{ns / 1000:.2f} us"
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Compare set9 and CRoaring set APIs")
+    parser.add_argument("--symbols", type=int, default=1000)
+    parser.add_argument("--bpp", type=int, default=32)
+    parser.add_argument("--rounds", type=int, default=7)
+    parser.add_argument("--fini-calls", type=int, default=5)
+    parser.add_argument("--cmp-calls", type=int, default=2000)
+    parser.add_argument("--cold-calls", type=int, default=20)
+    parser.add_argument("--skip-build", action="store_true")
+    args = parser.parse_args()
+    if args.symbols < 2 or not 10 <= args.bpp <= 32:
+        parser.error("symbols must be >= 2 and bpp must be in 10..32")
+    if min(args.rounds, args.fini_calls, args.cmp_calls, args.cold_calls) < 1:
+        parser.error("rounds and call counts must be positive")
+
+    if not args.skip_build:
+        subprocess.run([str(HERE / "build.sh")], check=True)
+
+    symbols = tuple(
+        f"symbol_{i:08d}_version_ALT_{i % 97}".encode() for i in range(args.symbols)
+    )
+    required = symbols[::2]
+    apis = {
+        "set9": SetAPI(BUILD / "libset9.so", result_needs_free=True),
+        "bitmap": SetAPI(BUILD / "libbitmap-set.so", result_needs_free=False),
+    }
+
+    gc.disable()
+    try:
+        timings = {}
+        lengths = {}
+        for name, api in apis.items():
+            provider = api.encode(symbols, args.bpp)
+            requirement = api.encode(required, args.bpp)
+            wire_format = provider[:2].decode() if provider.startswith(b"R1") else "golomb"
+            lengths[name] = (len(provider), wire_format)
+            timings[name] = (
+                median_fini(api, symbols, args.bpp, args.fini_calls, args.rounds),
+                median_build(api, symbols, args.bpp, args.fini_calls, args.rounds),
+                median_cmp_cold(api, provider, requirement, args.cold_calls, args.rounds),
+                median_cmp(api, provider, requirement, args.cmp_calls, args.rounds),
+            )
+    finally:
+        gc.enable()
+
+    print(f"symbols={args.symbols} required={len(required)} bpp={args.bpp}")
+    print("implementation  set_chars  format")
+    for name in apis:
+        print(f"{name:<14} {lengths[name][0]:>9}  {lengths[name][1]}")
+    print("\noperation                 set9       bitmap   bitmap/set9")
+    labels = ("set_fini only", "new+add+fini", "rpmsetcmp cold", "rpmsetcmp warm")
+    for index, label in enumerate(labels):
+        old = timings["set9"][index]
+        new = timings["bitmap"][index]
+        print(f"{label:<22} {format_time(old):>10} {format_time(new):>10} {new / old:>12.2f}x")
+
+
+if __name__ == "__main__":
+    main()

new_version/roaring_bitmap/bitmap_set.c

@@ -0,0 +1,290 @@
+#include <roaring/roaring.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+/*
+ * This is intentionally a new set-string format.  It is not compatible with
+ * the Rice-Golomb/base62 strings produced by the original lib/set.c.
+ *
+ *     R1<two decimal bpp digits><hex CRoaring portable serialization>
+ */
+#define FORMAT_PREFIX "R1"
+#define FORMAT_HEADER_LEN 4
+#define MAX_SERIALIZED_SIZE (64u * 1024u * 1024u)
+
+struct set {
+  roaring_bitmap_t* hashes;
+  size_t added;
+  char* encoded;
+};
+
+static void* xmalloc(size_t size) {
+  void* ptr = malloc(size);
+  if (!ptr) abort();
+  return ptr;
+}
+
+static uint32_t hash_symbol(const char* str) {
+  uint32_t hash = UINT32_C(0x9e3779b9);
+  const unsigned char* p = (const unsigned char*)str;
+
+  while (*p) {
+    hash += *p++;
+    hash += hash << 10;
+    hash ^= hash >> 6;
+  }
+
+  hash += hash << 3;
+  hash ^= hash >> 11;
+  hash += hash << 15;
+  return hash;
+}
+
+struct set* set_new(void) {
+  struct set* set = xmalloc(sizeof(*set));
+  set->hashes = roaring_bitmap_create();
+  if (!set->hashes) abort();
+  set->added = 0;
+  set->encoded = NULL;
+  return set;
+}
+
+void set_add(struct set* set, const char* sym) {
+  if (!set || !sym) return;
+  roaring_bitmap_add(set->hashes, hash_symbol(sym));
+  set->added++;
+}
+
+static roaring_bitmap_t* truncate_bitmap(const roaring_bitmap_t* source, unsigned bpp) {
+  roaring_bitmap_t* result;
+  roaring_uint32_iterator_t iterator;
+  uint32_t mask;
+
+  if (bpp == 32) {
+    result = roaring_bitmap_copy(source);
+    if (!result) abort();
+    return result;
+  }
+
+  mask = (UINT32_C(1) << bpp) - 1;
+  result = roaring_bitmap_create();
+  if (!result) abort();
+
+  roaring_iterator_init(source, &iterator);
+  while (iterator.has_value) {
+    roaring_bitmap_add(result, iterator.current_value & mask);
+    roaring_uint32_iterator_advance(&iterator);
+  }
+
+  return result;
+}
+
+static char hex_digit(unsigned value) {
+  return (char)(value < 10 ? '0' + value : 'a' + value - 10);
+}
+
+const char* set_fini(struct set* set, int bpp) {
+  roaring_bitmap_t* truncated;
+  size_t portable_size;
+  size_t written;
+  unsigned char* portable;
+  char* output;
+
+  if (!set || set->added == 0 || bpp < 10 || bpp > 32) return NULL;
+
+  truncated = truncate_bitmap(set->hashes, (unsigned)bpp);
+  if (roaring_bitmap_get_cardinality(truncated) < set->added)
+    fprintf(stderr, "warning: hash collision\n");
+
+  roaring_bitmap_run_optimize(truncated);
+  portable_size = roaring_bitmap_portable_size_in_bytes(truncated);
+  if (portable_size > MAX_SERIALIZED_SIZE) {
+    roaring_bitmap_free(truncated);
+    return NULL;
+  }
+
+  portable = xmalloc(portable_size);
+  written = roaring_bitmap_portable_serialize(truncated, (char*)portable);
+  if (written != portable_size) abort();
+  if (portable_size > (SIZE_MAX - FORMAT_HEADER_LEN - 1) / 2) abort();
+
+  output = xmalloc(FORMAT_HEADER_LEN + portable_size * 2 + 1);
+  memcpy(output, FORMAT_PREFIX, sizeof(FORMAT_PREFIX) - 1);
+  output[2] = (char)('0' + bpp / 10);
+  output[3] = (char)('0' + bpp % 10);
+
+  for (size_t i = 0; i < portable_size; ++i) {
+    output[FORMAT_HEADER_LEN + i * 2] = hex_digit(portable[i] >> 4);
+    output[FORMAT_HEADER_LEN + i * 2 + 1] = hex_digit(portable[i] & 0x0f);
+  }
+  output[FORMAT_HEADER_LEN + portable_size * 2] = '\0';
+
+  free(portable);
+  roaring_bitmap_free(truncated);
+  free(set->encoded);
+  set->encoded = output;
+  return set->encoded;
+}
+
+struct set* set_free(struct set* set) {
+  if (set) {
+    roaring_bitmap_free(set->hashes);
+    free(set->encoded);
+    free(set);
+  }
+  return NULL;
+}
+
+static int hex_value(char c) {
+  if (c >= '0' && c <= '9') return c - '0';
+  if (c >= 'a' && c <= 'f') return c - 'a' + 10;
+  if (c >= 'A' && c <= 'F') return c - 'A' + 10;
+  return -1;
+}
+
+static roaring_bitmap_t* decode_bitmap(const char* str, unsigned* bpp) {
+  roaring_bitmap_t* bitmap;
+  unsigned char* bytes;
+  const char* hex;
+  const char* reason = NULL;
+  size_t hex_len;
+  size_t str_len;
+  size_t byte_count;
+  size_t expected;
+
+  if (!str || !bpp) return NULL;
+  if (strncmp(str, "set:", 4) == 0) str += 4;
+  str_len = strlen(str);
+  if (str_len < FORMAT_HEADER_LEN) return NULL;
+  if (strncmp(str, FORMAT_PREFIX, sizeof(FORMAT_PREFIX) - 1) != 0) return NULL;
+  if (str[2] < '0' || str[2] > '9' || str[3] < '0' || str[3] > '9') return NULL;
+
+  *bpp = (unsigned)(str[2] - '0') * 10u + (unsigned)(str[3] - '0');
+  if (*bpp < 10 || *bpp > 32) return NULL;
+
+  hex = str + FORMAT_HEADER_LEN;
+  hex_len = str_len - FORMAT_HEADER_LEN;
+  if (hex_len == 0 || (hex_len & 1u) != 0) return NULL;
+  byte_count = hex_len / 2;
+  if (byte_count > MAX_SERIALIZED_SIZE) return NULL;
+  bytes = xmalloc(byte_count);
+
+  for (size_t i = 0; i < byte_count; ++i) {
+    int high = hex_value(hex[i * 2]);
+    int low = hex_value(hex[i * 2 + 1]);
+    if (high < 0 || low < 0) {
+      free(bytes);
+      return NULL;
+    }
+    bytes[i] = (unsigned char)((high << 4) | low);
+  }
+
+  expected = roaring_bitmap_portable_deserialize_size((const char*)bytes, byte_count);
+  if (expected != byte_count) {
+    free(bytes);
+    return NULL;
+  }
+
+  bitmap = roaring_bitmap_portable_deserialize_safe((const char*)bytes, byte_count);
+  free(bytes);
+  if (!bitmap) return NULL;
+  if (!roaring_bitmap_internal_validate(bitmap, &reason) || roaring_bitmap_is_empty(bitmap)) {
+    roaring_bitmap_free(bitmap);
+    return NULL;
+  }
+
+  return bitmap;
+}
+
+int rpmsetcmp(const char* set1, const char* set2) {
+  roaring_bitmap_t* bitmap1;
+  roaring_bitmap_t* bitmap2;
+  unsigned bpp1;
+  unsigned bpp2;
+  unsigned common_bpp;
+  bool one_in_two;
+  bool two_in_one;
+
+  bitmap1 = decode_bitmap(set1, &bpp1);
+  if (!bitmap1) return -3;
+  bitmap2 = decode_bitmap(set2, &bpp2);
+  if (!bitmap2) {
+    roaring_bitmap_free(bitmap1);
+    return -4;
+  }
+
+  common_bpp = bpp1 < bpp2 ? bpp1 : bpp2;
+  if (bpp1 != common_bpp) {
+    roaring_bitmap_t* truncated = truncate_bitmap(bitmap1, common_bpp);
+    roaring_bitmap_free(bitmap1);
+    bitmap1 = truncated;
+  }
+  if (bpp2 != common_bpp) {
+    roaring_bitmap_t* truncated = truncate_bitmap(bitmap2, common_bpp);
+    roaring_bitmap_free(bitmap2);
+    bitmap2 = truncated;
+  }
+
+  one_in_two = roaring_bitmap_is_subset(bitmap1, bitmap2);
+  two_in_one = roaring_bitmap_is_subset(bitmap2, bitmap1);
+  roaring_bitmap_free(bitmap1);
+  roaring_bitmap_free(bitmap2);
+
+  if (one_in_two && two_in_one) return 0;
+  if (two_in_one) return 1;
+  if (one_in_two) return -1;
+  return -2;
+}
+
+#ifdef SELF_TEST
+#include <assert.h>
+
+static char* make_set(const char* const* symbols, size_t count, int bpp) {
+  struct set* set = set_new();
+  const char* encoded;
+  char* copy;
+
+  for (size_t i = 0; i < count; ++i) set_add(set, symbols[i]);
+  encoded = set_fini(set, bpp);
+  assert(encoded);
+  copy = xmalloc(strlen(encoded) + 1);
+  strcpy(copy, encoded);
+  set = set_free(set);
+  assert(!set);
+  return copy;
+}
+
+int main(void) {
+  static const char* const small[] = {"malloc", "printf"};
+  static const char* const large[] = {"free", "malloc", "printf"};
+  static const char* const other[] = {"calloc", "malloc"};
+  char* small16 = make_set(small, 2, 16);
+  char* small32 = make_set(small, 2, 32);
+  char* large16 = make_set(large, 3, 16);
+  char* other16 = make_set(other, 2, 16);
+  char* prefixed = xmalloc(strlen(small16) + 5);
+
+  sprintf(prefixed, "set:%s", small16);
+  assert(rpmsetcmp(small16, small16) == 0);
+  assert(rpmsetcmp(prefixed, small16) == 0);
+  assert(rpmsetcmp(small32, small16) == 0);
+  assert(rpmsetcmp(large16, small16) == 1);
+  assert(rpmsetcmp(small16, large16) == -1);
+  assert(rpmsetcmp(small16, other16) == -2);
+  assert(rpmsetcmp("bad", small16) == -3);
+  assert(rpmsetcmp("R21600", small16) == -3);
+  assert(rpmsetcmp(small16, "R116xyz") == -4);
+  assert(rpmsetcmp("R", small16) == -3);
+
+  free(prefixed);
+  free(other16);
+  free(large16);
+  free(small32);
+  free(small16);
+  puts("bitmap_set self-test: OK");
+  return 0;
+}
+#endif

new_version/roaring_bitmap/build.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+HERE=$(cd "$(dirname "$0")" && pwd)
+ROOT=$(cd "$HERE/../.." && pwd)
+BUILD="$HERE/build"
+mkdir -p "$BUILD"
+touch "$BUILD/rpmlib.h" "$BUILD/system.h"
+
+if [[ -z ${ROARING_CFLAGS+x} || -z ${ROARING_LIBS+x} ]]; then
+  if pkg-config --exists roaring; then
+    ROARING_CFLAGS=$(pkg-config --cflags roaring)
+    ROARING_LIBS=$(pkg-config --libs roaring)
+  else
+    CROARING_SRC="$BUILD/CRoaring"
+    CROARING_BUILD="$BUILD/CRoaring-pic-build"
+    if [[ ! -d $CROARING_SRC/.git ]]; then
+      rm -rf "$CROARING_SRC"
+      git clone --depth 1 https://github.com/RoaringBitmap/CRoaring.git "$CROARING_SRC"
+    fi
+    if [[ ! -f $CROARING_BUILD/src/libroaring.a ]]; then
+      cmake -S "$CROARING_SRC" -B "$CROARING_BUILD" \
+        -DROARING_BUILD_STATIC=ON -DENABLE_ROARING_TESTS=OFF \
+        -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON
+      cmake --build "$CROARING_BUILD" --parallel
+    fi
+    ROARING_CFLAGS="-I$CROARING_SRC/include"
+    ROARING_LIBS="$CROARING_BUILD/src/libroaring.a"
+  fi
+fi
+read -r -a ROARING_CFLAGS_A <<<"$ROARING_CFLAGS"
+read -r -a ROARING_LIBS_A <<<"$ROARING_LIBS"
+CFLAGS=(-O2 -std=gnu11 -D_GNU_SOURCE -Wall -Wextra -I"$HERE" -I"$BUILD")
+
+for tool in mkset setcmp; do
+  cc "${CFLAGS[@]}" -include "$ROOT/scripts/rpmsetcmp/newset_compat.h" \
+    "$ROOT/reimplement/set9.c" "$ROOT/scripts/rpmsetcmp/$tool.c" \
+    -o "$BUILD/$tool-set9"
+  cc "${CFLAGS[@]}" "${ROARING_CFLAGS_A[@]}" \
+    "$HERE/bitmap_set.c" "$ROOT/scripts/rpmsetcmp/$tool.c" \
+    "${ROARING_LIBS_A[@]}" -o "$BUILD/$tool-bitmap"
+done
+
+cc "${CFLAGS[@]}" -fPIC -shared -include "$ROOT/scripts/rpmsetcmp/newset_compat.h" \
+  "$ROOT/reimplement/set9.c" -o "$BUILD/libset9.so"
+cc "${CFLAGS[@]}" -fPIC -shared "${ROARING_CFLAGS_A[@]}" \
+  "$HERE/bitmap_set.c" "${ROARING_LIBS_A[@]}" -o "$BUILD/libbitmap-set.so"
+
+printf 'Built tools and benchmark libraries in %s\n' "$BUILD"

new_version/roaring_bitmap/set.h

@@ -0,0 +1,12 @@
+#ifndef ARSV_SET_H
+#define ARSV_SET_H
+
+struct set;
+
+int rpmsetcmp(const char *set1, const char *set2);
+struct set *set_new(void);
+void set_add(struct set *set, const char *sym);
+const char *set_fini(struct set *set, int bpp);
+struct set *set_free(struct set *set);
+
+#endif

other/extract_alt_set_symbols.py

@@ -1,384 +0,0 @@
-#!/usr/bin/env python3
-"""Extract ALT Linux set:version strings and optional ELF nm symbol stats.
-
-The script uses the public rdb.altlinux.org API to read package dependency
-metadata. With --download-nm it also downloads binary RPMs, extracts ELF files
-with bsdtar, and runs nm on dynamic symbols.
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import re
-import shutil
-import statistics
-import subprocess
-import sys
-import tempfile
-import urllib.error
-import urllib.parse
-import urllib.request
-from collections import Counter
-from dataclasses import dataclass
-from pathlib import Path
-from typing import Iterable
-
-API_BASE = "https://rdb.altlinux.org/api"
-DEFAULT_PACKAGES = [
-    "glibc-core",
-    "libcrypto3",
-    "libssl3",
-    "zlib",
-    "libgcc1",
-    "libcurl",
-    "libsystemd",
-    "libqt6-core",
-    "libgtk+3",
-    "libsqlite3",
-    "libxml2",
-    "libX11",
-    "libxcb",
-    "coreutils",
-    "curl",
-    "openssl",
-    "systemd",
-    "python3-base",
-]
-
-SET_PREFIX = "set:"
-SET_ALPHABET = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
-IDENTISH_SYMBOL_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*(?:@@?[A-Za-z0-9_.]+)?$")
-
-
-@dataclass(frozen=True)
-class SetDependency:
-    package: str
-    dep_type: str
-    name: str
-    set_string: str
-
-    @property
-    def payload(self) -> str:
-        return self.set_string[len(SET_PREFIX) :]
-
-    @property
-    def bpp(self) -> int | None:
-        return set_char_to_int(self.payload[0]) if self.payload else None
-
-    @property
-    def mshift(self) -> int | None:
-        return set_char_to_int(self.payload[1]) if len(self.payload) > 1 else None
-
-
-@dataclass
-class NmReport:
-    package: str
-    mode: str
-    files_seen: int
-    symbols: list[str]
-    file_examples: list[str]
-
-
-def set_char_to_int(char: str) -> int | None:
-    try:
-        return SET_ALPHABET.index(char)
-    except ValueError:
-        return None
-
-
-def api_json(path: str, params: dict[str, object] | None = None) -> dict:
-    url = API_BASE + path
-    if params:
-        url += "?" + urllib.parse.urlencode(params, doseq=True)
-    req = urllib.request.Request(url, headers={"User-Agent": "arsv-set-symbol-extractor/1.0"})
-    with urllib.request.urlopen(req, timeout=60) as response:
-        return json.load(response)
-
-
-def get_pkghash(package: str, branch: str, arch: str) -> str:
-    data = api_json(
-        "/site/pkghash_by_binary_name",
-        {"branch": branch, "name": package, "arch": arch},
-    )
-    return str(data["pkghash"])
-
-
-def get_set_dependencies(package: str, branch: str, arch: str) -> tuple[str, list[SetDependency]]:
-    pkghash = get_pkghash(package, branch, arch)
-    deps = api_json(f"/dependencies/binary_package_dependencies/{pkghash}")["dependencies"]
-    set_deps = []
-    for dep in deps:
-        version = dep.get("version") or ""
-        if version.startswith(SET_PREFIX):
-            set_deps.append(
-                SetDependency(
-                    package=package,
-                    dep_type=dep.get("type", ""),
-                    name=dep.get("name", ""),
-                    set_string=version,
-                )
-            )
-    return pkghash, set_deps
-
-
-def package_download_url(pkghash: str, branch: str, arch: str) -> str:
-    data = api_json(
-        f"/site/package_downloads_bin/{pkghash}",
-        {"branch": branch, "arch": arch},
-    )
-    downloads = data.get("downloads") or []
-    if not downloads or not downloads[0].get("packages"):
-        raise RuntimeError(f"no download URL for pkghash={pkghash}")
-    return downloads[0]["packages"][0]["url"]
-
-
-def download_file(url: str, destination: Path) -> None:
-    req = urllib.request.Request(url, headers={"User-Agent": "arsv-set-symbol-extractor/1.0"})
-    with urllib.request.urlopen(req, timeout=120) as response, destination.open("wb") as out:
-        shutil.copyfileobj(response, out)
-
-
-def run_text(command: list[str], cwd: Path | None = None, check: bool = True) -> str:
-    proc = subprocess.run(command, cwd=cwd, text=True, capture_output=True)
-    if check and proc.returncode != 0:
-        joined = " ".join(command)
-        raise RuntimeError(f"{joined} failed with {proc.returncode}: {proc.stderr.strip()}")
-    return proc.stdout
-
-
-def rpm_members(rpm_path: Path) -> list[str]:
-    return [line for line in run_text(["bsdtar", "-tf", str(rpm_path)]).splitlines() if line]
-
-
-def select_library_members(members: Iterable[str], max_files: int) -> list[str]:
-    selected = [
-        member
-        for member in members
-        if not member.endswith("/") and re.search(r"(^|/)lib[^/]*\.so(?:\.|$)", member)
-    ]
-    return selected[:max_files]
-
-
-def select_executable_members(members: Iterable[str], max_files: int) -> list[str]:
-    prefixes = ("./bin/", "./usr/bin/", "./sbin/", "./usr/sbin/", "./usr/lib/systemd/")
-    selected = [member for member in members if not member.endswith("/") and member.startswith(prefixes)]
-    return selected[:max_files]
-
-
-def nm_symbols(path: Path, mode: str) -> list[str]:
-    flag = "-U" if mode == "defined" else "-u"
-    proc = subprocess.run(
-        ["nm", "--dynamic", "-j", flag, str(path)],
-        text=True,
-        capture_output=True,
-    )
-    if proc.returncode != 0:
-        return []
-    return [line.strip() for line in proc.stdout.splitlines() if line.strip()]
-
-
-def extract_nm_report(
-    package: str,
-    pkghash: str,
-    branch: str,
-    arch: str,
-    mode: str,
-    max_files: int,
-    workdir: Path,
-) -> NmReport:
-    rpm_url = package_download_url(pkghash, branch, arch)
-    rpm_path = workdir / Path(urllib.parse.urlparse(rpm_url).path).name
-    download_file(rpm_url, rpm_path)
-    members = rpm_members(rpm_path)
-    selected = (
-        select_library_members(members, max_files)
-        if mode == "defined"
-        else select_executable_members(members, max_files)
-    )
-    extract_dir = workdir / f"extract-{package.replace('/', '_').replace('+', '_')}-{mode}"
-    extract_dir.mkdir(parents=True, exist_ok=True)
-    if selected:
-        run_text(["bsdtar", "-xf", str(rpm_path), "-C", str(extract_dir), *selected])
-
-    symbols: list[str] = []
-    for member in selected:
-        member_path = extract_dir / member
-        if member_path.exists():
-            symbols.extend(nm_symbols(member_path, mode))
-
-    return NmReport(
-        package=package,
-        mode=mode,
-        files_seen=len(selected),
-        symbols=symbols,
-        file_examples=selected[:3],
-    )
-
-
-def char_classes(chars: Iterable[str]) -> dict[str, object]:
-    char_set = sorted(set(chars))
-    return {
-        "alphabet": "".join(char_set),
-        "upper": sum(ch.isupper() for ch in char_set),
-        "lower": sum(ch.islower() for ch in char_set),
-        "digit": sum(ch.isdigit() for ch in char_set),
-        "underscore": "_" in char_set,
-        "at": "@" in char_set,
-        "dot": "." in char_set,
-        "dollar": "$" in char_set,
-        "other": "".join(ch for ch in char_set if not (ch.isalnum() or ch in "_.@$")),
-    }
-
-
-def print_set_report(set_deps: list[SetDependency], prefix_len: int) -> None:
-    print("# set:version dependency strings")
-    if not set_deps:
-        print("No set: dependencies found.")
-        return
-
-    lengths = [len(dep.payload) for dep in set_deps]
-    alphabet = "".join(sorted(set("".join(dep.payload for dep in set_deps))))
-    print(f"count: {len(set_deps)}")
-    print(f"payload length min/median/max: {min(lengths)}/{statistics.median(lengths)}/{max(lengths)}")
-    print(f"observed encoded alphabet: {alphabet}")
-    print(f"bpp counts: {dict(sorted(Counter(dep.bpp for dep in set_deps).items()))}")
-    print(f"Mshift counts: {dict(sorted(Counter(dep.mshift for dep in set_deps).items()))}")
-    print()
-    print("package\ttype\tbpp\tMshift\tlen\tdependency\tset-prefix")
-    for dep in sorted(set_deps, key=lambda item: (item.package, item.dep_type, item.name)):
-        print(
-            f"{dep.package}\t{dep.dep_type}\t{dep.bpp}\t{dep.mshift}\t"
-            f"{len(dep.payload)}\t{dep.name}\t{dep.payload[:prefix_len]}"
-        )
-
-
-def print_nm_report(reports: list[NmReport], sample_limit: int) -> None:
-    print("\n# nm dynamic symbol strings")
-    if not reports:
-        print("Skipped. Pass --download-nm to download RPMs and run bsdtar/nm.")
-        return
-
-    for mode in ("defined", "undefined"):
-        mode_reports = [report for report in reports if report.mode == mode]
-        if not mode_reports:
-            continue
-        all_symbols = [symbol for report in mode_reports for symbol in report.symbols]
-        print(f"\n## {mode} symbols")
-        print(f"packages: {len(mode_reports)}")
-        print(f"symbols total/unique: {len(all_symbols)}/{len(set(all_symbols))}")
-        if all_symbols:
-            print(f"max symbol length: {max(len(symbol) for symbol in all_symbols)}")
-            print(f"character classes: {char_classes(''.join(all_symbols))}")
-            odd = sorted({symbol for symbol in all_symbols if not IDENTISH_SYMBOL_RE.match(symbol)})
-            mangled = sorted({symbol for symbol in all_symbols if symbol.startswith("_Z")})
-            print(f"non identifier-ish examples: {odd[:sample_limit]}")
-            print(f"C++ mangled examples: {mangled[:sample_limit]}")
-        print("package\tfiles\tsymbols\tfile-examples\tsymbol-examples")
-        for report in mode_reports:
-            print(
-                f"{report.package}\t{report.files_seen}\t{len(report.symbols)}\t"
-                f"{', '.join(report.file_examples)}\t{', '.join(report.symbols[:sample_limit])}"
-            )
-
-
-def parse_args(argv: list[str]) -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Print ALT package set:version strings and optional nm symbol statistics."
-    )
-    parser.add_argument("packages", nargs="*", default=DEFAULT_PACKAGES, help="binary package names")
-    parser.add_argument("--branch", default="sisyphus", help="ALT repository branch/packageset")
-    parser.add_argument("--arch", default="x86_64", help="binary package architecture")
-    parser.add_argument("--download-nm", action="store_true", help="download RPMs and run nm on ELF files")
-    parser.add_argument("--nm-defined", action="store_true", help="with --download-nm, inspect defined symbols from libraries")
-    parser.add_argument("--nm-undefined", action="store_true", help="with --download-nm, inspect undefined symbols from executables")
-    parser.add_argument("--max-libs", type=int, default=8, help="max library files per package for defined-symbol nm")
-    parser.add_argument("--max-bins", type=int, default=20, help="max executable files per package for undefined-symbol nm")
-    parser.add_argument("--prefix-len", type=int, default=48, help="number of set payload chars to print per row")
-    parser.add_argument("--sample-limit", type=int, default=12, help="number of symbol examples to print")
-    return parser.parse_args(argv)
-
-
-def main(argv: list[str]) -> int:
-    args = parse_args(argv)
-    if args.download_nm and not args.nm_defined and not args.nm_undefined:
-        args.nm_defined = True
-        args.nm_undefined = True
-
-    if args.download_nm:
-        missing = [cmd for cmd in ("bsdtar", "nm") if shutil.which(cmd) is None]
-        if missing:
-            print(f"missing required command(s) for --download-nm: {', '.join(missing)}", file=sys.stderr)
-            return 2
-
-    package_hashes: dict[str, str] = {}
-    set_deps: list[SetDependency] = []
-    errors: list[str] = []
-
-    for package in args.packages:
-        try:
-            pkghash, deps = get_set_dependencies(package, args.branch, args.arch)
-            package_hashes[package] = pkghash
-            set_deps.extend(deps)
-        except (urllib.error.URLError, urllib.error.HTTPError, KeyError, RuntimeError) as exc:
-            errors.append(f"{package}: {exc}")
-
-    print(f"branch: {args.branch}")
-    print(f"arch: {args.arch}")
-    print(f"packages requested: {', '.join(args.packages)}")
-    if errors:
-        print("\n# lookup errors", file=sys.stderr)
-        for error in errors:
-            print(error, file=sys.stderr)
-    print()
-    print_set_report(set_deps, args.prefix_len)
-
-    nm_reports: list[NmReport] = []
-    if args.download_nm:
-        with tempfile.TemporaryDirectory(prefix="arsv-alt-rpms-") as tmp:
-            workdir = Path(tmp)
-            for package, pkghash in package_hashes.items():
-                if args.nm_defined:
-                    try:
-                        nm_reports.append(
-                            extract_nm_report(
-                                package,
-                                pkghash,
-                                args.branch,
-                                args.arch,
-                                "defined",
-                                args.max_libs,
-                                workdir,
-                            )
-                        )
-                    except Exception as exc:  # keep processing other packages
-                        errors.append(f"{package} defined nm: {exc}")
-                if args.nm_undefined:
-                    try:
-                        nm_reports.append(
-                            extract_nm_report(
-                                package,
-                                pkghash,
-                                args.branch,
-                                args.arch,
-                                "undefined",
-                                args.max_bins,
-                                workdir,
-                            )
-                        )
-                    except Exception as exc:  # keep processing other packages
-                        errors.append(f"{package} undefined nm: {exc}")
-
-    print_nm_report(nm_reports, args.sample_limit)
-
-    if errors:
-        print("\n# errors", file=sys.stderr)
-        for error in errors:
-            print(error, file=sys.stderr)
-        return 1
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main(sys.argv[1:]))

scripts/old/check_alt_set_impl.py

@@ -1,429 +0,0 @@
-#!/usr/bin/env python3
-"""Check package compatibility using this repo's Python set implementation.
-
-This script intentionally does *not* compare with ALT's existing set.c-produced
-set strings. It uses ALT only as a source of real binary RPMs:
-
-* Provided labels:  `nm --dynamic -j -U <shared-library>`
-* Required labels:  `nm --dynamic -j -u <elf-file>`
-
-Both sides are encoded with `reimplement/set.py`, then compared with that same
-implementation's `rpmsetcmp()`. In other words, it checks whether the current
-Python implementation is internally useful for real ALT package symbol labels.
-"""
-
-from __future__ import annotations
-
-import argparse
-import contextlib
-import json
-import re
-import shutil
-import subprocess
-import sys
-import tempfile
-import urllib.parse
-import urllib.request
-from dataclasses import dataclass
-from pathlib import Path
-from typing import Iterable
-
-REPO_ROOT = Path(__file__).resolve().parents[1]
-if str(REPO_ROOT) not in sys.path:
-    sys.path.insert(0, str(REPO_ROOT))
-
-from reimplement import set as rpmset  # noqa: E402
-
-API_BASE = "https://rdb.altlinux.org/api"
-DEFAULT_PROVIDERS = ["glibc-core", "zlib", "libssl3", "libcrypto3"]
-DEFAULT_REQUIRERS = ["coreutils", "curl", "openssl"]
-
-
-@dataclass(frozen=True)
-class PackageRPM:
-    name: str
-    pkghash: str
-    rpm_path: Path
-    extract_dir: Path
-    members: list[str]
-
-
-@dataclass(frozen=True)
-class LabelSet:
-    role: str
-    package: str
-    member: str
-    labels: tuple[str, ...]
-    set_string: str
-
-    @property
-    def label_count(self) -> int:
-        return len(self.labels)
-
-    @property
-    def set_len(self) -> int:
-        return len(self.set_string)
-
-
-@dataclass(frozen=True)
-class CompatibilityResult:
-    provider_package: str
-    provider_member: str
-    requirer_package: str
-    requirer_member: str
-    provider_labels: int
-    required_labels: int
-    cmp_result: int
-    status: str
-
-
-def api_json(path: str, params: dict[str, object] | None = None) -> dict:
-    url = API_BASE + path
-    if params:
-        url += "?" + urllib.parse.urlencode(params, doseq=True)
-    req = urllib.request.Request(url, headers={"User-Agent": "arsv-alt-set-compat/1.0"})
-    with urllib.request.urlopen(req, timeout=60) as response:
-        return json.load(response)
-
-
-def get_pkghash(package: str, branch: str, arch: str) -> str:
-    data = api_json(
-        "/site/pkghash_by_binary_name",
-        {"branch": branch, "name": package, "arch": arch},
-    )
-    return str(data["pkghash"])
-
-
-def package_download_url(pkghash: str, branch: str, arch: str) -> str:
-    data = api_json(
-        f"/site/package_downloads_bin/{pkghash}",
-        {"branch": branch, "arch": arch},
-    )
-    downloads = data.get("downloads") or []
-    if not downloads or not downloads[0].get("packages"):
-        raise RuntimeError(f"no download URL for pkghash={pkghash}")
-    return downloads[0]["packages"][0]["url"]
-
-
-def run_text(command: list[str], cwd: Path | None = None, check: bool = True) -> str:
-    proc = subprocess.run(command, cwd=cwd, text=True, capture_output=True)
-    if check and proc.returncode != 0:
-        raise RuntimeError(f"{' '.join(command)} failed with {proc.returncode}: {proc.stderr.strip()}")
-    return proc.stdout
-
-
-def download_file(url: str, destination: Path) -> None:
-    req = urllib.request.Request(url, headers={"User-Agent": "arsv-alt-set-compat/1.0"})
-    with urllib.request.urlopen(req, timeout=120) as response, destination.open("wb") as out:
-        shutil.copyfileobj(response, out)
-
-
-def rpm_members(rpm_path: Path) -> list[str]:
-    return [line for line in run_text(["bsdtar", "-tf", str(rpm_path)]).splitlines() if line]
-
-
-def is_shared_library_member(member: str) -> bool:
-    name = Path(member).name
-    return not member.endswith("/") and re.search(r"(?:^|/)lib[^/]*\.so(?:\.|$)", member) is not None and ".debug" not in name
-
-
-def select_provider_members(members: Iterable[str]) -> list[str]:
-    """Files whose defined dynamic symbols are Provided labels."""
-    return [member for member in members if is_shared_library_member(member)]
-
-
-def select_requirer_members(members: Iterable[str]) -> list[str]:
-    """Files whose undefined dynamic symbols are Required labels."""
-    executable_prefixes = ("./bin/", "./usr/bin/", "./sbin/", "./usr/sbin/", "./usr/lib/systemd/")
-    selected = []
-    for member in members:
-        if member.endswith("/"):
-            continue
-        if is_shared_library_member(member) or member.startswith(executable_prefixes):
-            selected.append(member)
-    return selected
-
-
-def extract_members(rpm_path: Path, extract_dir: Path, members: Iterable[str]) -> None:
-    unique_members = sorted(set(members))
-    if unique_members:
-        run_text(["bsdtar", "-xf", str(rpm_path), "-C", str(extract_dir), *unique_members])
-
-
-def nm_symbols(path: Path, mode: str) -> list[str]:
-    command = ["nm", "--dynamic", "-j", "-U", str(path)] if mode == "provided" else ["nm", "--dynamic", "-u", str(path)]
-    proc = subprocess.run(command, text=True, capture_output=True)
-    if proc.returncode != 0:
-        return []
-    if mode == "required":
-        return parse_required_nm_output(proc.stdout)
-    return [line.strip() for line in proc.stdout.splitlines() if line.strip()]
-
-
-def parse_required_nm_output(output: str) -> list[str]:
-    """Parse `nm --dynamic -u` output, ignoring weak undefined references.
-
-    Plain `nm -j -u` discards the symbol type, but real ALT RPMs contain weak
-    undefined hooks like `__gmon_start__` and `_ITM_*`. Those are optional ELF
-    references, not hard Required labels, so keep only strong `U` entries.
-    """
-    symbols = []
-    for line in output.splitlines():
-        parts = line.split()
-        if len(parts) < 2:
-            continue
-        symbol_type, symbol = parts[-2], parts[-1]
-        if symbol_type == "U":
-            symbols.append(symbol)
-    return symbols
-
-
-def normalize_required_symbol(symbol: str) -> str:
-    """Normalize `foo@VER` from undefined nm output to provider-like `foo@@VER`.
-
-    `nm -u` prints required versioned symbols with a single `@`, while defined
-    default-version symbols commonly use `@@`. This normalization is for this
-    script's compatibility model only; it is not a set.c compatibility shim.
-    """
-    if "@@" in symbol or "@" not in symbol:
-        return symbol
-    name, version = symbol.split("@", 1)
-    if not name or not version:
-        return symbol
-    return f"{name}@@{version}"
-
-
-def labels_to_set_string(labels: Iterable[str], bpp: int) -> str | None:
-    item_set = rpmset.set_new()
-    for label in labels:
-        rpmset.set_add(item_set, label)
-    # reimplement/set.py prints collision warnings to stdout; keep this script's
-    # stdout machine-readable and route those warnings to stderr instead.
-    with contextlib.redirect_stdout(sys.stderr):
-        return rpmset.set_fini(item_set, bpp)
-
-
-def generate_label_set(role: str, member: str, labels: Iterable[str], bpp: int, package: str = "") -> LabelSet:
-    unique_labels = tuple(sorted(set(label for label in labels if label)))
-    set_string = labels_to_set_string(unique_labels, bpp)
-    if set_string is None:
-        raise ValueError(f"no labels for {role} {package}:{member}")
-    return LabelSet(role=role, package=package, member=member, labels=unique_labels, set_string=set_string)
-
-
-def compare_status(cmp_result: int) -> str:
-    # provider is first argument; compatible means provider is equal or superset.
-    if cmp_result in (0, 1):
-        return "compatible"
-    return "incompatible"
-
-
-def compare_label_sets(provider: LabelSet, requirer: LabelSet) -> CompatibilityResult:
-    cmp_result = rpmset.rpmsetcmp(provider.set_string, requirer.set_string)
-    return CompatibilityResult(
-        provider_package=provider.package,
-        provider_member=provider.member,
-        requirer_package=requirer.package,
-        requirer_member=requirer.member,
-        provider_labels=provider.label_count,
-        required_labels=requirer.label_count,
-        cmp_result=cmp_result,
-        status=compare_status(cmp_result),
-    )
-
-
-def build_dependency_results(provider_sets: list[LabelSet], requirer_sets: list[LabelSet], bpp: int) -> list[CompatibilityResult]:
-    """Compare each library only with the symbols actually required from it.
-
-    A package executable/library has one undefined-symbol list containing symbols
-    required from all of its DT_NEEDED libraries. Comparing that whole list with
-    one provider library gives false ``-2`` results: e.g. ``/usr/bin/curl`` needs
-    symbols from libc, libssl, libcrypto, zlib, etc., and no single library is
-    supposed to provide all of them.
-
-    For this local set.py check, keep the symbol-level ground truth around and
-    split every requirer's labels by provider library: provider labels ∩ required
-    labels. Each non-empty subset is then encoded as the requirement for exactly
-    that provider and compared with ``rpmsetcmp(provider, required_subset)``.
-    """
-    results: list[CompatibilityResult] = []
-    for requirer in requirer_sets:
-        required_labels = set(requirer.labels)
-        for provider in provider_sets:
-            required_from_provider = sorted(required_labels.intersection(provider.labels))
-            if not required_from_provider:
-                continue
-            split_requirer = generate_label_set(
-                "required",
-                requirer.member,
-                required_from_provider,
-                bpp,
-                package=requirer.package,
-            )
-            results.append(compare_label_sets(provider, split_requirer))
-    return results
-
-
-def fetch_package_rpm(package: str, branch: str, arch: str, workdir: Path) -> PackageRPM:
-    pkghash = get_pkghash(package, branch, arch)
-    rpm_url = package_download_url(pkghash, branch, arch)
-    rpm_path = workdir / Path(urllib.parse.urlparse(rpm_url).path).name
-    if not rpm_path.exists():
-        download_file(rpm_url, rpm_path)
-    members = rpm_members(rpm_path)
-    extract_dir = workdir / f"extract-{package.replace('/', '_').replace('+', '_')}"
-    extract_dir.mkdir(parents=True, exist_ok=True)
-    return PackageRPM(package, pkghash, rpm_path, extract_dir, members)
-
-
-def build_provider_sets(package_rpm: PackageRPM, bpp: int, max_files: int) -> list[LabelSet]:
-    members = select_provider_members(package_rpm.members)[:max_files]
-    extract_members(package_rpm.rpm_path, package_rpm.extract_dir, members)
-    sets = []
-    for member in members:
-        labels = nm_symbols(package_rpm.extract_dir / member, "provided")
-        if labels:
-            sets.append(generate_label_set("provided", member, labels, bpp, package_rpm.name))
-    return sets
-
-
-def build_requirer_sets(
-    package_rpm: PackageRPM,
-    bpp: int,
-    max_files: int,
-    normalize_versions: bool,
-) -> list[LabelSet]:
-    members = select_requirer_members(package_rpm.members)[:max_files]
-    extract_members(package_rpm.rpm_path, package_rpm.extract_dir, members)
-    sets = []
-    for member in members:
-        labels = nm_symbols(package_rpm.extract_dir / member, "required")
-        if normalize_versions:
-            labels = [normalize_required_symbol(label) for label in labels]
-        if labels:
-            sets.append(generate_label_set("required", member, labels, bpp, package_rpm.name))
-    return sets
-
-
-def aggregate_label_sets(role: str, package_names: list[str], label_sets: list[LabelSet], bpp: int) -> LabelSet:
-    labels = [label for label_set in label_sets for label in label_set.labels]
-    return generate_label_set(role, "+".join(package_names), labels, bpp, package="aggregate")
-
-
-def print_label_sets(title: str, label_sets: list[LabelSet]) -> None:
-    print(f"\n# {title}")
-    print("role\tpackage\tmember\tlabels\tset_len\tset_prefix")
-    for label_set in label_sets:
-        print(
-            f"{label_set.role}\t{label_set.package}\t{label_set.member}\t"
-            f"{label_set.label_count}\t{label_set.set_len}\t{label_set.set_string[:48]}"
-        )
-
-
-def print_results(results: list[CompatibilityResult]) -> None:
-    print("\n# compatibility")
-    print("status\tcmp\tprovider_pkg\tprovider_member\tprovider_labels\trequirer_pkg\trequirer_member\trequired_labels")
-    for result in results:
-        print(
-            f"{result.status}\t{result.cmp_result}\t{result.provider_package}\t{result.provider_member}\t"
-            f"{result.provider_labels}\t{result.requirer_package}\t{result.requirer_member}\t{result.required_labels}"
-        )
-    summary = {status: sum(1 for result in results if result.status == status) for status in sorted({r.status for r in results})}
-    print(f"summary: {summary}")
-
-
-def parse_package_list(values: list[str] | None) -> list[str]:
-    packages = []
-    for value in values or []:
-        packages.extend(part for part in value.split(",") if part)
-    return packages
-
-
-def parse_args(argv: list[str]) -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Generate Provided/Required set strings from ALT RPM labels with reimplement/set.py and compare them."
-    )
-    parser.add_argument("packages", nargs="*", help="packages used as both providers and requirers if explicit lists are omitted")
-    parser.add_argument("--provider", action="append", help="provider package; can be repeated or comma-separated")
-    parser.add_argument("--requirer", action="append", help="requirer package; can be repeated or comma-separated")
-    parser.add_argument("--branch", default="sisyphus", help="ALT repository branch/packageset")
-    parser.add_argument("--arch", default="x86_64", help="binary package architecture")
-    parser.add_argument("--bpp", type=int, default=32, help="bits per hash used by local set.py")
-    parser.add_argument("--max-provider-files", type=int, default=64, help="max provider ELF files per package")
-    parser.add_argument("--max-requirer-files", type=int, default=64, help="max requirer ELF files per package")
-    parser.add_argument("--all-pairs", action="store_true", help="compare every provider file set with every requirer file set")
-    parser.add_argument(
-        "--no-normalize-required-version",
-        action="store_true",
-        help="keep nm -u single-@ required symbols unchanged instead of converting foo@VER to foo@@VER",
-    )
-    parser.add_argument("--keep-workdir", action="store_true", help="keep downloaded RPMs/extracted files")
-    parser.add_argument("--workdir", type=Path, help="directory for downloads/extraction")
-    return parser.parse_args(argv)
-
-
-def main(argv: list[str]) -> int:
-    args = parse_args(argv)
-    missing = [cmd for cmd in ("bsdtar", "nm") if shutil.which(cmd) is None]
-    if missing:
-        print(f"missing required command(s): {', '.join(missing)}", file=sys.stderr)
-        return 2
-
-    positional = args.packages or []
-    providers = parse_package_list(args.provider) or positional or DEFAULT_PROVIDERS
-    requirers = parse_package_list(args.requirer) or positional or DEFAULT_REQUIRERS
-
-    cleanup = False
-    if args.workdir:
-        workdir = args.workdir
-        workdir.mkdir(parents=True, exist_ok=True)
-    else:
-        workdir = Path(tempfile.mkdtemp(prefix="arsv-alt-set-compat-"))
-        cleanup = not args.keep_workdir
-
-    try:
-        provider_sets: list[LabelSet] = []
-        requirer_sets: list[LabelSet] = []
-        for package in providers:
-            provider_sets.extend(
-                build_provider_sets(fetch_package_rpm(package, args.branch, args.arch, workdir), args.bpp, args.max_provider_files)
-            )
-        for package in requirers:
-            requirer_sets.extend(
-                build_requirer_sets(
-                    fetch_package_rpm(package, args.branch, args.arch, workdir),
-                    args.bpp,
-                    args.max_requirer_files,
-                    normalize_versions=not args.no_normalize_required_version,
-                )
-            )
-
-        print(f"branch: {args.branch}")
-        print(f"arch: {args.arch}")
-        print(f"bpp: {args.bpp}")
-        print(f"providers: {', '.join(providers)}")
-        print(f"requirers: {', '.join(requirers)}")
-        print(f"required symbol version normalization: {not args.no_normalize_required_version}")
-        print_label_sets("generated Provided sets", provider_sets)
-        print_label_sets("generated Required sets", requirer_sets)
-
-        if not provider_sets or not requirer_sets:
-            print("\nNo comparable sets generated.", file=sys.stderr)
-            return 1
-
-        if args.all_pairs:
-            results = [compare_label_sets(provider, requirer) for provider in provider_sets for requirer in requirer_sets]
-        else:
-            results = build_dependency_results(provider_sets, requirer_sets, args.bpp)
-        print_results(results)
-    finally:
-        if cleanup:
-            shutil.rmtree(workdir, ignore_errors=True)
-        else:
-            print(f"workdir: {workdir}")
-
-    return 0 if all(result.status == "compatible" for result in results) else 1
-
-
-if __name__ == "__main__":
-    raise SystemExit(main(sys.argv[1:]))

scripts/old/check_arch_set_impl.py

@@ -1,172 +0,0 @@
-#!/usr/bin/env python3
-"""Check reimplement/set.py against ELF libraries installed on this Arch Linux system.
-
-This is the local-system analogue of check_alt_set_impl.py. It treats shared
-libraries as providers (defined dynamic symbols) and executables/shared objects
-as requirers (undefined dynamic symbols). Required labels are split by provider
-library before comparing, so each library is checked only against symbols that
-it actually exports.
-"""
-
-from __future__ import annotations
-
-import argparse
-import os
-import shutil
-import sys
-from pathlib import Path
-
-SCRIPT_DIR = Path(__file__).resolve().parent
-if str(SCRIPT_DIR) not in sys.path:
-    sys.path.insert(0, str(SCRIPT_DIR))
-
-import check_alt_set_impl as compat  # noqa: E402
-
-DEFAULT_PROVIDER_DIRS = [Path("/usr/lib")]
-DEFAULT_REQUIRER_DIRS = [Path("/usr/bin"), Path("/usr/lib")]
-
-
-def parse_path_list(values: list[str] | None, defaults: list[Path]) -> list[Path]:
-    paths: list[Path] = []
-    for value in values or []:
-        paths.extend(Path(part) for part in value.split(",") if part)
-    return paths or defaults
-
-
-def is_shared_library_path(path: Path) -> bool:
-    name = path.name
-    return ".debug" not in name and name.startswith("lib") and ".so" in name
-
-
-def iter_files(paths: list[Path], recursive: bool) -> list[Path]:
-    files: list[Path] = []
-    seen: set[Path] = set()
-    for root in paths:
-        if root.is_file():
-            candidates = [root]
-        elif recursive:
-            candidates = (path for path in root.rglob("*") if path.is_file())
-        else:
-            candidates = (path for path in root.iterdir() if path.is_file()) if root.is_dir() else []
-        for path in candidates:
-            try:
-                resolved = path.resolve()
-            except OSError:
-                continue
-            if resolved in seen:
-                continue
-            seen.add(resolved)
-            files.append(path)
-    return sorted(files)
-
-
-def executable_or_library(path: Path) -> bool:
-    return is_shared_library_path(path) or os.access(path, os.X_OK)
-
-
-def build_local_provider_sets(paths: list[Path], bpp: int, recursive: bool, max_files: int) -> list[compat.LabelSet]:
-    provider_files = [path for path in iter_files(paths, recursive) if is_shared_library_path(path)][:max_files]
-    sets: list[compat.LabelSet] = []
-    for path in provider_files:
-        labels = compat.nm_symbols(path, "provided")
-        if labels:
-            sets.append(compat.generate_label_set("provided", str(path), labels, bpp, package="arch-local"))
-    return sets
-
-
-def build_local_requirer_sets(
-    paths: list[Path],
-    bpp: int,
-    recursive: bool,
-    max_files: int,
-    normalize_versions: bool,
-) -> list[compat.LabelSet]:
-    requirer_files = [path for path in iter_files(paths, recursive) if executable_or_library(path)][:max_files]
-    sets: list[compat.LabelSet] = []
-    for path in requirer_files:
-        labels = compat.nm_symbols(path, "required")
-        if normalize_versions:
-            labels = [compat.normalize_required_symbol(label) for label in labels]
-        if labels:
-            sets.append(compat.generate_label_set("required", str(path), labels, bpp, package="arch-local"))
-    return sets
-
-
-def print_unmatched_required_labels(provider_sets: list[compat.LabelSet], requirer_sets: list[compat.LabelSet], limit: int) -> None:
-    provided = {label for provider in provider_sets for label in provider.labels}
-    rows: list[tuple[str, int, list[str]]] = []
-    for requirer in requirer_sets:
-        missing = sorted(set(requirer.labels).difference(provided))
-        if missing:
-            rows.append((requirer.member, len(missing), missing[:limit]))
-
-    print("\n# required labels not exported by scanned provider libraries")
-    print("requirer\tmissing_labels\texamples")
-    for member, count, examples in rows[:limit]:
-        print(f"{member}\t{count}\t{', '.join(examples)}")
-    if len(rows) > limit:
-        print(f"... {len(rows) - limit} more requirer files with unmatched labels")
-
-
-def parse_args(argv: list[str]) -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Generate Provided/Required set strings from local Arch Linux ELF files and compare them with reimplement/set.py."
-    )
-    parser.add_argument("--provider-dir", action="append", help="directory/file containing provider libraries; repeat or comma-separate")
-    parser.add_argument("--requirer-dir", action="append", help="directory/file containing requirer ELF files; repeat or comma-separate")
-    parser.add_argument("--bpp", type=int, default=32, help="bits per hash used by local set.py")
-    parser.add_argument("--max-provider-files", type=int, default=256, help="max provider libraries to inspect")
-    parser.add_argument("--max-requirer-files", type=int, default=256, help="max requirer files to inspect")
-    parser.add_argument("--recursive", action="store_true", help="scan directories recursively")
-    parser.add_argument("--all-pairs", action="store_true", help="compare every provider set with every full requirer set")
-    parser.add_argument(
-        "--no-normalize-required-version",
-        action="store_true",
-        help="keep nm -u single-@ required symbols unchanged instead of converting foo@VER to foo@@VER",
-    )
-    parser.add_argument("--unmatched-limit", type=int, default=20, help="max unmatched-label rows/examples to print")
-    return parser.parse_args(argv)
-
-
-def main(argv: list[str]) -> int:
-    args = parse_args(argv)
-    if shutil.which("nm") is None:
-        print("missing required command: nm", file=sys.stderr)
-        return 2
-
-    provider_paths = parse_path_list(args.provider_dir, DEFAULT_PROVIDER_DIRS)
-    requirer_paths = parse_path_list(args.requirer_dir, DEFAULT_REQUIRER_DIRS)
-
-    provider_sets = build_local_provider_sets(provider_paths, args.bpp, args.recursive, args.max_provider_files)
-    requirer_sets = build_local_requirer_sets(
-        requirer_paths,
-        args.bpp,
-        args.recursive,
-        args.max_requirer_files,
-        normalize_versions=not args.no_normalize_required_version,
-    )
-
-    print("system: arch-local")
-    print(f"bpp: {args.bpp}")
-    print(f"provider paths: {', '.join(str(path) for path in provider_paths)}")
-    print(f"requirer paths: {', '.join(str(path) for path in requirer_paths)}")
-    print(f"required symbol version normalization: {not args.no_normalize_required_version}")
-    compat.print_label_sets("generated Provided sets", provider_sets)
-    compat.print_label_sets("generated Required sets", requirer_sets)
-
-    if not provider_sets or not requirer_sets:
-        print("\nNo comparable sets generated.", file=sys.stderr)
-        return 1
-
-    if args.all_pairs:
-        results = [compat.compare_label_sets(provider, requirer) for provider in provider_sets for requirer in requirer_sets]
-    else:
-        results = compat.build_dependency_results(provider_sets, requirer_sets, args.bpp)
-    compat.print_results(results)
-    print_unmatched_required_labels(provider_sets, requirer_sets, args.unmatched_limit)
-
-    return 0 if results and all(result.status == "compatible" for result in results) else 1
-
-
-if __name__ == "__main__":
-    raise SystemExit(main(sys.argv[1:]))