Skip to content

Release 1.7.0 - #406

Merged
balat merged 22 commits into
mainfrom
release-1.7.0
Nov 13, 2026
Merged

balat merged 22 commits into
mainfrom
release-1.7.0

Conversation

@balat

@balat balat commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

This prepares the 1.7.0 release, needed by Irmin 4 (it uses Index_unix.Private.Platform, which is only on main).

  • Includes the fix from Fix RO sync: clear LRU for all log updates #399 (thanks @art-w!). The second commit of Fix RO sync: clear LRU for all log updates #399 is no longer needed, since main already fixed the bench build. Fixes Updated values are not seen in RO instances #398.
  • Fixes a similar case: a read-only instance could still return a stale value from its LRU when sync loads a new log_async during a merge. With a test.
  • The tests now enable the LRU for these cases (the suite disables it by default).
  • Fixes a race in sync: it cleared the LRU before loading the new bindings, without the lock that find takes, so a find from another thread or domain could cache the old value again. sync now holds rename_lock. With a stress test on two domains. Trade-off: a sync that reloads the whole log now blocks the finds of other domains while it runs.
  • sync now removes only the keys of the new bindings from the LRU, instead of clearing it. The LRU is still cleared when the generation of the log changes.
  • Includes the fix from Quickfix for situations where Value hash length 0 #387 (thanks @Ngoguey42!), with regression tests. With zero-size values, Repr.decode_bin returns the log scratch buffer itself as the key, so merges could lose entries and iter could give wrong keys. Irmin's snapshot export uses such an index. The decoder contract is now documented in Data.Key.decode.
  • Includes the documentation of Raw from Add docs for src/unix/raw.mli #380 (thanks @tomjridge!), corrected and completed. Small fixes in Raw: a truncated header now raises Not_written, a version that is not 8 bytes long raises Invalid_argument, and a pwrite that writes nothing raises Failure instead of being ignored.
  • Fixes a race between instances: all the instances of an Index.Make application shared the buffer in which find prefetches entries, so finds on two instances, from two threads or domains, could miss bindings. Each instance now has its own buffer. With a test on two domains.
  • Adds x-maintenance-intent (opam file: add x-maintenance-intent #405).
  • Documents the per-domain Stats functions and Platform.FMT_TTY.
  • bench.Dockerfile: runs apt-get update before installing depexts, since the apt lists of the base image were stale.
  • Completes CHANGES.md.

Tested with OCaml 5.5.1 and cmdliner 2.1.1. irmin-pack 3.11.0 and irmin-pack-tools 3.11.0, the only reverse dependencies on opam, still build. Does this look good to you?

art-w and others added 6 commits October 9, 2026 16:38
Only the latest release is maintained. Fixes #405.
The apt lists of the base image can be stale, and depexts then fail
with 404 errors.
Enable the LRU in it: the suite disables it by default.
During a merge, writes go to log_async. A read-only instance that
loads it on sync could still return the old values cached in its LRU.
Ngoguey42 and others added 6 commits November 9, 2026 13:13
With zero-size values, the log scratch buffer holds exactly one key, and a
decoder like Repr.decode_bin returns it without copying. The merge then
wrote the same key several times and lost the others, and iter gave wrong
keys.
Describe the header layout with byte ranges and encoding, and the
unchecked bounds of the unsafe functions. Correct a few points: the offset
does not depend on fsync, Not_written is raised only when the file ends at
or before the first byte read, Version.set does not check its argument, and
files using Header_prefix have a different layout.
Io_array used one prefetch buffer for all the instances of an Index.Make
application. Finds on two instances are guarded by distinct locks, so,
from two threads or domains, they could overwrite each other's prefetched
entries and miss bindings.
balat added 9 commits November 9, 2026 17:13
A header field that ended in the middle failed an assertion, or was
silently decoded from uninitialised bytes with -noassert. A read-only
instance now handles it like a missing header.
Version.set read past the end of a shorter string, as pwrite does not
check bounds. Header.set and Header_prefix.set only checked it with an
assertion. All three now raise Invalid_argument.
really_write stopped silently, leaving the rest of the buffer unwritten,
while the caller then updated the offset as if it was.
They were turned into an Assert_failure that hid the original exception.
find only compares the key, so it does not need the copy.
Otherwise the merge thread stays blocked in its hook.
Sync cleared the LRU before loading the new bindings, and only held
sync_lock. A find on the same instance, from another thread or domain,
could then cache the old value right after the clear. Sync now holds
rename_lock instead, which find also takes.
Clearing the whole LRU on every sync that reads new bindings dropped all
the cached values of a read-only instance that syncs often. The LRU is
still cleared when the generation changes.
Cover a sync that appends to the log_async already loaded, and one that
reloads the log_async of a second merge.
Complete the changelog with the changes merged since 1.6.2.
@balat
balat merged commit 0f32254 into main Nov 13, 2026
2 of 3 checks passed
jmid pushed a commit to ocaml/opam-repository that referenced this pull request Nov 17, 2026
CHANGES:

## Added

- Add `Stats.merge` and `Stats.merge_in`, to combine the statistics recorded
  by several domains (mirage/index#404, @lyrm)
- Expose the Unix platform as `Index_unix.Private.Platform`, so that
  `Index.Make` can be applied to it directly (mirage/index#396, @art-w)
- Document the file format of `Index_unix.Private.Raw` (mirage/index#380, @tomjridge,
  @balat)

## Changed

- Breaking: OCaml 5.0 or later is now required (mirage/index#404, @lyrm)
- Breaking: statistics are now recorded per domain, in a `Domain.DLS` key,
  instead of in a global value. `Stats.get` and `Stats.reset_stats` only see
  the calling domain (mirage/index#404, @lyrm)
- Mirage support: `index` no longer depends on `unix`, and `index.unix` is
  now an optional library (mirage/index#396, @art-w)
- Breaking: `Platform.S` and `Checks.Platform_args` now require a `Progress`
  and a `Fmt_tty` module, and `Private.Logs.setup` and
  `Private.Logs.setup_term` take an extra `(module Platform.FMT_TTY)`
  argument (mirage/index#396, @art-w)
- Support cmdliner 2.0.0. Cmdliner 1.1.0 or later is now required (mirage/index#382,
  @MisterDA, mirage/index#402, @art-w)

## Fixed

- Lock files are now opened with the `O_CLOEXEC` flag (mirage/index#394, @vect0r-vicall)
- Read-only instances could return stale values after `sync`. `sync` now
  removes from their LRU the keys of the entries it loads, from the log or
  from the log used during merges. It also holds the lock that `find` takes,
  so that a concurrent `find` cannot cache an outdated value in the middle of
  a `sync` (mirage/index#398, mirage/index#399, @art-w, mirage/index#406, @balat)
- Merges could lose entries, and `iter` could give wrong keys, when values
  are encoded on zero bytes and the key decoder returns its input string
  without copying it, as `Repr.decode_bin` does (mirage/index#387, @Ngoguey42)
- Concurrent `find`s on distinct instances of the same `Index.Make`
  application, from several threads or domains, could miss bindings: they
  shared the buffer in which they prefetch entries of the data file. Each
  instance now has its own buffer, created with
  `Private.Io_array.prefetch_buffer` and passed to `Private.Io_array.v`
  (mirage/index#406, @balat)
- `Index_unix.Private.Raw`: reading a truncated header now raises
  `Not_written`, as for a missing header, instead of failing an assertion.
  Writing a version that is not 8 bytes long raises `Invalid_argument`, and a
  write that makes no progress raises `Failure` instead of being dropped
  (mirage/index#406, @balat)
- Add the missing `logs.threaded` dependency of the Unix tests (mirage/index#400,
  @anmonteiro)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Updated values are not seen in RO instances

4 participants