Repository navigation
Release 1.7.0 - #406
Merged
Merged
Release 1.7.0#406
Conversation
Only the latest release is maintained. Fixes #405.
The apt lists of the base image can be stale, and depexts then fail with 404 errors.
Enable the LRU in it: the suite disables it by default.
During a merge, writes go to log_async. A read-only instance that loads it on sync could still return the old values cached in its LRU.
With zero-size values, the log scratch buffer holds exactly one key, and a decoder like Repr.decode_bin returns it without copying. The merge then wrote the same key several times and lost the others, and iter gave wrong keys.
Describe the header layout with byte ranges and encoding, and the unchecked bounds of the unsafe functions. Correct a few points: the offset does not depend on fsync, Not_written is raised only when the file ends at or before the first byte read, Version.set does not check its argument, and files using Header_prefix have a different layout.
Io_array used one prefetch buffer for all the instances of an Index.Make application. Finds on two instances are guarded by distinct locks, so, from two threads or domains, they could overwrite each other's prefetched entries and miss bindings.
A header field that ended in the middle failed an assertion, or was silently decoded from uninitialised bytes with -noassert. A read-only instance now handles it like a missing header.
Version.set read past the end of a shorter string, as pwrite does not check bounds. Header.set and Header_prefix.set only checked it with an assertion. All three now raise Invalid_argument.
really_write stopped silently, leaving the rest of the buffer unwritten, while the caller then updated the offset as if it was.
They were turned into an Assert_failure that hid the original exception.
find only compares the key, so it does not need the copy.
Otherwise the merge thread stays blocked in its hook.
Sync cleared the LRU before loading the new bindings, and only held sync_lock. A find on the same instance, from another thread or domain, could then cache the old value right after the clear. Sync now holds rename_lock instead, which find also takes.
Clearing the whole LRU on every sync that reads new bindings dropped all the cached values of a read-only instance that syncs often. The LRU is still cleared when the generation changes.
Cover a sync that appends to the log_async already loaded, and one that reloads the log_async of a second merge.
Complete the changelog with the changes merged since 1.6.2.
This was referenced Nov 13, 2026
This was referenced Nov 13, 2026
jmid
pushed a commit
to ocaml/opam-repository
that referenced
this pull request
Nov 17, 2026
CHANGES: ## Added - Add `Stats.merge` and `Stats.merge_in`, to combine the statistics recorded by several domains (mirage/index#404, @lyrm) - Expose the Unix platform as `Index_unix.Private.Platform`, so that `Index.Make` can be applied to it directly (mirage/index#396, @art-w) - Document the file format of `Index_unix.Private.Raw` (mirage/index#380, @tomjridge, @balat) ## Changed - Breaking: OCaml 5.0 or later is now required (mirage/index#404, @lyrm) - Breaking: statistics are now recorded per domain, in a `Domain.DLS` key, instead of in a global value. `Stats.get` and `Stats.reset_stats` only see the calling domain (mirage/index#404, @lyrm) - Mirage support: `index` no longer depends on `unix`, and `index.unix` is now an optional library (mirage/index#396, @art-w) - Breaking: `Platform.S` and `Checks.Platform_args` now require a `Progress` and a `Fmt_tty` module, and `Private.Logs.setup` and `Private.Logs.setup_term` take an extra `(module Platform.FMT_TTY)` argument (mirage/index#396, @art-w) - Support cmdliner 2.0.0. Cmdliner 1.1.0 or later is now required (mirage/index#382, @MisterDA, mirage/index#402, @art-w) ## Fixed - Lock files are now opened with the `O_CLOEXEC` flag (mirage/index#394, @vect0r-vicall) - Read-only instances could return stale values after `sync`. `sync` now removes from their LRU the keys of the entries it loads, from the log or from the log used during merges. It also holds the lock that `find` takes, so that a concurrent `find` cannot cache an outdated value in the middle of a `sync` (mirage/index#398, mirage/index#399, @art-w, mirage/index#406, @balat) - Merges could lose entries, and `iter` could give wrong keys, when values are encoded on zero bytes and the key decoder returns its input string without copying it, as `Repr.decode_bin` does (mirage/index#387, @Ngoguey42) - Concurrent `find`s on distinct instances of the same `Index.Make` application, from several threads or domains, could miss bindings: they shared the buffer in which they prefetch entries of the data file. Each instance now has its own buffer, created with `Private.Io_array.prefetch_buffer` and passed to `Private.Io_array.v` (mirage/index#406, @balat) - `Index_unix.Private.Raw`: reading a truncated header now raises `Not_written`, as for a missing header, instead of failing an assertion. Writing a version that is not 8 bytes long raises `Invalid_argument`, and a write that makes no progress raises `Failure` instead of being dropped (mirage/index#406, @balat) - Add the missing `logs.threaded` dependency of the Unix tests (mirage/index#400, @anmonteiro)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This prepares the 1.7.0 release, needed by Irmin 4 (it uses
Index_unix.Private.Platform, which is only onmain).mainalready fixed the bench build. Fixes Updated values are not seen in RO instances #398.syncloads a newlog_asyncduring a merge. With a test.sync: it cleared the LRU before loading the new bindings, without the lock thatfindtakes, so afindfrom another thread or domain could cache the old value again.syncnow holdsrename_lock. With a stress test on two domains. Trade-off: asyncthat reloads the whole log now blocks thefinds of other domains while it runs.syncnow removes only the keys of the new bindings from the LRU, instead of clearing it. The LRU is still cleared when the generation of the log changes.Repr.decode_binreturns the log scratch buffer itself as the key, so merges could lose entries anditercould give wrong keys. Irmin's snapshot export uses such an index. The decoder contract is now documented inData.Key.decode.Rawfrom Add docs for src/unix/raw.mli #380 (thanks @tomjridge!), corrected and completed. Small fixes inRaw: a truncated header now raisesNot_written, a version that is not 8 bytes long raisesInvalid_argument, and apwritethat writes nothing raisesFailureinstead of being ignored.Index.Makeapplication shared the buffer in whichfindprefetches entries, so finds on two instances, from two threads or domains, could miss bindings. Each instance now has its own buffer. With a test on two domains.x-maintenance-intent(opam file: add x-maintenance-intent #405).Statsfunctions andPlatform.FMT_TTY.bench.Dockerfile: runsapt-get updatebefore installing depexts, since the apt lists of the base image were stale.Tested with OCaml 5.5.1 and cmdliner 2.1.1. irmin-pack 3.11.0 and irmin-pack-tools 3.11.0, the only reverse dependencies on opam, still build. Does this look good to you?