Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
a2e885d
Fix RO sync: clear LRU for all log updates
art-w May 24, 2024
73e78dd
opam: add x-maintenance-intent
balat Oct 9, 2026
7a5bef7
bench.Dockerfile: update apt lists before installing depexts
balat Nov 6, 2026
f12eedc
Move the #398 regression test to the readonly suite
balat Nov 6, 2026
02dc2bf
Fix RO sync: also clear the LRU when loading a new log_async
balat Nov 6, 2026
6a08952
Document the per-domain Stats functions and Platform.FMT_TTY
balat Nov 6, 2026
e5546a8
Quickfix for situations where Value hash length 0
Ngoguey42 Apr 15, 2022
90fb666
Add regression tests for zero-size values
balat Nov 9, 2026
ba46e56
Add docs for src/unix/raw.mli
tomjridge Feb 8, 2022
ccdadc8
Fix and complete the Raw documentation
balat Nov 9, 2026
7b63b4a
Give each instance its own prefetch buffer
balat Nov 9, 2026
b420234
Document that decoders must not keep a reference to their input
balat Nov 9, 2026
af702d7
Raw: raise Not_written on a truncated header
balat Nov 9, 2026
f70c86c
Raw: reject versions that are not 8 bytes long
balat Nov 9, 2026
98c811e
Raw: fail when pwrite writes nothing
balat Nov 9, 2026
7d732bb
Io_array: let decoding errors propagate
balat Nov 9, 2026
d52923c
Log_file: copy zero-size entries only when they are kept
balat Nov 9, 2026
563abd0
Test: release the merge if the log_async sync fails
balat Nov 9, 2026
f698bdd
Fix RO sync: hold rename_lock, like find
balat Nov 9, 2026
ccd6e12
RO sync: only remove the updated keys from the LRU
balat Nov 9, 2026
caf8032
Test RO syncs that update a loaded log_async
balat Nov 9, 2026
6e3897f
Release 1.7.0
balat Oct 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 47 additions & 5 deletions CHANGES.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,53 @@
# Unreleased
# 1.7.0 (2026-11-13)

## Added

- Add `Stats.merge` and `Stats.merge_in`, to combine the statistics recorded
by several domains (#404, @lyrm)
- Expose the Unix platform as `Index_unix.Private.Platform`, so that
`Index.Make` can be applied to it directly (#396, @art-w)
- Document the file format of `Index_unix.Private.Raw` (#380, @tomjridge,
@balat)

## Changed

- Replace global value of Stats by a DLS.key (#404, @lyrm)
- Lock files are now opened with O_CLOEXEC flag (#394, @vect0r-vicall)
- Update to cmdliner.1.1.0 (#382, @MisterDA)
- Mirage support: optional dependency to unix (#396, @art-w)
- Breaking: OCaml 5.0 or later is now required (#404, @lyrm)
- Breaking: statistics are now recorded per domain, in a `Domain.DLS` key,
instead of in a global value. `Stats.get` and `Stats.reset_stats` only see
the calling domain (#404, @lyrm)
- Mirage support: `index` no longer depends on `unix`, and `index.unix` is
now an optional library (#396, @art-w)
- Breaking: `Platform.S` and `Checks.Platform_args` now require a `Progress`
and a `Fmt_tty` module, and `Private.Logs.setup` and
`Private.Logs.setup_term` take an extra `(module Platform.FMT_TTY)`
argument (#396, @art-w)
- Support cmdliner 2.0.0. Cmdliner 1.1.0 or later is now required (#382,
@MisterDA, #402, @art-w)

## Fixed

- Lock files are now opened with the `O_CLOEXEC` flag (#394, @vect0r-vicall)
- Read-only instances could return stale values after `sync`. `sync` now
removes from their LRU the keys of the entries it loads, from the log or
from the log used during merges. It also holds the lock that `find` takes,
so that a concurrent `find` cannot cache an outdated value in the middle of
a `sync` (#398, #399, @art-w, #406, @balat)
- Merges could lose entries, and `iter` could give wrong keys, when values
are encoded on zero bytes and the key decoder returns its input string
without copying it, as `Repr.decode_bin` does (#387, @Ngoguey42)
- Concurrent `find`s on distinct instances of the same `Index.Make`
application, from several threads or domains, could miss bindings: they
shared the buffer in which they prefetch entries of the data file. Each
instance now has its own buffer, created with
`Private.Io_array.prefetch_buffer` and passed to `Private.Io_array.v`
(#406, @balat)
- `Index_unix.Private.Raw`: reading a truncated header now raises
`Not_written`, as for a missing header, instead of failing an assertion.
Writing a version that is not 8 bytes long raises `Invalid_argument`, and a
write that makes no progress raises `Failure` instead of being dropped
(#406, @balat)
- Add the missing `logs.threaded` dependency of the Unix tests (#400,
@anmonteiro)

# 1.6.2 (2023-06-06)

Expand Down
2 changes: 1 addition & 1 deletion bench.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ RUN sudo chown opam .
COPY *.opam ./
RUN opam remote add origin https://github.com/ocaml/opam-repository.git && opam update
RUN opam pin -yn --with-version=dev .
RUN opam install -y --deps-only --with-test .
RUN sudo apt-get update && opam install -y --deps-only --with-test .
COPY . ./
ENTRYPOINT opam exec -- make bench
1 change: 1 addition & 0 deletions index-bench.opam
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@ build: [
["dune" "runtest" "-p" name "-j" jobs] {with-test}
]
dev-repo: "git+https://github.com/mirage/index.git"
x-maintenance-intent: [ "(latest)" ]
1 change: 1 addition & 0 deletions index.opam
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ homepage: "https://github.com/mirage/index"
bug-reports: "https://github.com/mirage/index/issues"
dev-repo: "git+https://github.com/mirage/index.git"
doc: "https://mirage.github.io/index/"
x-maintenance-intent: [ "(latest)" ]

build: [
["dune" "subst"] {dev}
Expand Down
10 changes: 9 additions & 1 deletion src/data.ml
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,23 @@ module type Key = sig

val decode : string -> int -> t
(** [decode s off] is the decoded form of the encoded value at the offset
[off] of string [s]. Must satisfy [decode (encode t) 0 = t]. *)
[off] of string [s]. Must satisfy [decode (encode t) 0 = t].

[s] may be a buffer that is overwritten later, so the result must not keep
a reference to [s], except when [off = 0] and [s] holds exactly one key:
[Repr.decode_bin] returns [s] itself in that case. *)
end

module type Value = sig
type t [@@deriving repr]

val encode : t -> string
val encoded_size : int

val decode : string -> int -> t
(** [decode s off] is the decoded form of the encoded value at the offset
[off] of string [s]. [s] may be a buffer that is overwritten later, so the
result must not keep a reference to [s]. *)
end

module Entry = struct
Expand Down
62 changes: 43 additions & 19 deletions src/index.ml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ struct
val create : int -> t
val add : t -> key -> value -> unit
val find : t -> key -> value option
val remove : t -> key -> unit
val clear : t -> unit
end = struct
module Lru =
Expand Down Expand Up @@ -98,6 +99,7 @@ struct
Lru.add k v t;
Lru.trim t

let remove t k = Lru.remove k !t
let clear t = t := Lru.create (Lru.capacity !t)
end

Expand Down Expand Up @@ -149,6 +151,8 @@ struct
header of [io]. *)
}

module IOArray = Io_array.Make (IO) (Entry)

type instance = {
config : config;
root : string; (** The root location of the index *)
Expand All @@ -169,11 +173,12 @@ struct
mutable open_instances : int;
(** The number of open instances that are shared through the [Cache.t]. *)
mutable lru : Lru.t;
prefetch_buffer : IOArray.prefetch_buffer;
(** The buffer in which [find] prefetches entries of [index], guarded by
[rename_lock]. *)
writer_lock : IO.Lock.t option;
(** A lock that prevents multiple RW instances to be open at the same
time. *)
sync_lock : Semaphore.t;
(** A lock that prevents multiple [sync] to happen at the same time. *)
merge_lock : Semaphore.t;
(** A lock that prevents multiple [merges] to happen at the same time. *)
rename_lock : Semaphore.t;
Expand Down Expand Up @@ -277,14 +282,15 @@ struct

(** {1 RO instances syncing} *)

(** Loads the log file at [path], if it exists. Used by RO instances to load
the temporary [log_async], or to fill the [log] field when they have been
created before their RW counterpart. *)
let try_load_log t path =
(** Loads the log file at [path], if it exists, calling [on_key] on the key of
each of its entries. Used by RO instances to load the temporary
[log_async], or to fill the [log] field when they have been created before
their RW counterpart. *)
let try_load_log ?on_key t path =
match IO.v_readonly path with
| Error `No_file_on_disk -> None
| Ok io ->
let log = Log_file.create io in
let log = Log_file.create ?on_key io in
Log.debug (fun l ->
l "[%s] loaded %d entries from %s" (Filename.basename t.root)
(Log_file.cardinal log) (Filename.basename path));
Expand All @@ -295,7 +301,10 @@ struct
match t.log_async with
| None ->
hook `Reload_log_async;
t.log_async <- try_load_log t (Layout.log_async ~root:t.root)
(* The entries of a new [log_async] may update keys cached in the LRU. *)
t.log_async <-
try_load_log ~on_key:(Lru.remove t.lru) t
(Layout.log_async ~root:t.root)
| Some log ->
let old_generation = t.generation in
let old_offset = IO.offset (Log_file.io log) in
Expand All @@ -310,10 +319,15 @@ struct
Log_file.close log;
(* check that file is on disk, reopen and reload everything. *)
hook `Reload_log_async;
t.log_async <- try_load_log t (Layout.log_async ~root:t.root)
(* Only the keys of the new [log_async] must leave the LRU: the
bindings of the previous one were copied into [log], and a merge
changes the generation of [log], which clears the LRU. *)
t.log_async <-
try_load_log ~on_key:(Lru.remove t.lru) t
(Layout.log_async ~root:t.root)
(* else if the disk offset is greater, reload the newest data. *))
else if old_offset < h.offset then
Log_file.sync_entries ~min:old_offset log
Log_file.sync_entries ~on_key:(Lru.remove t.lru) ~min:old_offset log
(* else if the offset is lesser, that means the [log_async] was
cleared, and the generation should have changed. *)
else if old_offset > h.offset then (
Expand All @@ -327,6 +341,7 @@ struct
Reloading the log to compensate." (Filename.basename t.root)
Int63.pp old_offset Int63.pp h.offset Int63.pp old_generation
Int63.pp h.generation);
Lru.clear t.lru;
Log_file.reload log)

(** Syncs the [index] of the instance by checking on-disk changes. *)
Expand All @@ -344,9 +359,16 @@ struct
else t.index <- Some { fan_out; io }

(** Syncs an instance entirely, by checking on-disk changes for [log], [sync],
and [log_async]. *)
and [log_async].

It holds [rename_lock], like [find]: a [find] running between the
invalidation of the LRU and the loading of the new bindings would put an
outdated value back in the LRU. This also serializes the syncs.

New bindings only remove their keys from the LRU. A new generation of
[log] clears it, as a merge or a clear may have changed any binding. *)
let sync_instance ?(hook = fun _ -> ()) t =
Semaphore.with_acquire "sync" t.sync_lock @@ fun () ->
Semaphore.with_acquire "sync" t.rename_lock @@ fun () ->
Log.debug (fun l ->
l "[%s] checking for changes on disk (generation=%a)"
(Filename.basename t.root) Int63.pp t.generation);
Expand Down Expand Up @@ -397,7 +419,7 @@ struct
Log.debug (fun l ->
l "[%s] new entries detected, reading log from disk"
(Filename.basename t.root));
Log_file.sync_entries ~min:log_offset log)
Log_file.sync_entries ~on_key:(Lru.remove t.lru) ~min:log_offset log)
else
(* Here the disk offset should be equal to the known one. A smaller
offset should not be possible, because that would mean a [clear] or
Expand All @@ -408,8 +430,6 @@ struct

(** {1 Find and Mem}*)

module IOArray = Io_array.Make (IO) (Entry)

module Search =
Search.Make (Entry) (IOArray)
(struct
Expand All @@ -436,14 +456,16 @@ struct
Int63.of_float rounded
end)

let interpolation_search index key =
let interpolation_search prefetch_buffer index key =
let hashed_key = K.hash key in
let low_bytes, high_bytes = Fan.search index.fan_out hashed_key in
let low, high =
Int63.
(div low_bytes Entry.encoded_sizeL, div high_bytes Entry.encoded_sizeL)
in
Search.interpolation_search (IOArray.v index.io) key ~low ~high
Search.interpolation_search
(IOArray.v prefetch_buffer index.io)
key ~low ~high

(** Finds the value associated to [key] in [t]. In order, checks in
[log_async] (in memory), then [log] (in memory), then [index] (on disk). *)
Expand All @@ -462,7 +484,9 @@ struct
in
let find_log () = find_if_exists ~name:"log" ~find:Log_file.find t.log in
let find_index () =
find_if_exists ~name:"index" ~find:interpolation_search t.index
find_if_exists ~name:"index"
~find:(interpolation_search t.prefetch_buffer)
t.index
in
Semaphore.with_acquire "find_instance" t.rename_lock @@ fun () ->
match Lru.find t.lru key with
Expand Down Expand Up @@ -608,10 +632,10 @@ struct
root;
index;
lru = Lru.create lru_size;
prefetch_buffer = IOArray.prefetch_buffer ();
open_instances = 1;
merge_lock = Semaphore.make true;
rename_lock = Semaphore.make true;
sync_lock = Semaphore.make true;
writer_lock;
pending_cancel = false;
}
Expand Down
5 changes: 4 additions & 1 deletion src/index_intf.ml
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,10 @@ module type Private = sig

- [`Before_offset_read]: before reading the generation number and the
offset.
- [`After_offset_read]: after reading the generation number and offset. *)
- [`After_offset_read]: after reading the generation number and offset.

The hooks run while [sync] holds the lock of the instance, so they must
not call [find], [mem] or [sync] on it. *)

val log : t -> (key * value) list option
val log_async : t -> (key * value) list option
Expand Down
Loading
Loading