FileSystemWatcher on Linux: event-processing thread exits silently after interleaved directory renames (unpaired IN_MOVED_FROM handling) - watcher stops receiving events, no Error raised.

[giancarlo-tomazelli]

EDIT: original text described a shared-instance design (read from main, which is the .NET 11 implementation); architecture claims removed - see comments.

Description

On .NET 10.x (Linux), an armed FileSystemWatcher is serviced by an inotify fd and a processing thread named ".NET File Watcher".

Under an easily produced filesystem workload - two directory renames applied back-to-back where the names collide (lib -> src while code -> lib) - the kernel delivers rename halves whose IN_MOVED_FROM/IN_MOVED_TO cookies don't pair up in one read window. After reading such a pair, the processing thread removes a child watch (inotify_rm_watch) and then exits cleanly (exit(0)) a few ms later, closing the inotify fd - while live watches are still registered and the managed FileSystemWatcher is still armed (EnableRaisingEvents == true).

From that moment the watcher receives no further events, ever. No Error event is raised, no exception surfaces - the failure is indistinguishable from "nothing changed on disk". The affected watcher never recovers.

.NET 7 is immune on the same machine and workload.

Reproduction Steps

Single-file console app, BCL only. It arms one FileSystemWatcher (IncludeSubdirectories = true) on a temp tree, runs the colliding-rename chain in tight bursts alongside light background file writes, and checks liveness with a canary file at the watched root. It prints its own /proc post-mortem (inotify fd count + presence of the ".NET File Watcher" thread).

dotnet publish -f net10.0 -r linux-x64 --self-contained -o out-net10
./out-net10/fswkill            # dies within seconds; exit 2
dotnet publish -f net7.0 -r linux-x64 --self-contained -o out-net7
./out-net7/fswkill             # control; survives 5000 rounds; exit 0

fswkill.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net7.0;net10.0</TargetFrameworks>
    <AssemblyName>fswkill</AssemblyName>
    <Nullable>disable</Nullable>
    <ImplicitUsings>disable</ImplicitUsings>
  </PropertyGroup>
</Project>

Program.cs (full source)

// fswkill - reproducer for the .NET 10 Linux FileSystemWatcher silent death
// under interleaved (name-colliding) directory renames.
// Exit codes: 0 = survived, 2 = watcher died silently, 3 = died but Error fired.

using System;
using System.IO;
using System.Threading;

class Program
{
    static long sEvents;
    static volatile string sLastCanarySeen = "";
    static volatile bool sErrorFired;

    static int Main(string[] args)
    {
        int maxRounds = args.Length > 0 ? int.Parse(args[0]) : 5000;
        string root = Path.Combine(
            args.Length > 1 ? args[1] : Path.GetTempPath(),
            "fswkill-" + Environment.ProcessId);

        Console.WriteLine("runtime: " + Environment.Version + " | root: " + root);

        string code = Path.Combine(root, "code");
        string lib = Path.Combine(root, "lib");
        string src = Path.Combine(root, "src");

        Directory.CreateDirectory(Path.Combine(code, "child"));
        Directory.CreateDirectory(Path.Combine(lib, "child"));
        Directory.CreateDirectory(Path.Combine(root, "beacon"));
        File.WriteAllText(Path.Combine(code, "child", "f.txt"), "x");
        File.WriteAllText(Path.Combine(lib, "child", "f.txt"), "x");

        using FileSystemWatcher fsw = new FileSystemWatcher(root);
        fsw.IncludeSubdirectories = true;
        fsw.NotifyFilter = NotifyFilters.FileName | NotifyFilters.DirectoryName
            | NotifyFilters.LastWrite | NotifyFilters.CreationTime | NotifyFilters.Size;

        FileSystemEventHandler onAny = delegate (object s, FileSystemEventArgs e)
        {
            Interlocked.Increment(ref sEvents);
            if (e.Name != null && e.Name.StartsWith("canary_"))
                sLastCanarySeen = e.Name;
        };
        fsw.Created += onAny;
        fsw.Changed += onAny;
        fsw.Deleted += onAny;
        fsw.Renamed += delegate { Interlocked.Increment(ref sEvents); };
        fsw.Error += delegate (object s, ErrorEventArgs e)
        {
            sErrorFired = true;
            Console.WriteLine("!! Error event fired: " + e.GetException());
        };
        fsw.EnableRaisingEvents = true;

        // background lock-file-style traffic keeps the inotify reads chunked
        bool stop = false;
        Thread writer = new Thread(delegate ()
        {
            string beacon = Path.Combine(root, "beacon", "lck");
            int i = 0;
            while (!Volatile.Read(ref stop))
            {
                try { File.WriteAllText(beacon, i++.ToString()); } catch { }
                Thread.Sleep(1);
            }
        });
        writer.IsBackground = true;
        writer.Start();

        Console.WriteLine("baseline: " + Vitals());

        for (int round = 1; round <= maxRounds; round++)
        {
            // the colliding rename chain (name reuse is the essential ingredient)
            try
            {
                Directory.Move(lib, src);
                Directory.Move(code, lib);
                Directory.Move(lib, code);
                Directory.Move(src, lib);
            }
            catch (Exception ex)
            {
                Console.WriteLine("rename hiccup at round " + round + ": " + ex.Message);
                EnsureLayout(code, lib, src);
                Thread.Sleep(20);
            }

            if (round % 25 != 0)
                continue;

            if (!CanaryDelivered(root, round))
            {
                Volatile.Write(ref stop, true);
                Console.WriteLine("== VERDICT: watcher DEAD at round " + round
                    + " | events delivered: " + Interlocked.Read(ref sEvents)
                    + " | Error fired: " + sErrorFired);
                Console.WriteLine("post-mortem: " + Vitals()
                    + " (watcher still armed: " + fsw.EnableRaisingEvents + ")");
                return sErrorFired ? 3 : 2;
            }
        }

        Volatile.Write(ref stop, true);
        Console.WriteLine("== VERDICT: SURVIVED " + maxRounds + " rounds | events delivered: "
            + Interlocked.Read(ref sEvents) + " | " + Vitals());
        return 0;
    }

    static bool CanaryDelivered(string root, int round)
    {
        string name = "canary_" + round;
        try { File.WriteAllText(Path.Combine(root, name + ".txt"), "alive?"); }
        catch (Exception ex) { Console.WriteLine("canary write failed: " + ex.Message); }

        for (int waited = 0; waited < 2000; waited += 10)
        {
            if (sLastCanarySeen.StartsWith(name))
                return true;
            Thread.Sleep(10);
        }
        return false;
    }

    static void EnsureLayout(string code, string lib, string src)
    {
        try
        {
            if (!Directory.Exists(lib) && Directory.Exists(src)) Directory.Move(src, lib);
            if (!Directory.Exists(code) && Directory.Exists(src)) Directory.Move(src, code);
            if (!Directory.Exists(code)) Directory.CreateDirectory(Path.Combine(code, "child"));
            if (!Directory.Exists(lib)) Directory.CreateDirectory(Path.Combine(lib, "child"));
        }
        catch { }
    }

    static string Vitals()
    {
        if (!OperatingSystem.IsLinux())
            return "(vitals: non-linux)";

        int inotifyFds = 0;
        try
        {
            foreach (string fd in Directory.GetFiles("/proc/self/fd"))
            {
                try
                {
                    FileSystemInfo target = new FileInfo(fd).ResolveLinkTarget(false);
                    if (target != null && target.FullName.Contains("inotify"))
                        inotifyFds++;
                }
                catch { }
            }
        }
        catch { }

        int watcherThreads = 0;
        try
        {
            foreach (string task in Directory.GetDirectories("/proc/self/task"))
            {
                try
                {
                    if (File.ReadAllText(Path.Combine(task, "comm")).Contains(".NET File Watch"))
                        watcherThreads++;
                }
                catch { }
            }
        }
        catch { }

        return "inotify fds: " + inotifyFds + ", '.NET File Watcher' threads: " + watcherThreads;
    }
}

Expected behavior

Events keep flowing for the still-registered watches - or, if event processing must stop, Error is raised on the affected FileSystemWatcher so the consumer can recover. A silent permanent stop while the watcher is armed should not be possible.

Actual behavior

$ ./out-net7/fswkill             # runtime 7.0.20, same machine
== VERDICT: SURVIVED 5000 rounds | events delivered: 21535 | inotify fds: 1, '.NET File Watcher' threads: 1
exit: 0

$ ./out-net10/fswkill            # runtime 10.0.11
baseline: inotify fds: 1, '.NET File Watcher' threads: 1
== VERDICT: watcher DEAD at round 125 | events delivered: 456 | Error fired: False
post-mortem: inotify fds: 0, '.NET File Watcher' threads: 0 (watcher still armed: True)
exit: 2

strace of the processing thread's final moments:

21037 20:56:43.795823 read(38<anon_inode:inotify>, "\4\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0lck\0\0\0\0\0\0\0\0\0\0\0\0\0", 8192) = 32
21037 20:56:43.795988 read(38<anon_inode:inotify>, "\1\0\0\0@\0\0@\31n\1\0\20\0\0\0lib\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 64
21037 20:56:43.797961 inotify_add_watch(38<anon_inode:inotify>, "/tmp/fswkill-21029/src", IN_MODIFY|IN_ATTRIB|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_ONLYDIR|IN_DONT_FOLLOW|IN_EXCL_UNLINK) = 2
21037 20:56:43.798056 read(38<anon_inode:inotify>, "\4\0\0\0\2\0\0\0\0\0\0\0\20\0\0\0lck\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 192
21037 20:56:43.798330 inotify_add_watch(38<anon_inode:inotify>, "/tmp/fswkill-21029/lib", IN_MODIFY|IN_ATTRIB|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_ONLYDIR|IN_DONT_FOLLOW|IN_EXCL_UNLINK) = 2
21037 20:56:43.799258 inotify_rm_watch(38<anon_inode:inotify>, 3) = 0
21037 20:56:43.799470 inotify_rm_watch(38<anon_inode:inotify>, 2) = 0
21037 20:56:43.803517 exit(0)           = ?
21037 20:56:43.803648 +++ exited with 0 +++

• The lck reads are the reproducer's background writer (IN_MODIFY).
• The 64-byte read on the root watch (wd 1) carries IN_MOVED_FROM|IN_ISDIR, cookie 0x16E19, name lib - plus a second event in the same read (strace truncates the payload display): the interleaved rename pair.
• The inotify_add_watch calls that follow are the watcher re-adding watches for the moved directories while still processing.
• Then two inotify_rm_watch calls and, 4 ms after the fatal read, exit(0) - clean, voluntary, no signal. No further reads on the fd ever; the process keeps running (the reproducer's verdict line prints afterwards), the watcher is still armed, and Error never fires..

By elimination on the exit path (no exception or Error, no overflow event, the rm_watch immediately preceding), the thread's exit follows directly from the unmatched-MOVED_FROM handling.

Regression?

Yes. .NET 7 (7.0.20) is immune on the same machine and workload (per-instance fd + thread design). Reproduced on 10.0.10 and 10.0.11 (latest servicing). .NET 8/9 not tested.

Known Workarounds

None satisfying. We currently detect the death with an out-of-band liveness probe (touch a watched path, require the event within a timeout) and rebuild the watcher - there is no callback-based way to notice it.

Configuration

• .NET 10.0.10 and 10.0.11, linux-x64, self-contained deployments
• Ubuntu x86_64 (also reproduced on Amazon EC2 images), ext4
• Not architecture-specific as far as we know; Windows unaffected

Other information

Real-world impact: we ship a version control system; a merge that applies "both sides moved directories" produces exactly this rename pattern in users' working trees, silently killing change detection for every workspace open in the process.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[adamsitnik]

Since the rewrite of the Linux FileSystemWatcher (motivated by #62869), all watchers in a process share one inotify fd and one processing thread (".NET File Watcher").

Reproduced on 10.0.10 and 10.0.11 (latest servicing).

The mentioned rewrite (#117148) was never backported to .NET 10, it was released as part of .NET 11.

@giancarlo-tomazelli could you please try .NET 11? Preview 7 or RC 1 (when available) should be fine.

cc @tmds

[dotnet-policy-service]

This issue has been marked needs-author-action and may be missing some important information.

[giancarlo-tomazelli]

Thanks @adamsitnik. Two updates - the verification you asked for, and a correction.

.NET 11: not reproducible. Same box, same reproducer that kills 10.0.10/10.0.11 within seconds:

$ ./out-net11/fswkill
runtime: 11.0.0 | root: /tmp/fswkill-20996        (SDK 11.0.100-preview.7.26381.103)
baseline: inotify fds: 1, '.NET File Watcher' threads: 1
== VERDICT: SURVIVED 5000 rounds | events delivered: 23025 | inotify fds: 1, '.NET File Watcher' threads: 1
exit: 0

Correction to my description (title updated accordingly): you're right about the architecture - I had described the shared-instance design after reading main, which post-#117148 is the .NET 11 implementation.

So, the affected range appears to be 10.0.x, with 7.0 unaffected and 11.0 unaffected via the rewrite. Since 10.0 is LTS: could a servicing fix be considered? Happy to validate any patched build or daily against this repro.

[adamsitnik]

So, the affected range appears to be 10.0.x, with 7.0 unaffected and 11.0 unaffected via the rewrite. Since 10.0 is LTS: could a servicing fix be considered? Happy to validate any patched build or daily against this repro.

Yes, we are going to consider that. I'll try to prepare a fix first and depending on how complex and risky it is proceed with the backport procedure.

@giancarlo-tomazelli big thanks for a quick validation!