Daemon crashes during artifact upload for large projects (Connection reset by peer)

[xinding33]

Description

When running hurry cargo build on large projects with many compilation units (e.g., ~800 units), the daemon crashes during the artifact upload phase, resulting in a "Connection reset by peer" error.

Environment

• OS: macOS (Darwin 24.6.0)
• Hurry: v0.4.7

Steps to Reproduce

1. Clone a large Rust project with many dependencies:

   git clone https://github.com/lancedb/lancedb
   cd lancedb

2. Run hurry cargo build:

   hurry cargo build

3. Wait for the build to complete. During the "Uploading cache" phase, the daemon crashes.

Expected Behavior

Artifacts should upload successfully without the daemon crashing.

Actual Behavior

The daemon process dies mid-upload, producing the following error:

Error:
   0: send upload status request to daemon at: http://[::1]:51379/api/v0/cargo/status
   1: error sending request for url (http://[::1]:51379/api/v0/cargo/status)
   2: client error (SendRequest)
   3: connection error
   4: Connection reset by peer (os error 54)

Location:
   packages/hurry/src/bin/hurry/cmd/cargo/build.rs:271

Daemon context:
   DaemonContext { pid: 72405, url: "[::1]:51379", log_file_path: /Users/xinding/Library/Caches/com.attunehq.hurry/v2/hurryd.72405.log }

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: hurry::cmd::cargo::build::wait_for_upload with request_id=2ce0491d-42eb-4a03-962c-49d706b75cac progress=[7.375µs] [0/794] Uploading cache
      at packages/hurry/src/bin/hurry/cmd/cargo/build.rs:247
   1: hurry::cmd::cargo::build::exec with options=Options { api_url: https://app.hurry.build/, api_token: Some([redacted]), skip_backup: false, skip_build: false, skip_restore: false, async_upload: false, help: false, argv: [] }
      at packages/hurry/src/bin/hurry/cmd/cargo/build.rs:108

Additional Context

• Daemon log file is empty (0 bytes): /Users/xinding/Library/Caches/com.attunehq.hurry/v2/hurryd.72405.log

[xinding33]

Also happens with https://github.com/influxdata/influxdb.

[xinding33]

Additional Investigation Findings

After deeper investigation, I've identified several potential root causes:

1. SIGPIPE Vulnerability (Most Likely Cause)

The daemon is spawned with piped stdout/stderr (Stdio::piped()), but the parent process immediately drops the Child handle without keeping the pipes alive:

// packages/hurry/src/cargo/cache.rs:67-73
let mut cmd = tokio::process::Command::new(hurry_binary);
cmd.arg("daemon")
    .arg("start")
    .stdout(Stdio::piped())
    .stderr(Stdio::piped());

cmd.spawn()?;  // <-- Child handle is dropped immediately, closing pipes!

The code comment at start.rs:53-58 explicitly acknowledges this problem:

"the invoking terminal exiting causes the STDOUT and STDERR pipes of this program to close, which means the process crashes with a SIGPIPE if it attempts to write to them."

While SIGHUP is handled, SIGPIPE is NOT handled. Any write to stdout/stderr after the pipes are closed will kill the daemon.

Potential SIGPIPE triggers in daemon context:

• tracing-subscriber's EnvFilter can emit eprintln! warnings (see builder.rs:228)
• Initial log statements at start.rs:60-61 use the CLI logger (stderr) before the file logger takes over
• Any library code that writes to stdout/stderr

2. Axum Body Size Limit

Axum has a 2MB default body limit. With 794 units for lancedb (and presumably similar for influxdb), the CargoUploadRequest JSON payload may exceed this.

However, this should return a 413 error, not crash the process. So this alone can't explain the crash.

3. Log Files Are Empty Because...

The daemon creates the log file but logs are filtered by HURRY_LOG env var. Without HURRY_LOG set, only WARN/ERROR level logs are shown. The daemon's normal operation uses info! and debug! levels, which would be filtered out.

Suggested Fixes

1. Ignore SIGPIPE in the daemon:

   #[cfg(unix)]
   unsafe {
       signal_hook::low_level::register(signal_hook::consts::SIGPIPE, || {})?;
   }

2. Don't pipe stdout/stderr when spawning daemon, or keep the Child handle alive:

   cmd.stdout(Stdio::null())
      .stderr(Stdio::null());

3. Disable or increase body size limit for the upload endpoint:

   .layer(DefaultBodyLimit::disable())
   // or
   .layer(DefaultBodyLimit::max(100 * 1024 * 1024))  // 100MB

4. Set a default log level for the daemon so logs are actually written:

   .with_default_directive(LevelFilter::INFO.into())

Reproduction Affects Multiple Projects

This issue has been confirmed with:

• https://github.com/lancedb/lancedb (794 units)
• https://github.com/influxdata/influxdb

Both are large Rust projects with many dependencies.