Reading full client body in NGX_HTTP_CONTENT_PHASE

[pschyska]

We need to read the full client bodies to implement a transparent tunneling protocol on top of TLS and HTTP. I could not find any example in rust on how to do that.
I can manually loop over the ngx_chain, and this enabled me to collect the client body when it is not being spilled over to a temp file:

pub fn handler(request: &mut Request) -> Status {
    let config = Module::location_conf(request).expect("location config is none");

    match config.asl {
        true => {
            let rc = unsafe {
                ngx_http_read_client_request_body(request.as_mut(), Some(client_request_body_cb))
            };

            if rc >= NGX_HTTP_SPECIAL_RESPONSE.try_into().unwrap() {
                Status(rc)
            } else {
                Status::NGX_DONE
            }
        }
        false => Status::NGX_DECLINED,
    }
}

extern "C" fn client_request_body_cb(request: *mut ngx_http_request_t) {
    let body = unsafe { *(*request).request_body };
    let mut chain = unsafe { *body.bufs };
    let mut bufs: Vec<MemoryBuffer> = vec![];
    let first = MemoryBuffer::from_ngx_buf(body.buf);
    if first.len() > 0 {
        bufs.push(first);
    }
    loop {
        if !chain.buf.is_null() {
            let t = MemoryBuffer::from_ngx_buf(chain.buf);
            if t.len() > 0 { // NOTE: There might be a slight bug/confusing behaviour when creating a MemoryBuffer from a len 0 ngx_buf, it panics at runtime
                bufs.push(t);
            }
        }
        if chain.next.is_null() {
            break;
        }
        chain = unsafe { *chain.next };
    }
}

I'm not 100% sure if that is the right way to handle a ngx_chain. Do I also need for account for last_buf and last_in_chain on the buffers themselves? Also: do I need to evaluate buf.memory and decide if I need a MemoryBuffer vs. TemporaryBuffer?

However, when nginx decides to spill the client body to temp, how do I handle this? My callback seems to be called with request.request_body having:

• a len 0 .buf
• a ngx_chain with a single element which also has len 0 in .bufs
• a ngx_temp_file with a .file.name pointing into my client body temp folder

The issue is that I've never seen that file in my file system. I also can't make sense of the temp_file.file.info.st_size, which is 0, and temp_file.offset, which is ~1MiB, which seems to correspond to the body I sent (curl -v http://localhost:8000/asl --data-binary @tempfile with tempfile being 1MiB). I would have expected the exact opposite: offset 0 and size ~1MiB. Is the file maybe not written yet, and I have to drive the aio threads, e.g., machinery myself to schedule the write and pass another callback when that is finished? Point in case: I think ngx_http_read_client_request_body returns -2, which should be NGX_AGAIN afaik. I tried to use ngx_read_file like this:

        let mut buffer: *mut u8 = unsafe {
            *Request::from_ngx_http_request(request)
                .pool()
                .alloc(size)
                .cast()
        };
        let x = unsafe {
            ngx_read_file(
                &mut temp_file.file,
                buffer,
                size,
                size.try_into().unwrap(),
            )
        };
        let v = unsafe {
            Vec::from_raw_parts(buffer, size.try_into().unwrap(), size.try_into().unwrap())
        };

… but it always reads 0 bytes.

Has anyone done this in Rust already and can give me some pointer? Thanks!

[bavshin-f5]

You don't need to copy from the chain, you can use the data in-place or convert it to a Vec of &[u8] slice references (see ngx_output_chain_to_iovec for a rough example). The chain's data is guaranteed to live until you free it, pass to a chain writer (important for phase handlers other than content handler), or terminate the request.
The reallocation and copying is only necessary if you want a single contiguous buffer, and even then you'd want to avoid TemporaryBuffer/MemoryBuffer and use something like Vec backed by ngx_pool_t instead.

temp_file.file.info is a zero-initialized reserved storage for ngx_fd_info (fstat(2)) results. ngx_http_read_client_request_body does not use ngx_fd_info, so the field remains uninitialized.
temp_file.offset is an offset for the next write operation, i.e. number of bytes already written to the temporary file. ngx_read_file(temp_file->file, buf, temp_file->offset, 0) should get you the whole request body. Which I wouldn't generally recommend, because using smaller read buffers and respecting flow control would be more efficient.
ngx_buf_t::{in_file, file, file_pos, file_last} represent the same state. Theoretically, file_pos — start of the request body in the file — can be > 0. Also theoretically, there might be more than one file-backed buffer in the chain. ngx_http_request_body.c doesn't seem to generate such buffers.

Not knowing what you want to do with the data, I can only suggest to check the corresponding section of nginx development guide, and examples of chain manipulation in src/core/ngx_buf.c, src/core/ngx_output_chain.c, src/http/ngx_http_upstream.c and src/http/modules/ngx_http_proxy_module.c.
We don't have a decent Rust API or even Rust examples for any of these operations right now.

---

Also in your code snippet,

        let v = unsafe {
            Vec::from_raw_parts(buffer, size.try_into().unwrap(), size.try_into().unwrap())
        };

seems to be a guaranteed double free, because both ngx_pool_t and Vec will attempt to free the underlying buffer.

I'd try to write this and similar fragments as

let v: Vec<u8, _> = Vec::new_in(request.pool());
v.try_reserve_exact(size)?;
unsafe {
  let n = ngx_read_file(&mut temp_file.file, v.as_mut_ptr().cast(), core::cmp::min(v.capacity(), size), 0);
  if n < 0 {
      return Err(...);
  }
  v.set_len(n as usize);
}