zombie with `LESSOPEN="|..."`

[vinc17fr]

When I use LESSOPEN with a pipe and the preprocessor outputs something, a zombie remains as a child of the less process.

This can simply be reproduced with something like:

LESSOPEN="|echo %s" less /etc/os-release

This seems to be an old bug (at least less 590 is affected), and this still occurs with less 700 (latest beta). Tested under Debian.

[avih]

I presume this process remains after you quit "less"?

Or does it terminate with "less"?

I don't think I can reproduce it (tried on Alpine linux). At least I don't think I identify unexpected processes using htop.

Which process is that? /bin/sh ? "less"? something else?

How do you identify that such zombie remains? (specific command which would show it)

[avih]

Or does it terminate with "less"?

When trying this:

LESSOPEN="|echo %s" less /etc/os-release

(for what it's worth /bin/echo is a symlink to coreutils in this setup)

While "less" is still running, I see this in htop:

 6140 user         20   0  1616  1412  1100 S   0.0  0.0  0:00.00    │        └─ ./less /etc/os-release
 6141 user         20   0     0     0     0 Z   0.0  0.0  0:00.00    │           └─ bash

Is bash here the zombie you're referring to?

However, when I quit "less", that process goes away too.

I wouldn't call it a zombie because it terminates with less, but I agree that one might expect it to terminate once less dequeued all its output, even if "less" didn't terminate yet.

But I'll leave it to @gwsw to reply.

[gwsw]

Less opens the pipe with popen() and calls pclose() when less exits, or when a different input file is opened. I'm not sure if it would be safe to do it otherwise, since it's not easy to tell when the pipe writer is finished sending data. I seem to recall situations where even after reading EOF from a pipe, more data could arrive later.

[vinc17fr]

I presume this process remains after you quit "less"?

The process has already terminated while less is still running. That's basically the definition of a zombie: the process has terminated, but it still takes an entry in the process table because the parent process has not acknowledged its termination with a system call of the wait* family. For instance, it appears like that in the ps -ef output:

vinc17   1625002 1625001  0 21:12 pts/22   00:00:00 [sh] <defunct>

(the <defunct> means that it is a zombie).

Once the parent process terminates, zombies always disappear (or this is either a bug in the kernel or there is a major hardware issue). Here, no problems when less terminates.

Less opens the pipe with popen() and calls pclose() when less exits, or when a different input file is opened.

I suppose that popen runs the command with sh -c, and this sh process has terminated, which gives the zombie, until pclose is called.

I'm not sure if it would be safe to do it otherwise, since it's not easy to tell when the pipe writer is finished sending data. I seem to recall situations where even after reading EOF from a pipe, more data could arrive later.

Well, when I run

{ echo foo ; sleep 3 ; echo bar ; } | less

I can see that less is still waiting for data just after foo is output.

The read(2) man page under Linux says "zero indicates end of file" for the return value, so I don't see how more data could arrive later. In the POSIX spec for read():

When attempting to read from an empty pipe or FIFO:

• If no process has the pipe open for writing, read() shall return 0 to indicate end-of-file.
• If some process has the pipe open for writing and O_NONBLOCK is set, read() shall return -1 and set errno to [EAGAIN].
• If some process has the pipe open for writing and O_NONBLOCK is clear, read() shall block the calling thread until some data is written or the pipe is closed by all processes that had the pipe open for writing.

[gwsw]

I think the case I was remembering was using a named pipe rather than an ordinary pipe. In a case like

mkfifo fifo
echo foo >>fifo; sleep 3; echo bar >>fifo

If less reads fifo, it will read EOF after the first echo exits, but then can receive more data from the second echo if it continues to read the fifo.

Anyway, I'm not sure that's relevant. I think you're suggesting that when less reads EOF on a LESSOPEN pipe, it should immediately call pclose. Am I understanding correctly?

[vinc17fr]

mkfifo fifo
echo foo >>fifo; sleep 3; echo bar >>fifo

If less reads fifo, it will read EOF after the first echo exits, but then can receive more data from the second echo if it continues to read the fifo.

But in this case, this is really an EOF, because any file descriptor that writes to the FIFO is closed when the first echo exits. So this is regarded as 2 different pipes. This can also be seen with cat, where the sleep value can normally be reduced (a small delay is just needed due to a race condition, to ensure that cat attempts a read while no file descriptors are open for writing):

$ while true ; do echo cat fifo ; cat fifo ; done
cat fifo
foo
cat fifo
bar
cat fifo

What happens for the first iteration (the other ones would be similar with additional sleep and echo >>fifo commands):

1. The echo writes to the FIFO, then blocks.
2. The cat fifo reads from the FIFO and outputs the string.
3. As the data have been read from the FIFO, the write(2) done by echo terminates, and echo terminates too.
4. The cat fifo attempts another read during the sleep (or just before), thus gets an end-of-file and terminates.

With "less", less -f fifo just sees foo (like what the first cat fifo does). But the bar is lost even before less terminates! This seems to be a bug in "less", which may have attempted to read additional data after the end-of-file (only the F command should do that). However, this is a corner case.

Anyway, I'm not sure that's relevant.

Indeed, for a normal pipe, once the writer has closed its file descriptor, I doubt that it can be reopened and connected to the pipe. In any case, this would not be a normal behavior for a lesspipe program, which should leave its standard output open as usual.

I think you're suggesting that when less reads EOF on a LESSOPEN pipe, it should immediately call pclose. Am I understanding correctly?

Yes.

[gwsw]

Can you explain the reasoning behind why you would like this change? I think it's a small change, but a potentially risky one, so I want to understand its importance.

Less can't pclose the pipe until it has read all the data in the pipe. So even after making this change, while the user is viewing the file, the piping process will still exist (either as a live process or a zombie) until the user reads all the way to the end of the file. The only change that would result from this change is that the zombie won't exist during the interval between reading to EOF and exiting less (or viewing another file). Why is this important?