implement perplexity in llama-server

[magikRUKKOLA]

server: add perplexity evaluation endpoints

This PR adds streaming perplexity calculation to llama-server, matching the logic and output format of the standalone llama-perplexity tool.

New HTTP endpoints

Method	Path	Description
POST	/perplexity	Submit a batch of tokens (or raw text) to evaluate. The server buffers input and evaluates full n_ctx-sized chunks, returning running PPL.
GET	/perplexity/state	Query current accumulated state (nll, nll2, count, perplexity, chunk_index).
POST	/perplexity/save	Serialize state to disk (binary format, versioned).
POST	/perplexity/load	Restore state from disk.
POST	/perplexity/reset	Clear all accumulated state.

Behavior

• Non-strided mode only (ppl_stride = 0).
• Evaluates the second half of each context window (first = n_ctx / 2), identical to perplexity.cpp.
• Supports resume: send tokens in chunks, save state, restart the client later, load state, and continue from the last fully-evaluated chunk.
• State file format is versioned (v2) and backward-compatible with v1.

Client usage

A reference bash client (llama-perplexity.sh) is included that:

1. Tokenizes the entire input file once via /tokenize.
2. Streams 512-token slices to /perplexity.
3. Prints compact [chunk]PPL, output matching the standalone tool.
4. Handles Ctrl+C gracefully by saving server state before exit.
5. On restart, loads state and resumes from the exact chunk where it stopped.

Testing

• Verified bit-exact PPL values against llama-perplexity for the first 30+ chunks of wiki.test.raw.
• Tested save/load/resume across server restarts.

Notes / follow-up

• Debug logging is gated behind LLAMA_LOG_DEBUG.
• The implementation currently uses a single sequence (n_seq = 1). Parallel sequence evaluation can be added later if needed.

[ikawrakow]

Why would we want to add yet another functionality to the already massively over bloated server? If save/resume a perplexity calculation is something that is useful to someone, it can be added to the perplexity tool. Or not?

[magikRUKKOLA]

If save/resume a perplexity calculation is something that is useful to someone, it can be added to the perplexity tool. Or not?

Well, I was thinking about the use-case when one would swap(/reload) a certain tensors and check the PPL without shutting down the server and starting the llama-perplexity.

The 'save/resumewill be useful for the Thireus benchmarks since its will become easy to contribute the PPL data. For this case such functionality can be indeed implemented inllama-perplexity`.

I will convert this PR into draft to think about the issue some more.