Scriptable processor for I/O (http, filesystem, etc)

[daniel-falk]

Use Case

This issue is created as a place to brainstorm about a new scriptable processor that can be used to enrich metrics with data, e.g. through HTTP requests, implement complex multi-API call, etc.

The inspiration for this is the processors.starlark plugin. That one, is however deterministic and sandboxed. Two features that are great for most use cases, but that also prevents it from being used for the kind of processing that is targeted by this issue.

The general idea is to not try to predict all patterns that a user might want to use (different authentication mechanisms, multi-API calling, response caching, templated URLs and payloads, advanced response parsing, etc.), but instead let them script the process.

Can't this already be done with the processors.execd plugin?

Yes, it can. There are however multiple drawbacks with that. One being that the telegraf workflow is no longer self-contained, it depends on other tools being installed in the environment. A bash/shell script does not work in Windows. Python installations differ from system to system, so does the libraries that are installed. Compiled binaries depend on the system architecture.

The same kind of problems is also affecting the #3813 GRPC proposal.

Keeping a predictable (and minimal) runtime encloused in the telegraf binary would make it a lot more portable and easier to manage. The workflows would also be more standardized and easier to share between different users, projects and domains.

Can we extend the Starlark processor with a sandbox: false option?

Yes, technically this would probably be possible and from a user perspective, I think this would be the best option. The implementation would however be complex since Starlark does not have support for any kind of I/O operation. We would need to implement these functions in go and expose them when spawning the environment. Not hard, but the complexity quickly grows with edge cases, options and features.

Can we embed another language that already supports I/O?

Yes, and this might be the best option. The drawback is that it will be yet another language that users needs to learn. Reusing code between the Starlark processor and the non-sandboxed processor will not be possible which I see as a big disadvantage.

Some options are:

• goja: A JavaScript engine in pure Go
• tengo: A fast script language for Go
• gopher-lua: VM and compiler for LUA in Go

I did some quick benchmarks on the binary size when building a simple Go application with the runtime embeded.

Note: see dicussion points below, the true size increase of the binary would probably be much smaller for all of these.

Binary	Size	Overhead
baseline-embed	869 KB	—
tengo-embed	6.0 MB	+5.2 MB
starlark-embed	6.4 MB	+5.6 MB
wazero-embed	7.2 MB	+6.4 MB
goja-embed	13 MB	+12 MB
gopher-lua-embed	5.7 MB	+4.8 MB
gpython-embed	7.9 MB	+7.0 MB

Is JavaScript a good option?

I think this is one of the strongest contestants:

• Already suports I/O through standard libraries (HTTP requests, file I/O, etc.)
• Most people are familiar with JavaScript

The main drawback I see is that the binary is the biggest of all of them.

Is LUA a good option?

This is one of my other favourites. LUA is a very old and stable language for embedding. Many people know it, but not at all as many as knows JavaScript or Python. It's powerful, and it's the smallest of all.

It does have a standard lib for file I/O, but the LuaSocket and LuaSec for network requests are not usable since they are implemented in C and would need CGO enabled.

Is Tengo a good option?

I like that it's small in size and seems very fast. It is however a bespoke language that none knows about. It does support file I/O through its standard lib, but not web requests, so we would need to implement that.

Is WebAssembly a good option?

It's a very interesting tool, because it allows you to implement the scripts in many different languages. I also think it has good support for configurable sandboxing which would be great.

But, a dealbreaker for me is that it needs to be compiled. Creating a new processor should not require a build environment and a compilation step. The script should be interpreted from source. I don't think this would be possible with WebAssembly? I also think we should strive for high-level script languages that are easy to learn. Go, Rust, C, etc. is not.

Can we embedd Python?

No. It would be nice beacuse it is a high-level language that many are familiar with and code would be more or less reusable between Python processors and Starlark processors.

The massive size is however a deal-breaker and it's so dependent on the library ecosystem so it would not make a lot of sense.

Can we use a smaller Python-like interpreter?

Probably not. Something like MicroPython would be great and small enough. But, as far as I know, the only one of them that is not depending on CGO is GPython. GPython does not support I/O and seems to be aging and poorly maintained. From what I have seen, telegraf does not allow CGO usage.

Other use cases

There's probably countless use cases for this processor apart from metric enriching and web requests. One issue that we have had is managing sqlite databases that is created by the outputs.sql plugin. They grow in size without any way to clean them. Sure, we could add dedicated implementation for that in the sql plugin, but there are so many different ways that you might want to handle it, e.g. remove after a certain age, after specific number of lines, when a maximum size is reached, when a maximum percentage of disk space is used etc. The drawback is that none of the afforemensioned language interpreters include sqlite libraries in their standard lib. Exactly the same need also applies to the outputs.remotefile plugin when used with the local filesystem as a target. This would just be simple file I/O.

Can we have configurable sandboxing

Yes, we can and I think we should. Most of these examles require us to expose the I/O libraries when spawning the runtime (even if they come from the standard library in the language). This means that we could have options such as allow_filesystem_access: true and allow_network_access: true which I think would be a good security practice.

Discussion points

• What plugin type should this be?: Optimally, we would have both an input, processor and output for this. A processor can however be used as an interval based input by using a trigger metric (but not as an event driven input), and it can be used as an output by ignoring the emitted metrics (with the drawback that it runs together with the processors at a specific order, not after the processing without consuming it as an output plugin does). As far as I know, there is no method in the Telegraf build system to share code between plugins.
• How can multiprocessing be supported?: For a processor that should handle network or file I/O to enrich metric, pooling or parallelization might be needed for high dataflows or slow networks. We should investigate how this would work with the different options.
• How large is actually the runtimes: In my benchmark, Starlark is about 6MB. But when I did benchmarks of building telegraf with and without it, I could see hardly any difference. I suspect this might be because all runtimes are pulling in standard Go functionality that telegraf is already using. It would be good to rerun all benchmarks in a realistic environment.
• Is implementing host functions a bad option?: From a code complexity persepctive, it would be great to use a language that already implements I/O functions to avoid reinventing the wheel. But from a standardization persepctive, it might be better to implement host functions for network requests so that we can reuse the same TLS/encryption library as other plugins are using. This makes it easier to add security updates such as e.g. TPM support accross all plugins in the same change.

Expected behavior

A self-contained plugin for I/O exists.

Actual behavior

It does not exist.

Additional info

No response