You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _data-prepper/pipelines/configuration/processors/delete-entries.md
+80-10Lines changed: 80 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,10 +5,14 @@ parent: Processors
5
5
grand_parent: Pipelines
6
6
nav_order: 110
7
7
---
8
-
8
+
9
9
# Delete entries processor
10
10
11
-
The `delete_entries` processor deletes entries, such as key-value pairs, from an event. You can define the keys you want to delete in the `with-keys` field following `delete_entries` in the YAML configuration file. Those keys and their values are deleted.
11
+
The `delete_entries` processor removes entries, such as key-value pairs, from an event. Use the `with_keys` field to specify
12
+
the exact keys to delete. To delete keys that match a regular expression pattern, use the `with_keys_regex` field. You can
13
+
prevent deletion of specific events when using regular expressions by configuring the `exclude_from_regex` field.
14
+
15
+
The only way to configure both `with_keys` and `with_keys_regex` in the same `delete_entries` processor is by using the `entries` field.
12
16
13
17
## Configuration
14
18
@@ -21,22 +25,27 @@ This table is autogenerated. Do not edit it.
|`with_keys`| No | An array that specifies the keys of the entries to delete. |
31
+
|`with_keys_regex`| No | An array of regular expression (regex) patterns used to match the keys of entries to delete. |
32
+
|`exclude_from_delete`| No | A set of entries to exclude from deletion when using the `with_keys_regex` configuration. |
33
+
|`entries`| No | A list of entries to delete from the event. |
34
+
|`delete_when`| No | Defines the condition under which the deletion is performed. For example, `value="/some_key == null"` deletes the key only if `/some_key` is null or does not exist. |
35
+
|`delete_from_element_when`| No | Defines the condition that determines whether a key–value pair should be removed from each element in the list specified by `iterate_on`. The condition is evaluated for each element, and deletion occurs only if the element's key matches one defined in `with_keys` or `with_keys_regex` and satisfies the condition. |
36
+
|`iterate_on`| No | Specifies the key of the list field that contains objects to iterate over. The processor applies any configured deletion rules, such as `with_keys`, `with_keys_regex`, or `delete_from_element_when`, to each element in the list. |
27
37
28
38
## Usage
29
-
30
39
To get started, create the following `pipeline.yaml` file:
31
40
32
41
```yaml
33
42
pipeline:
34
43
source:
35
44
...
36
-
....
45
+
....
37
46
processor:
38
47
- delete_entries:
39
-
with_keys: ["message"]
48
+
with_keys: ["message"]
40
49
sink:
41
50
```
42
51
{% include copy.html %}
@@ -52,7 +61,68 @@ For example, before you run the `delete_entries` processor, if the `logs_json.lo
52
61
When you run the `delete_entries` processor, it parses the message into the following output:
53
62
54
63
```json
55
-
{"message2": "goodbye"}
64
+
{"message2": "goodbye", "message3": "test"}
65
+
```
66
+
67
+
68
+
### Deleting keys that match a pattern
69
+
70
+
First, create the following `pipeline.yaml` file:
71
+
72
+
```yaml
73
+
pipeline:
74
+
source:
75
+
...
76
+
....
77
+
processor:
78
+
- delete_entries:
79
+
with_keys_regex: [ "^tes.*" ]
80
+
exclude_from_delete: [ "test" ]
81
+
sink:
82
+
```
83
+
{% include copy.html %}
84
+
85
+
If your `logs_json.log` file contains the following event record:
Copy file name to clipboardExpand all lines: _data-prepper/pipelines/configuration/processors/select-entries.md
+51-6Lines changed: 51 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,13 +16,14 @@ Only the selected entries remain in the processed event and while all other entr
16
16
You can configure the `select_entries` processor using the following options.
17
17
18
18
| Option | Required | Description |
19
-
| :--- | :--- | :--- |
20
-
|`include_keys`| Yes | A list of keys to be selected from an event. |
21
-
|`select_when`| No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"'`, that will be evaluated to determine whether the processor will be run on the event. If the condition is not met, then the event continues through the pipeline unmodified with all the original fields present. |
19
+
| :--- |:---------| :--- |
20
+
|`include_keys`| No | A list of keys to be selected from an event. |
21
+
|`include_keys_regex`| No | A regular expression (regex) pattern that matches the keys to be selected from an event. |
22
+
|`select_when`| No | A [conditional expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/), such as `/some-key == "test"`, that is evaluated to determine whether the processor will be executed on the event. If the condition is not met, then the event continues through the pipeline unmodified, with all the original fields present. |
22
23
23
24
## Usage
24
25
25
-
The following example shows how to configure the `select_entries` processor in the`pipeline.yaml` file:
26
+
First, create the following`pipeline.yaml` file:
26
27
27
28
```yaml
28
29
pipeline:
@@ -41,15 +42,59 @@ pipeline:
41
42
For example, when your source contains the following event record:
The `select_entries` processor includes only `key1` and `key2`in the processed output:
53
+
After processing, only the keys listed in `include_keys` are retained in the event; all other keys are removed:
48
54
49
55
```json
50
56
{"key1": "value1", "key2": "value2"}
51
57
```
52
58
59
+
### Selecting keys using a regex
60
+
61
+
The following example shows how to configure the `include_keys_regex` field in the `pipeline.yaml` file:
62
+
63
+
```yaml
64
+
pipeline:
65
+
source:
66
+
...
67
+
....
68
+
processor:
69
+
- select_entries:
70
+
include_keys: [ "key1", "key2" ]
71
+
include_keys_regex: ["^ran.*"]
72
+
select_when: '/some_key == "test"'
73
+
sink:
74
+
```
75
+
{% include copy.html %}
76
+
77
+
For example, when your source contains the following event record:
78
+
79
+
```json
80
+
{
81
+
"message": "hello",
82
+
"key1": "value1",
83
+
"key2": "value2",
84
+
"some_key": "test",
85
+
"random1": "another",
86
+
"random2": "set",
87
+
"random3": "of",
88
+
"random4": "values"
89
+
}
90
+
```
91
+
92
+
The processor retains keys explicitly listed in `include_keys` and any keys matching the` include_keys_regex` pattern, removing all other keys from the event:
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4&size=40){kind=avatar}
0 commit comments