Skip to content

Commit 5cf1bd4

Browse files
dvarasani-crestdvarasani-crest
authored
docs: Userguide enhancements (#890)
This PR updates the doc for how_to_use section and add the documentation for requirement tests. ref: https://splunk.atlassian.net/browse/ADDON-73384
1 parent a968cd1 commit 5cf1bd4

File tree

5 files changed

+226
-20
lines changed

5 files changed

+226
-20
lines changed

‎docs/how_to_use.md‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# How To Use
22

33

4-
Create a test file in the tests folder
4+
1. Create a test file in the tests folder
55

66
```python
77
from pytest_splunk_addon.standard_lib.addon_basic import Basic
@@ -10,6 +10,10 @@ class Test_App(Basic):
1010
pass
1111
```
1212

13+
2. Create sample files and pytest-splunk-addon-data.conf (Can be skipped if the events are already present in the splunk environment)
14+
15+
- For detailed information on how to create samples and pytest-splunk-addon-data.conf, please refer [here](./sample_generator.md).
16+
1317
## Test Execution
1418

1519
There are three ways to execute the tests:
@@ -34,7 +38,6 @@ There are three ways to execute the tests:
3438
ARG SPLUNK_APP_ID=TA_UNKNOWN
3539
ARG SPLUNK_APP_PACKAGE=$SPLUNK_APP_PACKAGE
3640
RUN echo Splunk VERSION=$SPLUNK_VERSION
37-
COPY deps/apps /opt/splunk/etc/apps/
3841
COPY $SPLUNK_APP_PACKAGE /opt/splunk/etc/apps/$SPLUNK_APP_ID
3942
```
4043

@@ -225,7 +228,7 @@ The tool assumes the Splunk Add-on is located in a folder "package" in the proje
225228

226229
______________________________________________________________________
227230

228-
There are 3 types of tests included in pytest-splunk-addon are:
231+
There are 4 types of tests included in pytest-splunk-addon are:
229232

230233
1. To generate test cases only for knowledge objects, append the following marker to pytest command:
231234

@@ -244,9 +247,15 @@ There are 3 types of tests included in pytest-splunk-addon are:
244247
```console
245248
-m splunk_indextime --splunk-data-generator=<Path to the conf file>
246249
```
247-
250+
248251
For detailed information on index time test execution, please refer [here](./index_time_tests.md).
249252

253+
4. To generate only requirement tests, append the folowing marker to pytest command:
254+
255+
```console
256+
-m splunk_requirements
257+
```
258+
250259
- To execute all the searchtime tests together, i.e both Knowledge objects and CIM compatibility tests,
251260
append the following marker to the pytest command:
252261

@@ -317,7 +326,7 @@ The following optional arguments are available to modify the default settings in
317326
6. Options to separate event generation, event ingestion and test execution stage
318327
:
319328

320-
```console
329+
```console
321330
--tokenized-event-source=new|store_new|pregenerated
322331
```
323332

@@ -438,6 +447,6 @@ def splunk_setup(splunk):
438447

439448
How can this be achieved :
440449

441-
- Make json representation of the data models, which satisfies this [DataModelSchema](https://github.com/splunk/pytest-splunk-addon/blob/main/pytest_splunk_addon/standard_lib/cim_tests/DatamodelSchema.json).
450+
- Make json representation of the data models, which satisfies this [DataModelSchema](https://github.com/splunk/pytest-splunk-addon/blob/main/pytest_splunk_addon/cim_tests/DatamodelSchema.json).
442451
- Provide the path to the directory having all the data models by adding `--splunk_dm_path path_to_dir` to the pytest command
443-
- The test cases will now be generated for the data models provided to the plugin and not for the [default data models](https://github.com/splunk/pytest-splunk-addon/tree/main/pytest_splunk_addon/standard_lib/data_models).
452+
- The test cases will now be generated for the data models provided to the plugin and not for the [default data models](https://github.com/splunk/pytest-splunk-addon/tree/main/pytest_splunk_addon/data_models).

‎docs/index.md‎

Lines changed: 26 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,41 @@
11
# Overview
22

3-
pytest-splunk-addon is an open-source dynamic test plugin for Splunk Apps and Add-ons
4-
which allows the user to test knowledge objects, CIM compatibility and index time properties.
3+
pytest-splunk-addon is an open-source dynamic test plugin for Splunk [Apps](https://docs.splunk.com/Splexicon:App) and [Add-ons](https://docs.splunk.com/Splexicon:Addon)
4+
which allows the user to test [knowledge objects](https://docs.splunk.com/Splexicon:Knowledgeobject), [CIM](https://docs.splunk.com/Splexicon:CommonInformationModel) compatibility and [index time properties](https://docs.splunk.com/Splexicon:Indexedfield).
5+
6+
## Prerequisites
7+
8+
- Splunk App or Add-on package
9+
- Splunk instance with App or Add-on installed (Not required if using Docker)
10+
- Docker (Not required if using external Splunk instance)
511

612
## Support
713

814
- **Python**: 3.7
915
- **Platforms**: Linux, Windows and MacOS
1016

11-
## Features
12-
13-
- Generate tests for Splunk Knowledge objects in Splunk Technology Add-ons.
14-
- Generate tests for checking CIM compatibility in Splunk Technology Add-ons.
15-
- Generate tests for checking Splunk index-time properties in Splunk Technology Add-ons.
16-
- Validate your add-ons using Splunk + Docker.
17-
18-
## Release notes
19-
20-
Find details about all the releases [here](https://github.com/splunk/pytest-splunk-addon/releases).
21-
2217
## Installation
2318

2419
pytest-splunk-addon can be installed via pip from PyPI:
2520

2621
```console
2722
pip3 install pytest-splunk-addon
2823
```
24+
25+
## Features
26+
27+
The pytest-splunk-addon works by dynamically generating different types of tests for Splunk apps and add-ons by parsing their configuration files. Specifically, it looks at the .conf files in the provided Splunk app/add-on to create:
28+
29+
1. [Knowledge Object Tests](./field_tests.md)
30+
31+
2. [CIM Compatibility Tests](./cim_tests.md)
32+
33+
3. [Index-Time Field Tests](./index_time_tests.md)
34+
35+
4. [Requirement Tests](./requirement_tests.md)
36+
37+
Refer to the [How to use](./how_to_use.md) section for detailed instructions on running the tests.
38+
39+
## Release notes
40+
41+
Find details about all the releases [here](https://github.com/splunk/pytest-splunk-addon/releases).

‎docs/requirement_tests.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Requirement Tests
2+
3+
## Overview
4+
5+
- The tests are written with a purpose of testing the proper functioning of the fields extractions of the add-on.
6+
7+
______________________________________________________________________
8+
9+
To generate only requirement tests, append the folowing marker to pytest command:
10+
11+
```console
12+
-m splunk_requirements
13+
```
14+
15+
## Test Scenarios
16+
17+
**1. Fields should be extracted as defined in the cim_fields and other_mappings.**
18+
19+
```python
20+
test_requirements_fields[sample_name::<stanza>::host::<host>]
21+
```
22+
23+
Testcase verifies that fields are getting extracted with the values defined in the cim_fields and other_mappings in sample xml.
24+
Here &lt;stanza&gt; is the stanza name defined in the pytest-splunk-addon-data.conf and
25+
&lt;host&gt; is the host of the event
26+
27+
**Workflow:**
28+
29+
- Plugin gets the list of events and the fields defined under cim_fields and other_mappings for each event
30+
- For each event, plugin generates a search query to retrieve all the fields extracted by splunk.
31+
- For each event, all the fields defined in the cim_fields and other_mappings should be extracted with the values defined in sample xml.
32+
33+
**2. Event should be mapped with the datamodel defined in the sample xml**
34+
35+
```python
36+
test_datamodels[<datamodel>::sample_name::<stanza>::host::<host>]
37+
```
38+
39+
Testcase verifies that the event is mapped with the datamodel defined in the sample xml.
40+
Here &lt;datamodel&gt; is the name of the datamodel that the event should be mapped with, &lt;stanza&gt; is the stanza name defined in the pytest-splunk-addon-data.conf and &lt;host&gt; is the host of the event
41+
42+
**Workflow:**
43+
44+
- Plugin gets the events and its expected datamodel from the sample xml
45+
- Plugin generates a search query for that event and validates if the values of tags field are according to the datamodel defined in sample xml.
46+
47+
**3. All the cim recommended field should be extracted**
48+
49+
```python
50+
test_cim_fields_recommended[<datamodel>::sample_name::<stanza>::host::<host>]
51+
```
52+
53+
Testcase verifies that all the cim recommended fields of mapped datamodel are extracted for the event defined in the sample xml.
54+
Here &lt;datamodel&gt; is the name of the datamodel that the event should be mapped with, &lt;stanza&gt; is the stanza name defined in the pytest-splunk-addon-data.conf and &lt;host&gt; is the host of the event
55+
56+
**Workflow:**
57+
58+
- For each events, plugin gets the list of fields defined under cim_fields and missing_recommended_fields.
59+
- Field list retrieved from the sample xml should be equal to the list of recommended fields defined in the datamodel schema.
60+
61+
## Testcase Troubleshooting
62+
63+
In the case of test-case failure check if:
64+
65+
- The add-on to be tested is installed on the Splunk instance.
66+
- Splunk licence has not expired.
67+
- Splunk instance is up and running.
68+
- Splunk instance's management port is accessible from the test machine.
69+
70+
If all the above conditions are satisfied, further analysis of the test is required.
71+
For every test case failure, there is a defined structure for the stack trace.
72+
73+
```text
74+
- test_requirements_fields
75+
AssertionError: <<error_message>>
76+
Field | Splunk Value | Expected Value
77+
------| ------------ | --------------
78+
str | str | str
79+
Search = <Query>
80+
81+
- test_datamodels
82+
AssertionError: <<error_message>>
83+
Expected datamodel | Expected tags | Found datamodel | Found tags
84+
-------------------| ------------- | --------------- | ----------
85+
str | str | str | str
86+
Search = <Query>
87+
88+
- test_cim_fields_recommended
89+
AssertionError: <<error_message>>
90+
Search = <Query>
91+
```
92+
93+
Get the search query from the stack trace and execute it on the Splunk instance and verify which specific event is causing failure.

‎docs/sample_generator.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,90 @@ The sample generation & ingestion takes place before executing the testcases.
55
For index-time test cases, there are multiple metadata required about the sample file for which `pytest-splunk-addon-data.conf` must be created and provided to the pytest command.
66

77

8+
## Generate sample files
9+
Pytest splunk addon supports 2 different formats of sample files
10+
11+
**1. Text format**
12+
13+
- Sample events can be generated from the live events by replacing the field values with the tokens. Token name should be enclosed with ##.
14+
- Token replacement setting for this samples should be added to pytest-splunk-addon-data.conf to generate events
15+
- For example,
16+
```
17+
Example live event:
18+
<111> 2020-02-12T03:27:09+10:00 10.0.0.3 RT_FLOW: RT_FLOW_SESSION_CREATE: action=allowed transport=ICMP dest=10.0.0.1 dest_ip=10.0.0.1 dest_port=5048 dest_zone=DUMMY_ZONE app=app1
19+
20+
Sample created from live event:
21+
<111> ##timestamp_token## ##src_ip_token## RT_FLOW: RT_FLOW_SESSION_CREATE: action=##action_token## transport=##transport_type_token## dest=##dest_ip_token## dest_ip=##dest_ip_token## dest_port=##dest_port_token## dest_zone=##dest_zone_token## app=##app_token##
22+
```
23+
24+
**2. XML format**
25+
26+
- Generate sample xml file using the format provided [here](https://github.com/splunk/pytest-splunk-addon/blob/main/pytest_splunk_addon/sample_generation/schema.xsd)
27+
- Using xml format for the sample events, user can also execute requirement tests for the event for fields mentioned in the cim_fields and other_mappings
28+
- For example,
29+
30+
```
31+
Example live event:
32+
<111> 2020-02-12T03:27:09+10:00 10.0.0.3 RT_FLOW: RT_FLOW_SESSION_CREATE: action=allowed transport=ICMP dest=10.0.0.1 dest_ip=10.0.0.1 dest_port=5048 dest_zone=DUMMY_ZONE app=app1
33+
```
34+
35+
<details>
36+
<summary>Sample xml created from live event:</summary>
37+
```xml
38+
<?xml version="1.0" encoding="UTF-8"?>
39+
<device>
40+
<vendor>Test</vendor>
41+
<product>Test product</product>
42+
<version id="8.21.0"/>
43+
<event code="" name="SyslogEvent" format="syslog" >
44+
<transport type="syslog" host="10.0.0.3"/>
45+
<source>
46+
<comment/>
47+
</source>
48+
<raw>
49+
<![CDATA[<111> 2020-02-12T03:27:09+10:00 10.0.0.3 RT_FLOW: RT_FLOW_SESSION_CREATE: action=allowed transport=ICMP dest=10.0.0.1 dest_ip=10.0.0.1 dest_port=5048 dest_zone=DUMMY_ZONE app=app1]]>
50+
</raw>
51+
<cim>
52+
<models>
53+
<model>Network Traffic</model>
54+
</models>
55+
<cim_fields>
56+
<field name="action" value="allowed"/>
57+
<field name="dest" value="10.0.0.1"/>
58+
<field name="dest_ip" value="10.0.0.1"/>
59+
<field name="dest_port" value="5048"/>
60+
<field name="dest_zone" value="DUMMY_ZONE"/>
61+
<field name="app" value="app1"/>
62+
<field name="transport" value="ICMP"/>
63+
</cim_fields>
64+
<missing_recommended_fields>
65+
<field>bytes</field>
66+
<field>bytes_in</field>
67+
<field>bytes_out</field>
68+
<field>dvc</field>
69+
<field>rule</field>
70+
<field>session_id</field>
71+
<field>src</field>
72+
<field>src_ip</field>
73+
<field>src_port</field>
74+
<field>src_zone</field>
75+
<field>src_interface</field>
76+
<field>user</field>
77+
<field>protocol</field>
78+
<field>vendor_product</field>
79+
</missing_recommended_fields>
80+
<exceptions>
81+
<field name="vendor_product" value="Incorrect vendor product" reason="testing exceptions"/>
82+
</exceptions>
83+
</cim>
84+
<other_mappings>
85+
<field name="event_name" value="RT_FLOW_SESSION_CREATE">
86+
</other_mappings>
87+
</event>
88+
</device>
89+
```
90+
</details>
91+
892
## pytest-splunk-addon-data.conf.spec
993

1094
**Default Values**:
@@ -18,6 +102,7 @@ sourcetype = pytest-splunk-addon
18102
source = pytest-splunk-addon:{{input_type}}
19103
sourcetype_to_search = {{sourcetype}}
20104
sample_count = 1
105+
requirement_test_sample = 1
21106
timestamp_type = event
22107
count = 0
23108
earliest = now
@@ -78,6 +163,11 @@ host_prefix = {{host_prefix}}
78163
- This parameter will be used to calculate the total number of events which will be generated from the sample file.
79164
- If `input_type = modinput`, do not provide this parameter.
80165

166+
**requirement_test_sample = 1**
167+
168+
- This parameter is used to run requirement tests for the provided sample xml file
169+
- only supported with the xml sample file
170+
81171
**expected_event_count = <count\>**
82172

83173
- The no. of events this sample stanza should generate.

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ nav:
6363
- CIM Compliance Report: "cim_compliance.md"
6464
- Knowledge Object Tests: "field_tests.md"
6565
- Index Time Tests: "index_time_tests.md"
66+
- Requirement Tests: "requirement_tests.md"
6667
- Data Generator: "sample_generator.md"
6768
- API Documentation:
6869
- API Overview: "api_reference/api_reference.md"

0 commit comments

Comments
 (0)