Skip to content

Commit 73328d2

Browse files
StacieClark-ElasticStacieClark-Elastic
committed
fix-parsing-error-due-to-duplicate-fields
Fixes flattening error in Actions list when the list is encoded json string instead of json objects. Adds fields ActorInfoString, OperationCount, TokenObjectId, TokenTenantId. Added fields Messages and Folders as ExchangeMessages and ExchangeFolder for record type 50: ExchangeItemAggregated
1 parent 11d4062 commit 73328d2

File tree

7 files changed

+1143
-4
lines changed

7 files changed

+1143
-4
lines changed

‎packages/o365/changelog.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,13 @@
11
# newer versions go on top
2+
- version: "2.31.1"
3+
changes:
4+
- description: >-
5+
Fix flattening errors in Action List items due to duplicate QueryTime fields.
6+
Added fields ActorInfoString, OperationCount, TokenObjectId, TokenTenantId.
7+
Added fields Messages and Folders as ExchangeMessages and ExchangeFolders
8+
for record type 50: `ExchangeItemAggregated`.
9+
type: bugfix
10+
link: https://github.com/elastic/integrations/pull/99999
211
- version: "2.31.0"
312
changes:
413
- description: Improve documentation.

‎packages/o365/data_stream/audit/_dev/test/pipeline/test-exchange-access-event.json‎

Lines changed: 222 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/o365/data_stream/audit/_dev/test/pipeline/test-exchange-access-event.json-expected.json‎

Lines changed: 787 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 55 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -132,11 +132,12 @@ processors:
132132
if (!(ctx.o365audit.Actions instanceof List)) {
133133
ctx.o365audit.Actions = [ctx.o365audit.Actions];
134134
}
135+
def regex = /,\"QueryTime\":\"[0-9\/]+\s[0-9]+:[0-9]+:[0-9]+\s[AP]M\"|\"QueryTime\":\"[0-9\/]+\s[0-9]+:[0-9]+:[0-9]+\s[AP]M\",/;
135136
for (def e: ctx.o365audit.Actions) {
136137
if (e instanceof Map) {
137138
actions.add(e);
138139
} else if (e instanceof String) {
139-
ctx._tmp.action_strings.add(e);
140+
ctx._tmp.action_strings.add(regex.matcher(e).replaceAll(''));
140141
}
141142
}
142143
if (actions.length == ctx.o365audit.Actions.length) {
@@ -672,11 +673,11 @@ processors:
672673
target_field: file.extension
673674
ignore_missing: true
674675
if: ctx.event?.code != null && ["SharePointFileOperation", "SharePointSharingOperation"].contains(ctx.event.code)
675-
- append:
676+
- append:
676677
field: event.category
677678
value: file
678679
if: 'ctx.event?.action != null && ["FileAccessed", "FileDeleted", "FileDownloaded", "FileModified", "FileMoved", "FileRenamed", "FileRestored", "FileUploaded", "FolderCopied", "FolderCreated", "FolderDeleted", "FolderModified", "FolderMoved", "FolderRenamed", "FolderRestored"].contains(ctx.event?.action)'
679-
- append:
680+
- append:
680681
field: event.category
681682
value: configuration
682683
if: ctx.event?.action == "ComplianceSettingChanged"
@@ -1398,6 +1399,26 @@ processors:
13981399
} else {
13991400
ctx.o365audit.YammerNetworkId = ctx.o365audit.YammerNetworkId.toString();
14001401
}
1402+
- script:
1403+
tag: convert_runningtime
1404+
description: Ensure that RunningTime is not rendered with e-notation or other numeric
1405+
if: ctx.o365audit?.RunningTime != null
1406+
source: |-
1407+
if (ctx.o365audit.RunningTime instanceof double) {
1408+
ctx.o365audit.RunningTime = ((long)ctx.o365audit.RunningTime).toString();
1409+
} else {
1410+
ctx.o365audit.RunningTime = ctx.o365audit.RunningTime.toString();
1411+
}
1412+
- script:
1413+
tag: convert_operationcount
1414+
description: Ensure that OperationCount is not rendered with e-notation or other numeric
1415+
if: ctx.o365audit?.OperationCount != null
1416+
source: |-
1417+
if (ctx.o365audit.OperationCount instanceof double) {
1418+
ctx.o365audit.OperationCount = ((long)ctx.o365audit.OperationCount).toString();
1419+
} else {
1420+
ctx.o365audit.OperationCount = ctx.o365audit.OperationCount.toString();
1421+
}
14011422
- append:
14021423
field: email.message_id
14031424
value: "{{{o365audit.InternetMessageId}}}"
@@ -1446,6 +1467,7 @@ processors:
14461467
field: o365audit.EndTimeUtc
14471468
target_field: o365audit.EndTimeUtc
14481469
tag: date_EndTimeUtc
1470+
timezone: "UTC"
14491471
formats:
14501472
- ISO8601
14511473
if: ctx.o365audit?.EndTimeUtc != null
@@ -1770,6 +1792,36 @@ processors:
17701792
copy_from: o365audit.ApplicationDisplayName
17711793
tag: set_application_name
17721794
ignore_empty_value: true
1795+
1796+
# ExchangeItemAggregated Schema
1797+
- append:
1798+
field: event.type
1799+
value: access
1800+
if: 'ctx.o365audit?.RecordType != null && ctx.o365audit?.RecordType == "50"'
1801+
- append:
1802+
field: event.category
1803+
value: email
1804+
if: 'ctx.o365audit?.RecordType != null && ctx.o365audit?.RecordType == "50"'
1805+
- rename:
1806+
field: o365audit.Messages
1807+
target_field: o365audit.ExchangeMessages
1808+
tag: rename_messages_exchange
1809+
if: 'ctx.o365audit?.Messages != null && ctx.o365audit?.RecordType != null && ctx.o365audit?.RecordType == "50"'
1810+
- remove:
1811+
field: o365audit.Messages
1812+
tag: remove_messages_field
1813+
if: 'ctx.o365audit?.Messages != null'
1814+
description: 'remove o365audit.Messages if we have not explicitly renamed them based on record type'
1815+
- rename:
1816+
field: o365audit.Folders
1817+
target_field: o365audit.ExchangeFolders
1818+
tag: rename_folders_exchange
1819+
if: 'ctx.o365audit?.Folders != null && ctx.o365audit?.RecordType != null && ctx.o365audit?.RecordType == "50"'
1820+
- remove:
1821+
field: o365audit.Folders
1822+
tag: remove_folders_field
1823+
if: 'ctx.o365audit?.Folders != null'
1824+
description: 'remove o365audit.Folders if we have not explicitly renamed them based on record type'
17731825
- script:
17741826
description: Handle _tmp.entities.ThreatDetectionMethods containing list of lists.
17751827
lang: painless

‎packages/o365/data_stream/audit/fields/fields.yml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@
1616
type: keyword
1717
- name: ActorContextId
1818
type: keyword
19+
- name: ActorInfoString
20+
type: keyword
1921
- name: ActorIpAddress
2022
type: keyword
2123
- name: ActorUserId
@@ -275,6 +277,53 @@
275277
# not expressible here; object_type_mapping_type cannot be 'boolean'.
276278
object_type: keyword
277279
object_type_mapping_type: '*'
280+
- name: ExchangeFolders
281+
type: nested
282+
description: List of folders
283+
fields:
284+
- name: Path
285+
type: keyword
286+
description: Path of the folder
287+
- name: Id
288+
type: keyword
289+
description: Folder ID
290+
- name: FolderItems
291+
type: nested
292+
description: Items in the folder
293+
fields:
294+
- name: SizeInBytes
295+
type: long
296+
description: Size of the item in bytes
297+
- name: Id
298+
type: keyword
299+
description: Item ID
300+
- name: ImmutableId
301+
type: keyword
302+
description: Immutable ID of the item
303+
- name: InternetMessageId
304+
type: keyword
305+
description: Internet message ID
306+
- name: ExchangeMessages
307+
type: nested
308+
description: List of messages
309+
fields:
310+
- name: Path
311+
type: keyword
312+
description: Path of the message
313+
- name: Id
314+
type: keyword
315+
description: Message ID
316+
- name: MessageItems
317+
type: nested
318+
description: Items in the message
319+
fields:
320+
- name: SizeInBytes
321+
type: long
322+
description: Size of the message item in bytes
323+
- name: Id
324+
type: keyword
325+
description: Message item ID
326+
278327
- name: ExchangeMetaData
279328
type: group
280329
fields:
@@ -415,6 +464,8 @@
415464
type: keyword
416465
- name: Operation
417466
type: keyword
467+
- name: OperationCount
468+
type: keyword
418469
- name: OperationId
419470
type: keyword
420471
- name: OperationProperties
@@ -604,6 +655,10 @@
604655
type: keyword
605656
- name: ThreatDetectionMethods
606657
type: keyword
658+
- name: TokenObjectId
659+
type: keyword
660+
- name: TokenTenantId
661+
type: keyword
607662
- name: Timestamp
608663
type: keyword
609664
- name: UniqueSharingId

‎packages/o365/docs/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,7 @@ An example event for `audit` looks as following:
237237
| o365.audit.Actor.ID | | keyword |
238238
| o365.audit.Actor.Type | | keyword |
239239
| o365.audit.ActorContextId | | keyword |
240+
| o365.audit.ActorInfoString | | keyword |
240241
| o365.audit.ActorIpAddress | | keyword |
241242
| o365.audit.ActorUserId | | keyword |
242243
| o365.audit.ActorYammerUserId | | keyword |
@@ -356,6 +357,16 @@ An example event for `audit` looks as following:
356357
| o365.audit.EventDeepLink | | keyword |
357358
| o365.audit.EventSource | | keyword |
358359
| o365.audit.ExceptionInfo.\* | | object |
360+
| o365.audit.ExchangeFolders.FolderItems.Id | Item ID | keyword |
361+
| o365.audit.ExchangeFolders.FolderItems.ImmutableId | Immutable ID of the item | keyword |
362+
| o365.audit.ExchangeFolders.FolderItems.InternetMessageId | Internet message ID | keyword |
363+
| o365.audit.ExchangeFolders.FolderItems.SizeInBytes | Size of the item in bytes | long |
364+
| o365.audit.ExchangeFolders.Id | Folder ID | keyword |
365+
| o365.audit.ExchangeFolders.Path | Path of the folder | keyword |
366+
| o365.audit.ExchangeMessages.Id | Message ID | keyword |
367+
| o365.audit.ExchangeMessages.MessageItems.Id | Message item ID | keyword |
368+
| o365.audit.ExchangeMessages.MessageItems.SizeInBytes | Size of the message item in bytes | long |
369+
| o365.audit.ExchangeMessages.Path | Path of the message | keyword |
359370
| o365.audit.ExchangeMetaData.\* | | long |
360371
| o365.audit.ExchangeMetaData.CC | | keyword |
361372
| o365.audit.ExchangeMetaData.MessageID | | keyword |
@@ -417,6 +428,7 @@ An example event for `audit` looks as following:
417428
| o365.audit.ObjectId | | keyword |
418429
| o365.audit.ObjectType | | keyword |
419430
| o365.audit.Operation | | keyword |
431+
| o365.audit.OperationCount | | keyword |
420432
| o365.audit.OperationId | | keyword |
421433
| o365.audit.OperationProperties | | object |
422434
| o365.audit.OrganizationId | | keyword |
@@ -501,6 +513,8 @@ An example event for `audit` looks as following:
501513
| o365.audit.TeamName | | keyword |
502514
| o365.audit.ThreatDetectionMethods | | keyword |
503515
| o365.audit.Timestamp | | keyword |
516+
| o365.audit.TokenObjectId | | keyword |
517+
| o365.audit.TokenTenantId | | keyword |
504518
| o365.audit.UniqueSharingId | | keyword |
505519
| o365.audit.UserAgent | | keyword |
506520
| o365.audit.UserId | | keyword |

‎packages/o365/manifest.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
name: o365
22
title: Microsoft Office 365
3-
version: "2.31.0"
3+
version: "2.31.1"
44
description: Collect logs from Microsoft Office 365 with Elastic Agent.
55
type: integration
66
format_version: "3.2.3"

0 commit comments

Comments
 (0)