Added logic to main.py to use the created_at and updated_at values if they exist

[aarju]

Summary - What I changed

During testing I found that for my custom rules when exported from a cluster for the first time the creation_date and updated_date values were being set to todays date instead of using the created_at and updated_at values from the detection rules.

This update adds logic to main.py (line 250) within the import_rules_into_repo function to use the values from the contents of the rule if those fields exist. This fixes the issue and now when rules are imported for the first time they retain the original values from the cluster.

This logic will only be called when the --dates-import | -di commandline option is called with the import_rules_into_repo function.

How To Test

import existing custom detection rules from a cluster to a repo where the created_at and updated_at dates are prior to today. Verify that the creation_date and updated_date use the correct date from the rule file and are not set to today.

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[eric-forte-elastic]

👋 Thanks for the PR, unless I am mistaken, generally speaking when this code was implemented the decision was made that created and updated dates are specific to the toml file rather than the rule file from Kibana. I am certainly open to this change, but as the code sits currently, this is intended behavior.

Given that I expect the outcome will be that we will want to include the functionality from the PR, we can continue to review and merge when we are satisfied, just wanting to note that this will be a change (or alternative) to the original import pipeline 👍

[eric-forte-elastic]

Instead of using these try, except blocks, it would be better to match the style/formatting we use in this function by using .get(). The addition is smaller and does not need to add additional exception handling.

It could look something like this instead for this PR:

        # Parse created_at and updated_at to creation_date and updated_date if they exist in contents
        now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S.%fZ")
        contents["creation_date"] = datetime.strptime(
            contents.get("created_at", now), "%Y-%m-%dT%H:%M:%S.%fZ"
        ).strftime("%Y/%m/%d")
        contents["updated_date"] = datetime.strptime(contents.get("updated_at", now), "%Y-%m-%dT%H:%M:%S.%fZ").strftime(
            "%Y/%m/%d"
        )

[aarju]

Thanks for the help, I just updated the functions to use the .get() function using your code.

[aarju]

just wanting to note that this will be a change (or alternative) to the original import pipeline

@eric-forte-elastic do you think this is a big enough change to justify wrapping this code in a new commandline option to allow users to enable it if they want it?

[aarju]

@eric-forte-elastic I updated the function to wrap it inside of a new --dates-import commandline option. This will provide this functionality without breaking existing use cases.

[eric-forte-elastic]

@eric-forte-elastic I updated the function to wrap it inside of a new --dates-import commandline option. This will provide this functionality without breaking existing use cases.

This looks great! Now you would just have to bump the pyptoject toml version from version = "1.5.22" to version = "1.5.23" and add a patch label for this issue.

[aarju]

@eric-forte-elastic I updated the pyproject.toml version number, but It looks like I don't have permissions to update the issue labels in this repo.

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[eric-forte-elastic]

@eric-forte-elastic I updated the pyproject.toml version number, but It looks like I don't have permissions to update the issue labels in this repo.

Roger, updated the issue with the labels.

Looks like just linting now.

[aarju]

I updated the code to fix the linting errors about timezone support. I tested the code again and it still works as intended on my system.