Add Support for Multiple Kibana Security Detection Rule Types

[Copilot]

Implements comprehensive support for Kibana Security Detection Rules with multiple rule types beyond the initial query rules.

Fixes #523

Changes Made

Core Infrastructure

• Analyze current implementation and comments
• Replace !IsNull() && !IsUnknown() pattern with utils.IsKnown()
• Use utils.ListTypeAs and utils.ListValueFrom for list operations
• Move functions to be receivers on SecurityDetectionRuleData struct
• Use clients.CompositeIdFromStrFw for ID parsing
• Add proper UUID validation to prevent panics
• Re-use Read logic in Create and Update instead of parsing response directly
• Address all specific comment suggestions
• Test and validate changes
• All tests pass and linting is clean
• Documentation generated successfully

Multi-Rule Type Support

• Schema Enhancement: Updated schema to support all 8 detection rule types (query, eql, esql, machine_learning, new_terms, saved_query, threat_match, threshold) with their specific fields
• Rule Creation: Implemented create logic for all rule types with type-specific field handling
• Rule Updates: Implemented update logic for query and EQL rules (others show helpful error messages)
• Rule Reading: Enhanced read logic to handle query and EQL rules with improved error messages for unsupported types
• Helper Functions: Added utility functions for ID extraction and rule type handling across different rule structures

Rule Type Specific Fields Added

• EQL Rules: tiebreaker_field support
• Machine Learning Rules: anomaly_threshold, machine_learning_job_id
• New Terms Rules: new_terms_fields, history_window_start
• Saved Query Rules: saved_id support
• Threat Match Rules: threat_index, threat_query, threat_mapping, threat_filters, threat_indicator_path, concurrent_searches, items_per_search
• Threshold Rules: threshold configuration with field, value, and cardinality settings
• Common Fields: timeline_id, timeline_title, threat (MITRE ATT&CK framework support)

Current Status

• Fully Supported: Query and EQL rules (create, update, read)
• Create Only: ESQL, ML, New Terms, Saved Query, Threat Match, and Threshold rules
• Remaining Work: Update and read operations for the 6 remaining rule types

This implementation provides a solid foundation for all Kibana Security Detection rule types while maintaining backward compatibility and following established patterns in the codebase.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[cla-checker-service]

💚 CLA has been signed

[nick-benoit]

The trickiest part of this PR is sorting out rule_id vs id in the update case: https://www.elastic.co/docs/api/doc/kibana/operation/operation-updaterule

Update a detection rule using the rule_id or id field. The original rule is replaced, and all unspecified fields are deleted.

The api gives back this error if both are included:

│ API returned status 400: {"message":["both \"id\" and \"rule_id\" cannot exist, choose one or the other"],"status_code":400}```

If provided we will use the `rule_id` and unset the `id`. Otherwise we use the `id`.

[tobio]

Pretty excellent first pass here, added a bunch of comments around using some shared utils and how we've generally tried to structure things but this is looking good.

IMO we should cover all the rule types in this PR, but I could be convinced they can be added in a follow up. We do need to cover them to consider the issue 'done' though.

[tobio]

Same as above, we should move this to the model.

[Copilot]

Moved function to be a receiver method toUpdateProps on the SecurityDetectionRuleData struct in 828ef7a.

[tobio]

Looking at the acceptance test failures:

The 7.17 test is failing because an expected index doesn't exist yet. I guess it wouldn't be hard to create another 7.x specific test which also first created the index, but I'm not sure it's worthwhile given it's 7.x.

The 8.10 test looks like we're getting a different response from Kibana in that version. Definitely worth investigating, but given 8.10 is 2 years old I don't think it's unreasonable to skip the test in 8.10 and mention in the docs that the resource isn't compatible with 8.10.

[nick-benoit]

@copilot Please support all detection rule types defined in https://www.elastic.co/docs/api/doc/kibana/group/endpoint-security-detections-api

This will require adding updating the schema in internal/kibana/security_detection_rule/schema.go and corresponding methods in internal/kibana/security_detection_rule/models.go (eg toCreateProps, etc). Additionally instead of always using SecurityDetectionsAPIQueryRuleCreateProps to create and SecurityDetectionsAPIQueryRuleUpdateProps to update we will need choose the appropriate query rule type based on the type of rules we are using.

[Copilot]

@copilot Please support all detection rule types defined in https://www.elastic.co/docs/api/doc/kibana/group/endpoint-security-detections-api

This will require adding updating the schema in `intern...

I'm implementing support for all detection rule types. I've updated the schema to support all 8 rule types (query, eql, esql, machine_learning, new_terms, saved_query, threat_match, threshold) with their specific fields, and I'm working on updating the models to handle creation and updates for each type. This is a significant change that requires updating the create, update, and read logic for all rule types. I'll commit the progress once the implementation is complete and tested.

[tobio]

I need to do some testing here as well, adding some comments in the meantime.

[tobio]

There's a lot of duplication in the create/update flows. I wonder if it would be simpler if the update code re-used the create model as an interim step. So tfModel -> createModel -> updateModel.

I don't feel strongly here, and it might end up as more of a mess.

[nick-benoit]

Yeah I agree. I was trying to find a clever way to reuse that logic. In general I think I prefer some duplication over assuming a false equivalence between the types. Especially when zooming out and thinking about applying this pattern with the LLM to other resources. I've been thinking through that lens as I work on this.

I'm not in love with this pattern, but we could also add something like setCommonCreateProps, but instead of sharing common properties among all createProps it would be for the common props between the update / create for each rule type?

I'll probably think of some other ideas after stewing on it a bit as well.

[tobio]

Should this be RequiresReplace?

[nick-benoit]

I think so 👍

[nick-benoit]

da655cb

[nick-benoit]

Actually this causes some strange behavior. The API gives us back a value when it is not provided. I made it a computed value to support this. This causes some strange behavior because when it goes from unknown -> the computed value it forces replacement.

I suppose we want something that only replaces for a non-computed value? I'm not quite sure the best way to achieve that.

[nick-benoit]

A careful look at the docs makes me realize there are some missing fields 😞

At least investigation_fields, exceptions_list (actually a bunch of nested types). I suppose maybe copilot just threw its hands in the air with the nested types?

docs

[tobio]

Looking good, it looks like we need to improve the case where read gets a 404 response.

[tobio]

This isn't really returning empty data since we initialise the defaults early on. Should the return type be a pointer so we can explicitly return nil here?

[tobio]

Hmmm, new GH review UX is working well it seems.

[nick-benoit]

Looking good, it looks like we need to improve the case where read gets a 404 response.

Just to make sure i'm understanding here. Are you doing something like:

• Creating a rule with TF
• Deleting the rule in Kibana UI
• TF apply with no changes

Currently it looks like it then tries to create the resource as if you were creating it from scratch.

What do you think the ideal handling is in a case like that? Or also it could be you tested this differently and found something more obviously broken.

[nick-benoit]

Looking good, it looks like we need to improve the case where read gets a 404 response.

@tobio

It looks like when I do the following:

• Create a rule in TF
• Delete rule in Kibana
• Run no-op TF apply

It recognizes the resource is gone, and tries to create a new resource as if it never existed in the first place. What do you think the ideal handling is there?

Also it could be that you tested it differently and found something more obviously broken. In either case i'm ++ on returning a pointer as you suggest below.