[Rule-based Autotagging] add security attributes to rules

[ruai0511]

Description

This PR enhances rule-based auto-tagging functionality by introducing security-related attributes. Specifically, we aim to extract user information (username and role) from the security context of incoming requests. In many real-world environments, security context is essential for customers with multiple teams, departments, or security boundaries. By enabling the use of security attributes in auto-tagging rules, this project will provide more practical and convenient ways to define the tenants.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 07a5cd4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for eb67fce: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[peternied]

@ruai0511 Thanks for this change this would be really useful for tracing requests to have these attributes. How will this change work with the subject from IdentityPlugin?

@cwperks love to get your thoughts on this change.

[cwperks]

Can we make this more generic so that there's no reference to username or role?

I'm not sure how WLM works, but I know that a request will match to the best workload group. How does the scoring work for matching? Can we assign weights to different attributes to help with the matching problem to determine the best group?

[ruai0511]

I believed we need to reference to username or role because they are auto tagging's attributes. I just made them Attribute instead of a String to make it clearer

[cwperks]

@cwperks love to get your thoughts on this change.

@peternied we've been discussion an extensible <-> extending plugin relationship here, especially since you can now define optional relationships with extending -> extensible plugin.

In this case, WLM needs attributes from security, but it differs from other plugins because its in the core. There's certainly prevalent usage of common-utils today but we should not introduce a dependency on common-utils in the core repo. (In general, we need to work towards removal of common-utils).

IdentityPlugin is one avenue, but that extension point is quite simple and only used for username and has no notion of roles or other attributes. We could certainly entertain expanding on the notion of subject to add roles, but I know there were differing opinions on that and as a result the IdentityPlugin can only be used to get username.

In this case, by making WLM extensible it can ask other plugins to define an attribute extractor to feed it back attributes (and weights to break ties) given a REST Request. For security that could be username + roles and it would give higher precedence to direct username matching to workload groups.

[kaushalmahi12]

I have the following initial set of comments which might change rest of the PR changes hence not reviewing it further.

[github-actions]

❌ Gradle check result for 756409d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 209f55d: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8b097bb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8f1a17c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[kaushalmahi12]

I had two minor comments, just address them and send it to approval with the maintainers

[kaushalmahi12]

I think rather than iterating over all the keys since we are simply interested in finding the valid prefixes of the key then iterating over the prefixes would be much faster. Do look into get related methods in patricia trie to make it more efficient

[ruai0511]

If we iterate thru prefix then it's be back to the previous implementation

List<String> results = new ArrayList<>();
for (int i = key.length(); i >= 0; i--) {
    String prefix = key.substring(0, i);
    String value = trie.get(prefix);
    if (value != null && !value.isEmpty()) {
        results.add(value);
    }
}

I looked into the patricia trie implementation, we can't use prefixMap method (it finds all keys in the trie that has the prefix, not the other way around) and don't have the access to the trie root (so we can't just walk the trie from top to bottom)

[github-actions]

❌ Gradle check result for 5762c3e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 9ef3651: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 4c9ccd7: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 83.19672% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.89%. Comparing base (e0aebd9) to head (422aeab).
⚠️ Report is 41 commits behind head on main.

Files with missing lines	Patch %	Lines
...in/java/org/opensearch/rule/SecurityAttribute.java	75.00%	7 Missing and 3 partials ⚠️
...ava/org/opensearch/rule/autotagging/Attribute.java	50.00%	4 Missing and 2 partials ⚠️
...pensearch/plugin/wlm/WorkloadManagementPlugin.java	50.00%	4 Missing ⚠️
...ensearch/rule/autotagging/AutoTaggingRegistry.java	0.00%	2 Missing and 1 partial ⚠️
...ule/service/IndexStoredRulePersistenceService.java	62.50%	2 Missing and 1 partial ⚠️
...earch/rule/storage/DefaultAttributeValueStore.java	82.35%	0 Missing and 3 partials ⚠️
...opensearch/plugin/wlm/AutoTaggingActionFilter.java	70.00%	1 Missing and 2 partials ⚠️
...opensearch/rule/InMemoryRuleProcessingService.java	83.33%	1 Missing and 1 partial ⚠️
...feature_value_resolver/CandidateFeatureValues.java	94.44%	0 Missing and 2 partials ⚠️
...e/feature_value_resolver/FeatureValueResolver.java	96.55%	0 Missing and 2 partials ⚠️
... and 3 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19232      +/-   ##
============================================
+ Coverage     72.81%   72.89%   +0.08%     
- Complexity    69631    69702      +71     
============================================
  Files          5658     5663       +5     
  Lines        320087   320259     +172     
  Branches      46345    46373      +28     
============================================
+ Hits         233057   233461     +404     
+ Misses        68118    67876     -242     
- Partials      18912    18922      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Gradle check result for de0e826: SUCCESS

[github-actions]

❌ Gradle check result for 3d5e169: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 0d53b23: SUCCESS

[github-actions]

✅ Gradle check result for 422aeab: SUCCESS

[jainankitk]

This is a large PR. Will be great if we can break this down into smaller more reviewable PRs?

[github-actions]

❌ Gradle check result for aad51ba: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?