Skip to content

Conversation

JordanRushing
Copy link
Contributor

What this PR does / why we need it:

Fixes the issue where "negative structured metadata bytes received" errors were being logged in Loki when processing OTLP logs.

Problem

The OTLP processing code was calculating entry-specific metadata size by subtracting resource and scope attribute sizes from the total metadata size after they were already combined. This calculation could result in negative values, which were caught by guard clauses and logged as errors.

Solution

This PR changes the approach to calculate the entry's metadata size before adding resource and scope attributes. This preserves the original intent (tracking entry-specific metadata separately) while ensuring the values are always positive.

The change maintains the same accounting logic but eliminates the possibility of negative values being passed to Prometheus counters.

Testing

  • Updated tests to reflect the new calculation method
  • Added a specific test case TestOTLPStructuredMetadataCalculation to verify structured metadata bytes are always positive

Which issue(s) this PR fixes:

N/A

Special notes for your reviewer:

N/A

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@JordanRushing JordanRushing self-assigned this May 9, 2025
@JordanRushing JordanRushing added opentelemetry Everything related to OpenTelemetry OTLP OTel area/structured metadata labels May 9, 2025
@pull-request-size pull-request-size bot added size/L and removed size/M labels May 9, 2025
… attributes

This change prevents negative structured metadata byte sizes by calculating
the entry's own metadata size before appending resource and scope attributes,
rather than attempting to subtract afterwards. This preserves the intent of
tracking entry-specific metadata separately while ensuring values are always
positive.
@JordanRushing JordanRushing force-pushed the negative-structured-metadata-bytes branch from e744ee2 to 0f0f70b Compare June 30, 2025 17:32
@JordanRushing JordanRushing marked this pull request as ready for review June 30, 2025 17:32
@JordanRushing JordanRushing requested a review from a team as a code owner June 30, 2025 17:32
…etadata byte tests in otlp_test.go

Signed-off-by: Jordan Rushing <[email protected]>
Copy link
Collaborator

@trevorwhitney trevorwhitney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, lgtm

@JordanRushing JordanRushing merged commit 2161c8c into grafana:main Jul 1, 2025
65 checks passed
@ashwanthgoli ashwanthgoli added type/bug Somehing is not working as expected backport release-3.5.x labels Sep 11, 2025
ashwanthgoli pushed a commit that referenced this pull request Sep 11, 2025
ashwanthgoli added a commit that referenced this pull request Sep 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/structured metadata backport release-3.5.x opentelemetry Everything related to OpenTelemetry OTLP OTel size/L type/bug Somehing is not working as expected
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants