2674 Commits

Author SHA1 Message Date
04b7a5af68 Upgrade alpine to 3.20.x (#3710)
* upgrade alpine

Signed-off-by: Joe Elliott <number101010@gmail.com>

* changelog

Signed-off-by: Joe Elliott <number101010@gmail.com>

---------

Signed-off-by: Joe Elliott <number101010@gmail.com>
v2.5.0-rc.1
2024-05-23 11:49:50 -04:00
5a601425b5 2.5.0-rc1 changelog (#3709)
Signed-off-by: Joe Elliott <number101010@gmail.com>
2024-05-23 10:31:52 -04:00
614c65fb18 Keep replicationFactor after compaction (#3702) 2024-05-23 08:51:24 +02:00
2e57b3a17c [Traceql] Add support for searching by trace:id and span:id (#3670)
* add support for trace:id and spa:id

* fix tests and add bytes predicate to left trim 00s
2024-05-22 20:39:00 -05:00
502503ce7e Fix typos in CHANGELOG.md (#3706)
* Fix typos in CHANGELOG.md

Fix typos.

* Update CHANGELOG.md

* Update CHANGELOG.md
2024-05-22 18:59:39 -04:00
7f8daba596 Document how to enable vParquet4 in the config section (#3699)
* Document how to enable vParquet4 in the config

* Mention vParquet4 as experimental
2024-05-23 08:58:55 +10:00
cdfd28d383 Fix TraceQL failures on boolean operations with statics and attributes (#3698)
* don't pass non-boolean conditions to the fetch layer

Signed-off-by: Joe Elliott <number101010@gmail.com>

* changelog

Signed-off-by: Joe Elliott <number101010@gmail.com>

* add parens

Signed-off-by: Joe Elliott <number101010@gmail.com>

* changelog

Signed-off-by: Joe Elliott <number101010@gmail.com>

---------

Signed-off-by: Joe Elliott <number101010@gmail.com>
2024-05-22 17:15:35 -04:00
e617d60ec5 Update backend_search.md (#3688)
Corrected max_concuccrent_queries to max_concurrent_queries
2024-05-22 15:06:14 -04:00
67e28bbd79 [DOC] update links to tempo-operator API docs (#3658)
* [DOC] update links to tempo-operator API docs

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>

* update page title to Reference

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>

---------

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
2024-05-22 13:48:18 -04:00
41a11b43ac [DOC] Updates for caching doc and linting fixes (#3693)
* Updates for caching doc and linting fixes

* Apply suggestions from code review

Co-authored-by: Joe Elliott <joe.elliott@grafana.com>

---------

Co-authored-by: Joe Elliott <joe.elliott@grafana.com>
2024-05-22 13:47:47 -04:00
9c5fc70ffe changelog cleanup (#3701)
Signed-off-by: Joe Elliott <number101010@gmail.com>
2024-05-22 11:41:54 -04:00
c1a1a29b99 remove sleep (#3687)
Co-authored-by: Joe Elliott <number101010@gmail.com>
v2.5.0-rc.0
2024-05-21 12:48:10 -04:00
46f65200f7 Tempo CLI - add percentage support for query blocks (#3697)
* add percentage param to query blocks

* fix lint

* fix lint

* add change log
2024-05-21 10:41:48 -04:00
4149af8130 refactor (#3692)
Signed-off-by: Joe Elliott <number101010@gmail.com>
2024-05-21 08:08:55 -04:00
c5dbee1029 Update make docs procedure (#3600)
* Update `make docs` procedure

* Update `make docs` procedure

* Trigger CI

---------

Co-authored-by: grafanabot <bot@grafana.com>
Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
2024-05-20 18:00:21 -04:00
8cac0de04e Remove unnecessary vale exceptions (#3631)
No longer needed the vale exceptions.
2024-05-20 17:57:24 -04:00
89b0080a65 Add replication factor to block meta [1 of 3] (#3628)
* Add replication factor to block meta

* Refactor CreateWAL func

* chlog

* Fixes

* Add data encoding to WAL constructor

* Implement in vParquet4

* Different fixes

* Add test and fix cutting traces
2024-05-20 16:00:39 +02:00
Ali
f51485312c "ingestion_rate_limit_bytes" is set too short (#3685)
* Fixed multiple typos

* The limit is short and causes the 'test integration e2e suite' to fail

* The limit is short and causes the 'test integration e2e suite' to fail

* Update _index.md

Reverted the change 
My mistake

* Update linux.md

Reverted the other change
My mistake

* Update tempo-distributed-a.yaml

Reverted the other change
My mistake

* Update tempo-distributed-b.yaml

Reverted the other change
My mistake

* Update tempo-distributed.yaml

Reverted the other change
My mistake

* Update tempo-distributed.yaml

Reverted the other change
My mistake

* Update tempo-distributed.yaml

Reverted the other change
My mistake

* Update tempo-distributed.yaml

Reverted the other change
My mistake

* Update tempo-distributed.yaml

Reverted the other change
My mistake

* Update tempo.yaml

Reverted the other change
My mistake

* Update tempo-scalable-single-binary.yaml

Reverted the other change
My mistake

* Update backend.go

Reverted the other change
My mistake

* Update tempo.yaml

Reverted the other change
My mistake

* Update azure_test.go

Reverted the other change
My mistake

* Update search_sharder_test.go

Reverted the other change
My mistake

* Update tempo-distributed.yaml

Reverted the other change
My mistake

* Update tempo-distributed.yaml

Reverted the other change
My mistake

* Reverted the other change
2024-05-20 07:19:21 -04:00
Ali
fa25422593 Fixed multiple typos (#3684)
* Fixed multiple typos

* The limit is short and causes the 'test integration e2e suite' to fail

* Update config-limits-partial-success.yaml

Sent it in another PR
2024-05-16 14:06:18 -04:00
671145b148 [DOC] Add link to local-blocks in metrics-generator (#3682) 2024-05-16 10:32:34 -04:00
805b3f5d09 Fix panic in query-frontend when combining results (#3683)
* Fix panic in query-frontend when combining results

* chlog
2024-05-16 15:23:39 +02:00
c8840f814d [DOC] Add 2.4.2 release notes (#3678) 2024-05-15 14:59:35 -04:00
977579613e Iterator Perfomance Improvements (#3667)
* steal benches

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Gen floats, ints and bool predicates

Signed-off-by: Joe Elliott <number101010@gmail.com>

* remove string allocs

Signed-off-by: Joe Elliott <number101010@gmail.com>

* added gen to make

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Gen strings

Signed-off-by: Joe Elliott <number101010@gmail.com>

* patch up vp2 and vp4

Signed-off-by: Joe Elliott <number101010@gmail.com>

* remove instrumented predicate

Signed-off-by: Joe Elliott <number101010@gmail.com>

* int32 rns

Signed-off-by: Joe Elliott <number101010@gmail.com>

* skip nulls

Signed-off-by: Joe Elliott <number101010@gmail.com>

* vp4 unsafe string

Signed-off-by: Joe Elliott <number101010@gmail.com>

* revert isnull check due to impact on query range

Signed-off-by: Joe Elliott <number101010@gmail.com>

* remove string range cond

Signed-off-by: Joe Elliott <number101010@gmail.com>

* changelog

Signed-off-by: Joe Elliott <number101010@gmail.com>

* lint and tests

Signed-off-by: Joe Elliott <number101010@gmail.com>

* exclude

Signed-off-by: Joe Elliott <number101010@gmail.com>

* again

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix second line

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix bool pred

Signed-off-by: Joe Elliott <number101010@gmail.com>

* change generated file name and exclude

Signed-off-by: Joe Elliott <number101010@gmail.com>

* additional tests

Signed-off-by: Joe Elliott <number101010@gmail.com>

---------

Signed-off-by: Joe Elliott <number101010@gmail.com>
2024-05-15 14:58:45 -04:00
16c6694319 Update changelog for v2.4.2 release (#3676) 2024-05-15 17:01:10 +00:00
868d066806 Pin the goreleaser version for Go version support (#3674)
* Pin the goreleaser version for Go version support

Without this change, goreleaser@latest requires a newer version of Go
than is currently supported by the project.  Here we pin the version.

* Pin the version of the tools image
2024-05-14 18:37:11 +00:00
d8c69d487e Fix panic in SpanDeduper when traceID is empty slice (#3668)
* Fix panic in SpanDeduper when traceID is empty slice

* changelog
2024-05-13 17:32:12 -04:00
496b450e44 Use tools image for lint in CI and allow passing base ref (#3660)
* Use tools image for lint in CI and allow passing base ref

* Add fetch detpth for main diff
2024-05-10 13:16:08 +00:00
3f420b6a4a vParquet4: update opentelemetry-proto and add scope attrs (#3649)
* Update opentelemetry-proto submodule to v1.3.0

* Generate proto

* Populate scope attributes

* Mention breaking change in CHANGELOG.md

* Run make check-fmt
2024-05-10 09:34:13 +10:00
29f4167bba [TraceQL Metrics] histogram_over_time (#3644)
* Add lang/parser support for quantile_over_time, fix missing stringify of metrics and hints

* First working draft of quantile_over_time implementation

* Validate the query in the frontend

* Histogram accumulate jobs by bucket as they come in instead of at the end. Refactor label handling to be shared. Validate quantiles, support quantile_over_time by any integer attribute

* Fix language definition to allow both floats or ints for quantiles

* Remove layer of proto->seriesset conversion. Fix roundtrip of __bucket label

* Rename to SimpleAdditionCombiner, slight interval calc cleanup

* Fix p0 returning 1 instead of minimum value, comments cleanup

* Rename frontend param and fix handling of ints

* Fix pre-existing bug in metrics optimization when asserting multiple conditions on the same attribute

* Fix to support 3 flavors of the metrics pipeline: query-frontend, across blocks, and across generators.  Error handling, move code around

* Update query_range frontend test for new behavior

* Consolidate histogram code between traceql and traceqlmetrics. quantile_over_time test, code cleanup

* lint

* changelog

* histogram_over_time

* Redo histograms to set __bucket label to the actual value instead of log2, for several reasons:  support non-log2 or native histogram buckets in the future, and where queriers on different versions may have different buckets during rollout

* Revert all changes to traceqlmetrics package, was getting too noisy

* Fix after merge

* lint

* changelog

* Add test for histogram_over_time

* Change quantile_over_time test to use more exported methods
2024-05-09 12:08:09 -04:00
e06c33905e 👋 So long vParquet1, and thanks for all the fish! (#3663)
* good bye vParquet!

* changelog
2024-05-09 11:04:36 -04:00
b5b1355bc5 Exit after running the tool's image (#3664)
* Exit after running the tool's image

* Run 'make fmt'
2024-05-09 13:51:44 +00:00
a8538521ba Split metrics slo out from search slo (#3659)
In #3584 we decided to count metrics queries as part of search slo. 

split metrics slo out of search because metrics are still experimental and should not share the SLO with search.
2024-05-09 10:56:27 +05:30
465dc89d0e Add tempo-cli sub command to convert from vParquet3 to vParquet4 (#3653)
* Add tempo-cli sub command to convert from vParquet3 to vParquet4

* Make 2to3 and 3to4 converter more consistent
2024-05-09 09:19:04 +10:00
b76ab98594 Golangci deprecations and update using tools pattern (#3610)
* Breadcrumbs for golangci

* Include golangci in tools

* Use tools image for lint make target

* Update changelog
2024-05-08 20:56:43 +00:00
9032132d0c Bump x/net package to 0.24/latest (#3613)
* Bump x/net package to 0.24/latest

Signed-off-by: Pavol Loffay <p.loffay@gmail.com>

* Fix

Signed-off-by: Pavol Loffay <p.loffay@gmail.com>

* Fix

Signed-off-by: Pavol Loffay <p.loffay@gmail.com>

---------

Signed-off-by: Pavol Loffay <p.loffay@gmail.com>
2024-05-08 17:21:01 +00:00
5b250a7e85 fix(traceql): fix extract matcher regexes to work with regexp-type matchers (#3641)
* fix(traceql): fix extract matcher regexes to work with regexp-type matchers

Partially fixes #3635.

I'm going to add a further test case which this doesn't fix.

* Add more test cases for extract matcher

* Add CHANGELOG entry
2024-05-08 12:38:22 -04:00
947de3531d Update deb and rpm packages to create and assign permissions for /var/tempo (#3657)
* test deb stuff

* more stuff

* create /var/tempo and assign perms

* Create and assign permissions of /var/tempo after creating user

* More precise dir perms

* whitespace

* changelog
2024-05-08 12:28:58 -04:00
80ed362352 Align OTEL semconv package to v1.25.0 (#3609)
* Update vendor for semconv updates

* Align OTEL semconv package to v1.25.0

* Update serverless go.mod

* Add a quick explicit test for semconv keys servicegraphs rely on
2024-05-08 16:12:21 +00:00
8df96701a3 [TraceQL] quantile_over_time 2 of 2 - engine (#3633)
* Add lang/parser support for quantile_over_time, fix missing stringify of metrics and hints

* First working draft of quantile_over_time implementation

* Validate the query in the frontend

* Histogram accumulate jobs by bucket as they come in instead of at the end. Refactor label handling to be shared. Validate quantiles, support quantile_over_time by any integer attribute

* Fix language definition to allow both floats or ints for quantiles

* Remove layer of proto->seriesset conversion. Fix roundtrip of __bucket label

* Rename to SimpleAdditionCombiner, slight interval calc cleanup

* Fix p0 returning 1 instead of minimum value, comments cleanup

* Rename frontend param and fix handling of ints

* Fix pre-existing bug in metrics optimization when asserting multiple conditions on the same attribute

* Fix to support 3 flavors of the metrics pipeline: query-frontend, across blocks, and across generators.  Error handling, move code around

* Update query_range frontend test for new behavior

* Consolidate histogram code between traceql and traceqlmetrics. quantile_over_time test, code cleanup

* lint

* changelog

* Redo histograms to set __bucket label to the actual value instead of log2, for several reasons:  support non-log2 or native histogram buckets in the future, and where queriers on different versions may have different buckets during rollout

* Revert all changes to traceqlmetrics package, was getting too noisy
2024-05-07 14:23:16 -04:00
884a344b60 SearchTagValuesV2 perf improvements (#3650)
* remove keep

Signed-off-by: Joe Elliott <number101010@gmail.com>

* null test

Signed-off-by: Joe Elliott <number101010@gmail.com>

* remove attr

Signed-off-by: Joe Elliott <number101010@gmail.com>

* consolidate

Signed-off-by: Joe Elliott <number101010@gmail.com>

* dont sent 2x

Signed-off-by: Joe Elliott <number101010@gmail.com>

* port to vp4

Signed-off-by: Joe Elliott <number101010@gmail.com>

* changelog

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix wal

Signed-off-by: Joe Elliott <number101010@gmail.com>

---------

Signed-off-by: Joe Elliott <number101010@gmail.com>
2024-05-07 13:15:53 -04:00
187e0c2b9c Union structural operators (#3637)
* Add DescendantOf Tests

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Add tests for basic functionality

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Add to language

Signed-off-by: Joe Elliott <number101010@gmail.com>

* add structural support for union

Signed-off-by: Joe Elliott <number101010@gmail.com>

* DescendantOf

Signed-off-by: Joe Elliott <number101010@gmail.com>

* SiblingOf

Signed-off-by: Joe Elliott <number101010@gmail.com>

* improved tests. built common structures

Signed-off-by: Joe Elliott <number101010@gmail.com>

* fix siblingOf many to many and own sibling

Signed-off-by: Joe Elliott <number101010@gmail.com>

* integration tests + fixes

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Moved descendantOf union and siblingOf to the same loop

Signed-off-by: Joe Elliott <number101010@gmail.com>

* rewrite union descendantOf to avoid pathological cases

Signed-off-by: Joe Elliott <number101010@gmail.com>

* tests, benches, and fixes

Signed-off-by: Joe Elliott <number101010@gmail.com>

* handle dupe slices

Signed-off-by: Joe Elliott <number101010@gmail.com>

* cleanup and tests

Signed-off-by: Joe Elliott <number101010@gmail.com>

* vp3 -> vp2

Signed-off-by: Joe Elliott <number101010@gmail.com>

* docs

Signed-off-by: Joe Elliott <number101010@gmail.com>

* benchmarks

Signed-off-by: Joe Elliott <number101010@gmail.com>

* revert non-union implementations due to inconsistent benches

Signed-off-by: Joe Elliott <number101010@gmail.com>

* lint

Signed-off-by: Joe Elliott <number101010@gmail.com>

* lint

Signed-off-by: Joe Elliott <number101010@gmail.com>

* wip: refactor

Signed-off-by: Joe Elliott <number101010@gmail.com>

* refactor to put union first

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Update docs/sources/tempo/traceql/_index.md

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>

* Update docs/sources/tempo/traceql/_index.md

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>

* port to vp4

Signed-off-by: Joe Elliott <number101010@gmail.com>

* param cleanup

Signed-off-by: Joe Elliott <number101010@gmail.com>

---------

Signed-off-by: Joe Elliott <number101010@gmail.com>
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
2024-05-07 09:35:52 -04:00
fc33ae5fba Add vparquet4 encoding (#3368)
* Copy vparquet3 to vparquet4 folder

* Create initial vParquet4 encoding

* vParquet4: handle unsupported / dropped attributes (#3123)

* Count dropped attributes and store them in separate column

* Remove current support for arrays and kv-lists

* Store per-trace service statistics in vparquet4 blocks (#2941)

* Add support for attributes with array values (#3221)

* Implement single values as arrays and introduce type column

* Add support for array attributes

* Tests more array values and fix attribute conversion

* Use a combination of snappy and delta for ValueType

* Vendor module google/go-cmp/cmp/cmpopts

* Use all types of array attributes in TestTraceToParquet

* Improve TestFieldsAreCleared

* links and events schema changes (#3163)

* Add attributes to instrumentation scope (#3322)

* Fix typo in vParquet4 event name enconding (#3336)

* Precalculate and reuse the vParquet4 schema before opening blocks

* Convert block in vparquet4/test-data directory

* CHANGELOG.md

* Update test cases in BenchmarkBackendBlockTraceQL

* Unsupported attribute values are no longer dropped

* Replace ValueType column with IsArray column

* Skip service stats map allocation when none are present

---------

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
Co-authored-by: Andreas Gerstmayr <agerstmayr@redhat.com>
Co-authored-by: Jennie Pham <94262131+ie-pham@users.noreply.github.com>
Co-authored-by: Andreas Gerstmayr <andreas@gerstmayr.me>
2024-05-07 10:01:03 +10:00
174b42841f Parquetquery join performance improvement (#3604)
* change for loop

* Dynamic reordering of iterators

* fix regression on complex query by peeking all iters at start and exiting early

* Add new complex query test

* move test
2024-05-06 15:21:19 -04:00
37f3308aeb Fix span-metrics' subprocessors bug (#3612)
* Fix span-metrics' subprocessors bug that applied wrong configs when running multiple tenants

* Add test

* Fix race condition

* Update CHANGELOG.md

Co-authored-by: Joe Elliott <joe.elliott@grafana.com>

* Copy when copying the config

* Fix bug

---------

Co-authored-by: Joe Elliott <joe.elliott@grafana.com>
2024-05-06 10:39:43 +02:00
3f831f9e31 Implement tenant deletion for empty tenants (#3611)
* Extend backend.Reader with Find() function

* Extend backend.Writer with Delete() function

* Add EmptyTenantDeletion age config to tempodb

* Add delete handling to poller

* Add integration tests for poller change

* Update backend implementations

* Fix lint

* Setup test defaults

* Integrate prefix test permutations

* Add doc for the new configuration option

* Update changelog

* Fix local implementation to ignore directories

* Tidy up todo after test coverage

* Update docs/sources/tempo/configuration/_index.md

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>

* Address some PR feedback

* Rename backend.FindOpts -> backend.FindMatch
* Adjust default delete time from 20m -> 12h

* Add log message for object deletion

* Add additional safety check to avoid deletion

* Add test default value

* Godoc update after rename

* Drop logging from the backend used for testing

* Require empty tenant deletion to be enabled in the config

* Add docs for _enabled config

---------

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
2024-05-03 20:48:42 +00:00
35aa72e692 Add new histogram to generator - messaging system latency (#3453)
* introduce new service-graph metric for messaging-system latency

* added tests for new histogram values

* fix linting

* make new metric optional via config

* fix typo

* fix failing tests

* add feature to changelog

* negative times diff consistency - return 0 instead of negative

* update docs

* Update docs/sources/tempo/metrics-generator/service_graphs/estimate-cardinality.md

use present when possible

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>

* change 1e9 to time const

* added a reference to the "wait" config of the processor

* fixed indentations and formatting stuff from rebasing

* removed mistaken println found by linter

---------

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
2024-05-03 12:49:23 -04:00
854cacaa42 TraceQL - Add support for scoped intrinsics using : (#3629)
* add scoped intrinsics support :

* lint

* test & changelog

* more test & changelog & doc

* rename

* add comment
2024-05-02 19:25:22 -05:00
93766e58a6 metrics-generator counter: reset length of pre-allocated slice (#3640) 2024-05-02 17:24:17 +02:00
251bf5a9a7 Surface new labels for uninstrumented services and systems (#3543)
* Surface new labels for uninstrumented services and systems

* Update CHANGELOG.md

* remove unnecessary Println in test

* Reuse dimensions and prefixes for this use-case

* Add docs

* keep only virtual_node behind the new feature

* add overrides

* Update docs/sources/tempo/metrics-generator/service_graphs/_index.md

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>

* forgot this test (fixed)

* add benchmarks

* add edge pooling/reuse

* update flag

* move label out of dimensions to avoid prefix logic

* lint

* Update modules/generator/processor/servicegraphs/config.go

Co-authored-by: Mario <mariorvinas@gmail.com>

* minor amends to names and docs

* leave the new virtual_node label as an extra dimension

* keep edge sync.Pool ops inside store

* Update modules/generator/processor/servicegraphs/store/store.go

The edge is not expired here, so it shouldn't be returned to the pool.

Co-authored-by: Mario <mariorvinas@gmail.com>

* leave the new label un-prefixed

---------

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
Co-authored-by: Mario <mariorvinas@gmail.com>
2024-04-30 16:53:05 -04:00
11e315c6ed Update set-up-monitoring.md (#3623)
* Update set-up-monitoring.md

Fix typo in admonition

* Update docs/sources/tempo/operations/monitor/set-up-monitoring.md

---------

Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
2024-04-30 13:28:33 -04:00