Query prometheus metrics from benchmark #3473
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Mar 4, 2025
This stack of pull requests is managed by Graphite. Learn more about stacking. |
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
5a6b566 to
2fc2b51
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
3 times, most recently
from
2666a06 to
25bb2d7
Compare
afck
reviewed
Mar 5, 2025
ndr-ds
changed the base branch from
03-04-pretty_print_chain_info_response_on_query_validators
to
graphite-base/3473
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
25bb2d7 to
e14cf17
Compare
ndr-ds
force-pushed
the
graphite-base/3473
branch
from
57ce110 to
99e35bc
Compare
ndr-ds
changed the base branch from
graphite-base/3473
to
03-04-end_value_fix_for_linear_bucket_interval
ndr-ds
force-pushed
the
03-04-end_value_fix_for_linear_bucket_interval
branch
from
99e35bc to
1b69a51
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
e14cf17 to
8f5e1c6
Compare
ndr-ds
changed the base branch from
03-04-end_value_fix_for_linear_bucket_interval
to
graphite-base/3473
ndr-ds
force-pushed
the
graphite-base/3473
branch
from
1b69a51 to
ba839b5
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
8f5e1c6 to
32b2bce
Compare
ndr-ds
changed the base branch from
graphite-base/3473
to
03-04-end_value_fix_for_linear_bucket_interval
ndr-ds
force-pushed
the
03-04-end_value_fix_for_linear_bucket_interval
branch
from
ba839b5 to
d400116
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
32b2bce to
26e8c6a
Compare
ndr-ds
force-pushed
the
03-04-end_value_fix_for_linear_bucket_interval
branch
from
d400116 to
34f9c49
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
2 times, most recently
from
5824eec to
73e32e8
Compare
ndr-ds
force-pushed
the
03-04-end_value_fix_for_linear_bucket_interval
branch
from
34f9c49 to
456101c
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
73e32e8 to
bb69877
Compare
ndr-ds
force-pushed
the
03-04-end_value_fix_for_linear_bucket_interval
branch
from
456101c to
4082ac3
Compare
ndr-ds
changed the base branch from
03-04-end_value_fix_for_linear_bucket_interval
to
graphite-base/3473
ndr-ds
force-pushed
the
graphite-base/3473
branch
from
4082ac3 to
b7e963a
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
bb69877 to
6caf011
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
6caf011 to
76c77e7
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
76c77e7 to
24ab520
Compare
ndr-ds
changed the base branch from
graphite-base/3473
to
03-05-adding_scylladb_cpu_usage_graph_some_other_changes
ndr-ds
changed the base branch from
03-05-adding_scylladb_cpu_usage_graph_some_other_changes
to
graphite-base/3473
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
24ab520 to
37b5af1
Compare
ndr-ds
force-pushed
the
graphite-base/3473
branch
from
87cb69a to
fcbe2c4
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
37b5af1 to
6508b5c
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
6508b5c to
6651527
Compare
afck
approved these changes
Mar 6, 2025
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
3 times, most recently
from
e016dce to
65d1362
Compare
ndr-ds
force-pushed
the
02-24-query_prometheus_metrics_from_benchmark
branch
from
65d1362 to
f3fd01e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Right now, the only way of knowing if the benchmark is completely destroying validators is by checking Grafana for the metrics (which have a 2m delay), and stopping the benchmark manually. This is not ideal, as if latency becomes too high, the validators can take a very long time to recover after it.
Proposal
Introduce a
--health_check_endpointstolinera benchmark, that if provided, will query the specified metrics host:port pairs for metrics, to determine the health of the validators. For now this health is determined by the p99 proxy latency being below 400 ms.We're querying the metrics endpoint from the proxy directly with this, so it works on all different kinds of deployments (docker compose, kubernetes, just binaries running locally, etc). The user is responsible for making sure the metrics endpoint is available at the provided endpoints (might need to port forward the metrics port, etc).
We'll query the endpoints once every 5 seconds, and calculate the p99 of the different validators in those last 5s. If we see one validator being above the 400 ms threshold, automatically stop the benchmark. We calculate the p99 doing the linear interpolation manually, based on the histogram data we get from the endpoints.
Test Plan
Ran a network locally, ran the benchmark against it, saw the p99 values being printed matched what Grafana reported as well, and that if we cross the threshold the benchmark gets stopped automatically.
Release Plan