Adding troubleshooting guide #361
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yorugac
force-pushed
the
docs/troubleshooting
branch
2 times, most recently
from
4e4b803
to
ef74a74
Compare
javaducky
reviewed
Jan 25, 2024
docs/troublehooting.md
Outdated
| @@ -0,0 +1,242 @@ | |||
| # Troubleshooting | |||
|
|
|||
| Just as any Kubernetes application, k6-operator can get into error scenarios which are sometimes a result of misconfigured test or setup. This document is meant to help troubleshoot such scenarios quicker. | |||
There was a problem hiding this comment.
Suggested change
| Just as any Kubernetes application, k6-operator can get into error scenarios which are sometimes a result of misconfigured test or setup. This document is meant to help troubleshoot such scenarios quicker. | |
| Just as any Kubernetes application, k6-operator can get into error scenarios which are sometimes a result of a misconfigured test or setup. This document is meant to help troubleshoot such scenarios quicker. |
docs/troublehooting.md
Outdated
|
|
||
| ## Common tricks | ||
|
|
||
| ## :warning: Highly recommended! :warning: |
There was a problem hiding this comment.
Should we maybe have this block wrapped with > [!IMPORTANT]? See https://github.com/orgs/community/discussions/16925 for an example of these callouts.
I'm not sure how the code blocks will actually appear, though. 🤔
There was a problem hiding this comment.
Surprisingly, it is rendered fine:
docs/troublehooting.md
Outdated
| MY_ENV_VAR=foo k6 run script.js --tag my_tag=bar | ||
| ``` | ||
|
|
||
| This ensures that the script has correct syntax and can be parsed with k6 in the first place. Additionally, local run will make it obvious if the configured options are doing what is expected. If there are any errors or unexpected results in the output of `k6 run`, make sure to fix those prior to deploying the script elsewhere. |
There was a problem hiding this comment.
Suggested change
| This ensures that the script has correct syntax and can be parsed with k6 in the first place. Additionally, local run will make it obvious if the configured options are doing what is expected. If there are any errors or unexpected results in the output of `k6 run`, make sure to fix those prior to deploying the script elsewhere. | |
| This ensures that the script has correct syntax and can be parsed with k6 in the first place. Additionally, running locally will make it obvious if the configured options are doing what is expected. If there are any errors or unexpected results in the output of `k6 run`, make sure to fix those prior to deploying the script elsewhere. |
docs/troublehooting.md
Outdated
|
|
||
| #### The pods | ||
|
|
||
| In case of one `TestRun` CR creation with `parallelism: n`, there are certain repeating patterns: |
There was a problem hiding this comment.
Custom Resource? We may want to be explicit (at least the first occurrence) writing as "Custom Resource (CR)."
docs/troublehooting.md
Outdated
| kubectl logs mytest-initializer-xxxxx | ||
| ``` | ||
|
|
||
| If the Pods seem to be working but not producing an expected result and there's not enough information in the logs of the Pods, it might make sense to turn on k6 [verbose option](https://k6.io/docs/using-k6/k6-options/reference/#options) in `TestRun` spec: |
There was a problem hiding this comment.
Suggested change
| If the Pods seem to be working but not producing an expected result and there's not enough information in the logs of the Pods, it might make sense to turn on k6 [verbose option](https://k6.io/docs/using-k6/k6-options/reference/#options) in `TestRun` spec: | |
| If the Pods seem to be working but not producing an expected result and there's not enough information in the logs of the Pods, it might make sense to turn on k6 [verbose option](https://grafana.com/docs/k6/latest/using-k6/k6-options/#options) in `TestRun` spec: |
docs/troublehooting.md
Outdated
|
|
||
| ### `PrivateLoadZone` deployment | ||
|
|
||
| If `PrivateLoadZone` CR was successfully created in Kubernetes, it should become visible in your account in GCk6 interface soon afterwards. If it doesnt appear in UI, then likely there is a problem to troubleshoot. |
There was a problem hiding this comment.
Suggested change
| If `PrivateLoadZone` CR was successfully created in Kubernetes, it should become visible in your account in GCk6 interface soon afterwards. If it doesnt appear in UI, then likely there is a problem to troubleshoot. | |
| If `PrivateLoadZone` CR was successfully created in Kubernetes, it should become visible in your account in GCk6 interface soon afterwards. If it doesn't appear in the UI, then there is likely a problem to troubleshoot. |
docs/troublehooting.md
Outdated
|
|
||
| ### Initializer logs an error but it's not about tags | ||
|
|
||
| Often, this happens because of lack of attention to the [highly recommended](#⚠️-highly-recommended-⚠️) step. One more command that can be tried here is to run the following: |
There was a problem hiding this comment.
🎗️ Don't forget to update if the anchor changes for the warning.
docs/troublehooting.md
Outdated
| ServiceAccount can be defined as `serviceAccountName` and `runner.serviceAccountName` in PrivateLoadZone and TestRun CRD respectfully. If the specified ServiceAccount does not exist, k6-operator will successfully create Jobs but corresponding Pods will fail to be deployed, and k6-operator will wait indefinitely for Pods to be `Ready`. This error can be best seen in the events of the Job: | ||
|
|
||
| ```bash | ||
| kubectl describe job plz-test-154546-1 |
There was a problem hiding this comment.
Suggested change
| kubectl describe job plz-test-154546-1 | |
| kubectl describe job plz-test-xxxxxx-x |
docs/troublehooting.md
Outdated
| kubectl describe job plz-test-154546-1 | ||
| ... | ||
| Events: | ||
| Warning FailedCreate 57s (x4 over 2m7s) job-controller Error creating: pods "plz-test-154546-1-" is forbidden: error looking up service account plz-ns/plz-sa: serviceaccount "plz-sa" not found |
There was a problem hiding this comment.
Suggested change
| Warning FailedCreate 57s (x4 over 2m7s) job-controller Error creating: pods "plz-test-154546-1-" is forbidden: error looking up service account plz-ns/plz-sa: serviceaccount "plz-sa" not found | |
| Warning FailedCreate 57s (x4 over 2m7s) job-controller Error creating: pods "plz-test-xxxxxx-x-" is forbidden: error looking up service account plz-ns/plz-sa: serviceaccount "plz-sa" not found |
Also...not sure if the trailing "-" was originally a typo, or valid. 🤔
There was a problem hiding this comment.
Yes, actually. It's a funny thing but apparently job-controller outputs the prefix for future pod here, hence with hyphen.
docs/troublehooting.md
Outdated
| This case is very similar to [ServiceAccount one](#non-existent-serviceaccount): the Pod creation will fail, only the error would be somewhat different: | ||
|
|
||
| ```bash | ||
| kubectl describe job plz-test-154546-1 |
There was a problem hiding this comment.
Suggested change
| kubectl describe job plz-test-154546-1 | |
| kubectl describe job plz-test-xxxxxx-x |
There was a problem hiding this comment.
This one should actually be the Pod description 🤦 I copied a wrong command; fixing.
yorugac
force-pushed
the
docs/troubleshooting
branch
from
137e6c8
to
5d7efc4
Compare
|
Thanks for the review @javaducky! 🙌 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is initial version for troubleshooting guide, issue: #331
It is expected to be iteratively improved in the future.