[CONTP-547] reduce language detection refresh period to 1 minute #1336
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pducolin
approved these changes
Jan 3, 2025
| @@ -344,6 +344,17 @@ func buildLinuxHelmValues(baseName, agentImagePath, agentImageTag, clusterAgentI | |||
| }, | |||
| "containers": pulumi.Map{ | |||
| "agent": pulumi.Map{ | |||
| "env": pulumi.StringMapArray{ | |||
| pulumi.StringMap{ | |||
| // TODO: remove this environment variable override once a retry mechanism is added to the language detection client | |||
There was a problem hiding this comment.
👏 praise
Seems to be indeed something that could impact customers too, though their environments run longer, so they are less impacted by start up flakiness
adel121
deleted the
adelhajhassan/reduce_languagedetection_client_refresh_period_to_1minute
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR fixes a flakiness in e2e test (See here).
Language detection client runs in the core agent, and periodically sends detected languages to the cluster agent.
When the cluster agent receives a detected language, it patches the related deployment with a language annotation.
The default period is 20 minutes.
If the first detected language is sent before the cluster agent is available, the cluster agent won't receive the detected language until after 20 minutes. This is caused by the absence of a retry mechanism in the language detection client.
As a result, some assertions in the E2E tests of datadog-agent fail, and are reported as flaky.
The ideal fix would be to implement a retry mechanism in the PLD client. In the meantime, a temporary fix to remove flakiness is to simply reduce the refresh period to 1 minute.
Which scenarios this will impact?
All scenarios installing the agent with helm.
Motivation
Fix flakiness in e2e.
Additional Notes