Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Loki] Add a configurable parameter for logs retention #177

Open
lukeskywlkr opened this issue Nov 20, 2024 · 0 comments
Open

[Loki] Add a configurable parameter for logs retention #177

lukeskywlkr opened this issue Nov 20, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@lukeskywlkr
Copy link
Contributor

With the release v4.0.0 we start using TSDB. This issue has been opened to track the addition of a parametric log retention in the loki configuration. We should use the compactor component to do so because it is the long term supported way to do this, as stated in the official documentation.
Using the compactor we should pay attention to users that still have not migrated their store to TSDB since compactor does not support legacy stores. Below there are some configuration examples.

Details ⚙️

  • Compactor specific configuration: as shown in the official documentation the compactor is responsible for logs retention. Here an example configuration can be found.

      compactor:
        working_directory: /data/retention
        compaction_interval: 10m
        retention_enabled: true
        retention_delete_delay: 2h
        retention_delete_worker_count: 150
        delete_request_store: s3
  • Limits configuration: the retention period is configured in the limits_config section of Loki configuration. Here is an example.

      ...
        limits_config:
          retention_period: 744h
      ...

    The value specified in retention_period is applied for all log streams. A more granular retention policy can be applied using retention_stream where a selector can be specified and the retention policy applies for all the streams that match the selector. Here an example:

      ...
        limits_config:
          retention_period: 744h
          retention_stream:
            - selector: '{namespace="dev"}'
               priority: 1
               period: 24h
      ...

    A per tenant override can also be specified to allow a further granular retention policy. Here the configuration:

      ...
        limits_config:
          retention_period: 744h
          retention_stream:
            - selector: '{namespace="dev"}'
               priority: 1
               period: 24h
          per_tenant_override_config: /etc/overrides.yaml
      ...

    Here the overrides.yaml example using the runtime overrides:

      overrides:
        "29":
            retention_period: 168h
            retention_stream:
            - selector: '{namespace="prod"}'
              priority: 2
              period: 336h
            - selector: '{container="loki"}'
              priority: 1
              period: 72h
        "30":
            retention_stream:
            - selector: '{container="nginx", level="debug"}'
              priority: 1
              period: 24h
@lukeskywlkr lukeskywlkr added the enhancement New feature or request label Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant