Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Failed to rollover index #3864

Open
ElfoLiNk opened this issue Dec 21, 2023 · 3 comments
Open

[BUG] Failed to rollover index #3864

ElfoLiNk opened this issue Dec 21, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@ElfoLiNk
Copy link

Describe the bug
Rollover index is not working:

{
    "cause": "no permissions for [indices:admin/rollover] and associated roles [dataprepper, own_index]",
    "message": "Failed to rollover index [index=otel-v1-apm-span-000001]"
}

To Reproduce

Install dataprepper and use a specific user with the following role

# DataPrepper Role
dataprepper:
reserved: true
cluster_permissions:
  - cluster_all
  - indices:admin/template/get
  - indices:admin/template/put
index_permissions:
  - index_patterns:
      - 'otel-v1*'
      - '.opendistro-ism-config'
      - 'events-*'
      - 'metrics-*'
    allowed_actions:
      - 'indices_all'
  - index_patterns:
      - '*'
    allowed_actions:
      - 'manage_aliases'

Expected behavior

Rolling policy should work

Environment (please complete the following information):

  • OS: 2.11.0
  • Version 2.6.0
@asifsmohammed
Copy link
Collaborator

@ElfoLiNk This looks like an OpenSearch issue and not related to Data Prepper. Can you provide further details?

@ElfoLiNk
Copy link
Author

ElfoLiNk commented Jan 2, 2024

Hi @asifsmohammed i configured data prepper user and role following https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/opensearch_security.md the policy are created automatically by data prepper no?

I found this on OS project opensearch-project/security#1861

@JannikBrand
Copy link
Contributor

JannikBrand commented Apr 18, 2024

This is probably not related to an OpenSearch issue but to the permission setup of Data Prepper.
I am not using the OpenSearch admin user but my own defined user with the permissions listed here.
I could fix it by adding the "indices:admin/rollover" permission to the OpenSearch role:

data_prepper_role:
      reserved: true
      description: "role description"
      cluster_permissions:
        - "cluster_all"
        - "indices:admin/index_template/get"
        - "indices:admin/index_template/put"
      index_permissions:
        - index_patterns:
            - "otel-v1-apm*"
            - ".opendistro-ism-config"
          allowed_actions:
            - "indices_all"
        - index_patterns:
            - "*"
          allowed_actions:
            - "manage_aliases"
            - "indices:admin/rollover"

Background: An ISM policy seems to be linked to the user who created it. This makes sense, since otherwise a user which does not have e.g. permissions to delete indices could create an ISM policy which would delete indices. Previously (probably due to an older OpenSearch version), the rollover actions worked without having this permissions. I suspect that there was a change in some OpenSearch version update that linked the user to the ISM policy. I tested the mitigation with version 1.3.15.

When I checked out the .opendistro-ism-config index e.g. like this...

GET .opendistro-ism-config/_search
{
  "query": {
    "match_all": {}
  },
  "size": <select number to show all hits>
}

...there is an entry like the following:

{
  "_index" : ".opendistro-ism-config",
  "_type" : "_doc",
  "_id" : "OYi_mi3vQG6sLArwXLVBoA",
  "_score" : 1.0,
  "_routing" : "OYi_mi3vQG6sLArwXLVBoA",
  "_source" : {
    "managed_index" : {
      "name" : "otel-v1-apm-span-000001",
      "enabled" : false,
      "index" : "otel-v1-apm-span-000001",
      "index_uuid" : "OYi_mi3vQG6sLArwXLVBoA",
      "schedule" : {
        "interval" : {
          "start_time" : 1706018691311,
          "period" : 5,
          "unit" : "Minutes"
        }
      },
      "last_updated_time" : 1706780244142,
      "enabled_time" : null,
      "policy_id" : "raw-span-policy",
      "policy_seq_no" : -2,
      "policy_primary_term" : 0,
      "policy" : {
        "policy_id" : "raw-span-policy",
        "description" : "Managing raw spans for trace analytics",
        "last_updated_time" : 1706018690526,
        "schema_version" : 13,
        "error_notification" : null,
        "default_state" : "current_write_index",
        "states" : [
          {
            "name" : "current_write_index",
            "actions" : [
              {
                "retry" : {
                  "count" : 3,
                  "backoff" : "exponential",
                  "delay" : "1m"
                },
                "rollover" : {
                  "min_size" : "50gb",
                  "min_index_age" : "24h"
                }
              }
            ],
            "transitions" : [ ]
          }
        ],
        "ism_template" : [
          {
            "index_patterns" : [
              "otel-v1-apm-span-*"
            ],
            "priority" : 0,
            "last_updated_time" : 1706018690526
          }
        ],
        "user" : {
          "name" : "data_prepper_user",
          "backend_roles" : [ ],
          "roles" : [
            "own_index",
            "data_prepper_role"
          ],
          "custom_attribute_names" : [ ],
          "user_requested_tenant" : null
        }
      },
      "change_policy" : null,
      "jitter" : 0.6
    }
  }
} 

As you can see there is the otel-v1-apm-span-000001 index which is linked to the raw-span-policy which has a user field and a link to your data_prepper_role.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

No branches or pull requests

3 participants