[BUG] Index "analysis" section is not properly compared against the existing section

[rhoens]

What is the bug?

On these lines the existing_analysis section is checked that it exists, and is the same, as in analysis. The problem is the existing_analysis section was not properly converted back to python types, so you can get diffs such as the following:

existing_analysis[section][k] = {'max_shingle_size': '3', 'min_shingle_size': '2', 'output_unigrams': 'false', 'type': 'shingle'}
analysis[section][k] = {'max_shingle_size': 3, 'min_shingle_size': 2, 'output_unigrams': False, 'type': 'shingle'}

These should obviously match, but they don't.

How can one reproduce the bug?

Create an index with an analysis section (e.g.,:

class TestDocument(Document):
    text = Text(
        required=True,
        fields={
            "raw": Text(),
            "2_3_s": Text(analyzer="2_3_shingles"),
        },
    )
    # NOTE: You must have at least this below for your index, but change the name to something else.
    class Index:
        name = "test_index"
        settings: Dict[str, Any] = {
            "index": {
                "max_ngram_diff": 2,
            },
            "analysis": {
                "analyzer": {
                    "2_3_shingles": {
                        "tokenizer": "standard",
                        "filter": ["asciifolding", "2_3_shingles"],
                    },
                },
                "filter": {
                    "2_3_shingles": {
                        "type": "shingle",
                        "min_shingle_size": 2,
                        "max_shingle_size": 3,
                        "output_unigrams": False,
                    },
                },
            },
        }

client = ...
TestDocument.init(using=client)

Then add a field to the document and try again, this will fail.

What is the expected behavior?

The existing_analysis and analysis sections should be properly deduped

What is your host/environment?

AWS

Do you have any screenshots?

Not applicable

Do you have any additional context?

None currently

[dblock]

@rhoens Thanks for reporting this, want to try to write a failing test and maybe a fix?

[rhoens]

@dblock is there some code inside of opensearchpy that would convert the dictionary to "opensearch native" format (i.e., the format that we'd get back from the db)?

[dblock]

@rhoens I don't know. Maybe @saimedhi?