First cut of the Transformation Playground Frontend and Backend

[chelma]

Description

• Added an initial cut of the Transformation Playground backend
• The user is able to start up the Django webserver and POST against the transforms/index/ path to create some Python code that will transform an input ES 6.8 Index Settings w/ multi-type mappings or ES 7.10 Index Settings into equivalent OpenSearch 2.X settings. The API returns the transform code, the result of invoking the transform against the user's input, and the validation results.
• The backend takes the user's input JSON, runs it through Claude 3.5 Sonnet to generate some Python transformation code, loads the transform code, invokes the transform against the input, and creates/deletes each transformed Index setting against the OpenSearch cluster the user provided for testing.
• The user is also able to spin up a quick React/Cloudscape frontend and hit the backend in their web browser. The frontend takes in the user's input JSON and request a GenAI recommendation. They can clear their current transform as well. There's dropdowns for different configuration options, though currently the only one with different options is the Source, which supports both ES 6.8 and ES 7.10. The user is able to set static guidance for the GenAI recommendation to shape its output, as well as directly modify and test their own transformation code without involving GenAI.

Issues Resolved

• https://opensearch.atlassian.net/browse/MIGRATIONS-2249
• https://opensearch.atlassian.net/browse/MIGRATIONS-2257
• https://opensearch.atlassian.net/browse/MIGRATIONS-2258
• https://opensearch.atlassian.net/browse/MIGRATIONS-2260

Testing

• Added unit tests
• Ran manual tests. Example:

(venv) chelma@80a9970a4d02 tp_backend % python3 manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
December 10, 2024 - 19:14:58
Django version 5.1.4, using settings 'tp_backend.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

(venv) chelma@80a9970a4d02 LP04 % curl -X POST "http://127.0.0.1:8000/transforms/index/" -H "Content-Type: application/json" -d '
{
    "transform_language": "Python",
    "source_version": "Elasticsearch 6.8",
    "target_version": "OpenSearch 2.17",
    "input_shape": {
        "index_name": "test-index",
        "index_json": {
            "settings": {
                "index": {
                    "number_of_shards": 1,
                    "number_of_replicas": 0
                }
            },
            "mappings": {
                "type1": {
                    "properties": {
                        "title": { "type": "text" }
                    }
                },
                "type2": {
                    "properties": {
                        "contents": { "type": "text" }
                    }
                }
            }
        }
    },
    "test_target_url": "http://localhost:29200"
}'

{
    "output_shape": [
        {
            "index_name": "test-index-type1",
            "index_json": {
                "settings": {
                    "index": {
                        "number_of_shards": 1,
                        "number_of_replicas": 0
                    }
                },
                "mappings": {
                    "properties": {
                        "title": {
                            "type": "text"
                        }
                    }
                }
            }
        },
        {
            "index_name": "test-index-type2",
            "index_json": {
                "settings": {
                    "index": {
                        "number_of_shards": 1,
                        "number_of_replicas": 0
                    }
                },
                "mappings": {
                    "properties": {
                        "contents": {
                            "type": "text"
                        }
                    }
                }
            }
        }
    ],
    "transform_logic": "from typing import Dict, Any, List\nimport copy\n\n\"\"\"\nThis transformation function converts Elasticsearch 6.8 index settings to OpenSearch 2.17 compatible format.\nIt handles the removal of type-based mappings by creating separate indexes for each type.\n\"\"\"\n\ndef transform(source_json: Dict[str, Any]) -> List[Dict[str, Any]]:\n    result = []\n    index_name = source_json['index_name']\n    index_json = source_json['index_json']\n    \n    # Extract settings\n    settings = index_json.get('settings', {})\n    \n    # Extract mappings\n    mappings = index_json.get('mappings', {})\n    \n    # Create a separate index for each type\n    for type_name, type_mapping in mappings.items():\n        new_index_name = f\"{index_name}-{type_name}\"\n        new_index_json = {\n            'settings': copy.deepcopy(settings),\n            'mappings': type_mapping\n        }\n        \n        result.append({\n            'index_name': new_index_name,\n            'index_json': new_index_json\n        })\n    \n    return result",
    "validation_report": [
        "Attempting to load the transform function...",
        "Loaded the transform function without exceptions",
        "Attempting to invoke the transform function against the input...",
        "Invoked the transform function without exceptions",
        "The transformed output has 2 Index entries.",
        "Using target cluster for testing: http://localhost:29200",
        "Attempting to create & delete index 'test-index-type1' with transformed settings...",
        "Created index 'test-index-type1'.  Response: \n{\"acknowledged\": true, \"shards_acknowledged\": true, \"index\": \"test-index-type1\"}",
        "Deleted index 'test-index-type1'.  Response: \n{\"acknowledged\": true}",
        "Attempting to create & delete index 'test-index-type2' with transformed settings...",
        "Created index 'test-index-type2'.  Response: \n{\"acknowledged\": true, \"shards_acknowledged\": true, \"index\": \"test-index-type2\"}",
        "Deleted index 'test-index-type2'.  Response: \n{\"acknowledged\": true}"
    ],
    "validation_outcome": "PASSED"
}

The default view when you open the GUI, with an input ES 6.8 JSON filled in

The GenAI recommended transformation for that input JSON

The modal window for modifying the GenAI recommendations

The results of applying the user guidance in a new, GenAI recommend transform

Check List

• New functionality includes testing
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peternied]

(Same as the other issue) Can we tweak this code not to include the error in the response? This will set off alarms in our other security scanning tools such as our internal sonarqube instance.

[peternied]

Thanks for getting this in front of the team @chelma

I did not get a chance to refine the associated jira's in advance of this change arriving - I expect there is going to be back and forth on a number of comments I've raised.

[peternied]

Thanks for adding tests, lets make sure to update the CI.yml to run these tests so we have code coverage information as well.

[chelma]

Will investigate!

[peternied]

How do you cycle back and forth between source / transformation logic / output?

[chelma]

I haven't implemented this all yet (obviously), but here's the user journey I'm envisioning:

• The user loads in their input shapes. Currently, that's just copy/pasting JSON into a text field, but soon I'd like to be able to load the contents of a snapshot into the Playground via a configurable process. Imagine a dialogue where you select the S3 location of your snapshot, something extracts the templates, indices, and a configurable number of documents (that's a whole workstream right there), and presents them in the left panel so you can see everything in your snapshot, then click on specific items to see their JSON, etc. Even once we've added the ability to read from snapshots, we'll want to leave open the possibility of adding manual input shapes. Example - the user doesn't have access to a snapshot of their production cluster but they do have access to the settings JSON and a few documents, which they can manually enter as shapes into the Playground for testing.
• The user then uses the dropdowns to select what type of transformation they are performing. The "transform type" (Index vs. template vs. documents) dropdown will disappear, because it will be obvious from context (the user selected an index on the input panel, so...).
• Once that is selected, we will have a pre-canned transform (we'll hand-craft and save, no GenAI) that is supplied based on those selections that will serve as a default. The default will be populated into the UI and pinned against that shape in the backend.
• The user can then hit a "test" button to see how that default transform will work against their input shape (e.g. we kick off the validation process).
• If the user wants to include functional tests into validation (e.g. actually creating/deleting indices against a real test cluster), they can go through a dialogue to set up and test a connection to their target cluster. Currently, that process is just "paste a URL into a text field and assume there's no auth to worry about").
• If the user likes the results of the default, pre-canned transform and it passes validation - great! No further work needed. If not, they can either modify the transform directly in the UI and run the validation process again, or they can ask the GenAI assistant to modify the existing transform in some way (not currently implemented, but expect to be easy). Either way, the new transform is then pinned against the input shape in the backend so that we know there's something custom going on.
• The user then selects a different input shape and goes through this process until everything is transforming as they desire, then they can either hit a button to "deploy" their transformed data/metadata to the target cluster (think for metadata migrations which are inherently "low scale") or "bundle" the transformations for inclusion into the Migrations Assistant backfill/replay processes (which are anything but "low scale"). For the "deploy" button, this is basically just skipping the cleanup step of the validation process we've already performed (e.g. take the transformed output and PUT it against the target, but don't delete it afterwards). For bundling, this is taking the stuff we have in the server-side DB, packaging it into a tarball/zip/whatever, and sticking it somewhere (S3?) so that the backfill and replayer processes can pick it up and load the transform objects.

[peternied]

Can we tweak this code not to include the error in the response? This will set off alarms in our other security scanning tools such as our internal sonarqube instance.

[peternied]

(Same as the other issue) Can we tweak this code not to include the error in the response? This will set off alarms in our other security scanning tools such as our internal sonarqube instance.