Update renewed Jira refresh token back to the secrets store #5324

san81 · 2025-01-13T00:22:40Z

Description

In Jira OAuth2 based authentication, if one node renew the refresh token, others are not aware of the renewed refreshed token. This PR enables a way to write the renewed token back into the secret store so that other OCU nodes can pick up the renewed token and continue their execution. The changes introduced in this PR are

Introduced PluginConfigVariable interface that underlying provides methods to update the secret using the SecretsSupplier
Enhanced Jira source plugin code to make use of this newly introduced config variable member
Enahced Jira OAuth2 token refresh flow to make use of this PluginConfigVariable to write the updated refreshToken back into the store

Issues Resolved

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

… plugins to get access to their underlying aws secrets store member to be able to update if needed Signed-off-by: Santhosh Gandhe <[email protected]>

…ecrets store Signed-off-by: Santhosh Gandhe <[email protected]>

Signed-off-by: Santhosh Gandhe <[email protected]>

… used for refreshToken Signed-off-by: Santhosh Gandhe <[email protected]>

Signed-off-by: Santhosh Gandhe <[email protected]>

chenqi0805 · 2025-01-13T23:02:45Z

...plugins/aws-plugin/src/main/java/org/opensearch/dataprepper/plugins/aws/SecretsSupplier.java

+     * @param newValueToSet The value of the secret to be updated
+     * @return The version id of the secret after the update
+     */
+    String updateValue(String secretId, String keyToUpdate, Object newValueToSet);


The above method assumes JSON(key-value pair) secret value. I think we will need the same method with more generic signature

String updateValue(String secretId, Object newValueToSet);

to support plain text secret value. Or we can try to replace the above with this more generic use case.

newValuetoSet can be plain text as well. The json we are constructing is the format accepted by the aws api to update the secret store.

Maybe we just support null here. When we construct the AwsPluginConfigVariable, we can provide a null value in that situation.

Are we trying to support a value without having kay for it? keyToUpdate can be null is what you are suggesting @dlvenable ?

chenqi0805 · 2025-01-13T23:05:48Z

...ugin/src/main/java/org/opensearch/dataprepper/plugins/aws/AwsSecretManagerConfiguration.java

@@ -78,6 +79,14 @@ public GetSecretValueRequest createGetSecretValueRequest() {
                .build();
    }

+    public PutSecretValueRequest putSecretValueRequest(String keyToUpdate, Object newValue) {
+        String updatedSecretString = String.format("{\"%s\": \"%s\"}", keyToUpdate, newValue);


It is possible we have other key-value pairs in the existing secret string. Shall we not overwriting them?

Yes, you are right. I thought this will only update the specific key that we are passing in the json string based on reading the documentation. I will verify and put in the fix.

@chenqi0805 Fixed the logic to only update the specific key and keeping all the existing values as is.

Signed-off-by: Santhosh Gandhe <[email protected]>

dlvenable · 2025-01-14T16:29:36Z

...-prepper-api/src/main/java/org/opensearch/dataprepper/model/plugin/PluginConfigVariable.java

+    /**
+     * If this variable is updatable, this method helps to set a new value for this variable
+     */
+    void setValue(Object someValue);


Suggested change

void setValue(Object someValue);

void setValue(Object updatedValue);

Let's use a clearer name here.

modified 👍

dlvenable · 2025-01-14T16:29:56Z

...-prepper-api/src/main/java/org/opensearch/dataprepper/model/plugin/PluginConfigVariable.java

+ * Interface for a Extension Plugin configuration variable.
+ * It gives access to the details of a defined extension variable.
+ *
+ * @since 1.2


Suggested change

* @since 1.2

* @since 2.11

The next release is 2.11.

dlvenable · 2025-01-14T16:30:34Z

...epper-plugin-framework/src/main/java/org/opensearch/dataprepper/plugin/VariableExpander.java

@@ -48,8 +48,13 @@ public <T> T translate(final JsonParser jsonParser, final Class<T> destinationTy
                    .filter(entry -> entry.getKey().matches())
                    .map(entry -> {
                        final String valueReferenceKey = entry.getKey().group(VALUE_REFERENCE_KEY);
-                        return objectMapper.convertValue(
-                                entry.getValue().translate(valueReferenceKey), destinationType);
+                        if (destinationType.equals(PluginConfigVariable.class)) {


This needs to be unit tested.

Added unit tests to cover the cases 👍

dlvenable · 2025-01-14T16:32:19Z

...gins/aws-plugin/src/main/java/org/opensearch/dataprepper/plugins/aws/AwsSecretsSupplier.java

+    public String updateValue(String secretId, String keyToUpdate, Object newValue) {
+        final Map<String, Object> keyValuePairs = (Map<String, Object>) secretIdToValue.get(secretId);
+        keyValuePairs.put(keyToUpdate, newValue);
+        String secretKeyValueMapAsString = (String) retrieveValue(secretId);


Keys are optional, so we should have support for both keyed and keyless secrets.

A key just sets a convention of having a JSON object.

dlvenable · 2025-01-14T16:33:24Z

...-prepper-api/src/main/java/org/opensearch/dataprepper/model/plugin/PluginConfigVariable.java

+ *
+ * @since 1.2
+ */
+public interface PluginConfigVariable {


Please add a boolean as well to indicate if the PluginConfigVariable supports updates.

e.g.

boolean isSetValueSupported();

dlvenable · 2025-01-14T16:43:49Z

...test/java/org/opensearch/dataprepper/plugins/source/jira/utils/TestUtilForPrivateFields.java

+/**
+ * Utility class to help test private fields
+ */
+public class TestUtilForPrivateFields {


We already have this that you can use: ReflectivelySetField

data-prepper/data-prepper-test-common/src/main/java/org/opensearch/dataprepper/test/helper/ReflectivelySetField.java

Lines 29 to 37 in 6648d9f

public static <T> void setField(final Class<T> configurationClass, final Object configurationObject,

final String fieldName, final Object value) throws NoSuchFieldException, IllegalAccessException {

final Field field = configurationClass.getDeclaredField(fieldName);

try {

field.setAccessible(true);

field.set(configurationObject, value);

} finally {

field.setAccessible(false);

}

dlvenable · 2025-01-14T16:44:12Z

...r-api/src/main/java/org/opensearch/dataprepper/model/plugin/PluginConfigValueTranslator.java

@@ -4,4 +4,6 @@ public interface PluginConfigValueTranslator {
    Object translate(final String value);

    String getPrefix();
+
+    PluginConfigVariable translateToPluginConfigVariable(final String value);


Please add Javadocs to this method.

dlvenable · 2025-01-14T16:44:47Z

...-prepper-api/src/main/java/org/opensearch/dataprepper/model/plugin/PluginConfigVariable.java

+public interface PluginConfigVariable {
+
+    /**
+     * Returns the value of this variable.


Please complete the Javadocs for this. Your IDE should help.

e.g. @returns, @throws if applicable, @since

dlvenable · 2025-01-14T16:44:53Z

...-prepper-api/src/main/java/org/opensearch/dataprepper/model/plugin/PluginConfigVariable.java

+    Object getValue();
+
+    /**
+     * If this variable is updatable, this method helps to set a new value for this variable


Complete Javadocs.

dlvenable · 2025-01-14T16:46:00Z

...src/test/java/org/opensearch/dataprepper/plugins/aws/AwsSecretsPluginConfigVariableTest.java

@@ -0,0 +1,67 @@
+/*


Please also use the new copyright notice in this file:

/* * Copyright OpenSearch Contributors * SPDX-License-Identifier: Apache-2.0 * * The OpenSearch Contributors require contributions made to * this file be licensed under the Apache-2.0 license or a * compatible open source license. * */

san81 added 4 commits January 12, 2025 16:07

Introduced PluginConfigVariable interaface to provide ability for the…

aa7ca59

… plugins to get access to their underlying aws secrets store member to be able to update if needed Signed-off-by: Santhosh Gandhe <[email protected]>

renewed access token and refresh tokens are now updated back in the s…

0c34119

…ecrets store Signed-off-by: Santhosh Gandhe <[email protected]>

better naming

cc82281

Signed-off-by: Santhosh Gandhe <[email protected]>

Merge branch 'opensearch-project:main' into secrets-variable-interface

2f640df

san81 changed the title ~~Updated renewed Jira refresh token back to the secrets store~~ Update renewed Jira refresh token back to the secrets store Jan 13, 2025

san81 added 2 commits January 12, 2025 22:34

fixing the test cases based on the new PluginConfigVariable attribute…

64bf7bc

… used for refreshToken Signed-off-by: Santhosh Gandhe <[email protected]>

improving the coverage

4df4fba

Signed-off-by: Santhosh Gandhe <[email protected]>

san81 marked this pull request as ready for review January 13, 2025 23:01

san81 requested review from sb2k16, chenqi0805, engechas, graytaylor0, dinujoh, kkondaka, KarstenSchnitter, dlvenable and oeyh as code owners January 13, 2025 23:01

chenqi0805 reviewed Jan 13, 2025

View reviewed changes

Keeping the existing values in the secret. Just updating an existing key

a384d85

Signed-off-by: Santhosh Gandhe <[email protected]>

dlvenable requested changes Jan 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update renewed Jira refresh token back to the secrets store #5324

Update renewed Jira refresh token back to the secrets store #5324

san81 commented Jan 13, 2025 •

edited

Loading

chenqi0805 Jan 13, 2025

san81 Jan 14, 2025 •

edited

Loading

dlvenable Jan 14, 2025

san81 Jan 15, 2025

chenqi0805 Jan 13, 2025

san81 Jan 14, 2025

san81 Jan 14, 2025

dlvenable Jan 14, 2025

san81 Jan 15, 2025

dlvenable Jan 14, 2025

san81 Jan 15, 2025

dlvenable Jan 14, 2025

san81 Jan 15, 2025

dlvenable Jan 14, 2025

dlvenable Jan 14, 2025

dlvenable Jan 14, 2025

dlvenable Jan 14, 2025

dlvenable Jan 14, 2025

dlvenable Jan 14, 2025

dlvenable Jan 14, 2025

san81 Jan 15, 2025

	void setValue(Object someValue);
	void setValue(Object updatedValue);

	public static <T> void setField(final Class<T> configurationClass, final Object configurationObject,
	final String fieldName, final Object value) throws NoSuchFieldException, IllegalAccessException {
	final Field field = configurationClass.getDeclaredField(fieldName);
	try {
	field.setAccessible(true);
	field.set(configurationObject, value);
	} finally {
	field.setAccessible(false);
	}

Update renewed Jira refresh token back to the secrets store #5324

Are you sure you want to change the base?

Update renewed Jira refresh token back to the secrets store #5324

Conversation

san81 commented Jan 13, 2025 • edited Loading

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

san81 Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

san81 commented Jan 13, 2025 •

edited

Loading

san81 Jan 14, 2025 •

edited

Loading