No need to notify agents for config update, if agents run with local config

[sophie-cluml]

Node's draft is an empty string when agents are told to run with the local configuration. So it would be useful to check the draft value in the applyNode GraphQL API and use that information to determine whether to notify each agent. This will be useful when the agents in the node are configured for the first time.

[sehkone]

@msk, before proceeding with PR #342 of this issue, I wanted to check if you have any further suggestions.

We need to differentiate cases where an agent is running on a node with its local configuration. To handle this, we set draft to an empty string to represent it, whereas when draft is set to None, it indicates that the agent is not enabled on the node. Could you check if this approach might any potential challenges?

[msk]

The change itself seems reasonable, and I wouldn’t mind seeing the PR merged. But since I haven’t been following the conversation on local configuration, I could use a bit more context to provide my perspective if that would still be helpful here.

In particular, I’m curious about a few points:

1. Under what circumstances should agents be set to run with local configurations? Is this something expected in regular agent operation, or is it more of a specialized use (testing, troubleshooting, etc.)?
2. When agents run locally configured, is that configuration initiated on the machine running the agent, or is it remotely set by an admin via the GraphQL API?
3. What is the expected behavior if an agent receives a config update notification while it’s operating with a local configuration? Should it ignore the update, or handle it differently?

I'd appreciate any details you can share.

[sehkone]

1. Under what circumstances should agents be set to run with local configurations? Is this something expected in regular agent operation, or is it more of a specialized use (testing, troubleshooting, etc.)?

Except for REconverge, all other agents use local configurations only for testing or troubleshooting. REconverge is difficult for users to configure through the UI at the moment, so it should be configured locally until we establish an easier configuration method.

2. When agents run locally configured, is that configuration initiated on the machine running the agent, or is it remotely set by an admin via the GraphQL API?

The latter case can be ruled out.

3. What is the expected behavior if an agent receives a config update notification while it’s operating with a local configuration? Should it ignore the update, or handle it differently?

Running with a local configuration should take priority, as it indicates testing, troubleshooting, or other contingencies. Therefore, the agent should ignore the update while notifying the central server.

[msk]

Thanks for the additional context. I don't have any further concerns with the PR, and I think it's good to go.

However, I did have a couple of follow-up questions regarding the original issue description. The original issue description says that

draft is an empty string when agents are told to run with the local configuration.

That implies that the web client (or the server) somehow knows that the agent is running with a local configuration. But @sehkone confirmed that the local configuration is initiated on the machine running the agent, not remotely via the GraphQL API.

So, I'm curious---is the agent supposed to report back to the server that it's running with a local configuration, or is there another mechanism at play here? I'm just trying to clarify the flow of information.

[sehkone]

The agent should report back to the server only when it receives a notification of a configuration update and is running with a local configuration. In all other cases, such as upon initialization, reporting is unnecessary.

In the UI, users initially set up information indicating that a specific agent is running with a local configuration, so this information might be incorrect.