MQTT - "As individual topics" seemingly produces nested topics

[nicoleise]

DESCRIPTION

NOTE: Version 3.7.1 used. I'm unable to apply the dev branch firmware using either Settings/Firmware or Settings/"Upload/Download" . The system log just prints 2025-01-11 22:08:50.838 E 17: [system] Firmware upload failed - HTTP code -1.

My actual problem:
MQTT topics seem to be nested in topics containing a long JSON string regardless of the setting "Topic/Payload format" under Settings/MQTT. Perhaps I misunderstand the option, but it seems to be pretty self explainatory that "as individual topics" would suggest everything is published as... well... individual topics. The documentation has nothing on this setting apart from the obvious (that it exists and has two settings - not anything about the outcome to expect), so I'm not entirely sure I'm understanding it right.

I restarted the gateway and broker as stated in the documentation.

A secondary problem:
The checkbox to activate Modbus is greyed out and cannot be enabled. There's nothing in the docs to suggest why this is? I don't necessarily need modbusTcp, but was thinking I could use it along side MQTT since it would obviously have to behave as desired (individual "topics" for each data point).

REQUESTED INFORMATION

{
  "system": {
    "version": "3.7.1",
    "uptime": "000+00:12:59.298",
    "uptimeSec": 779,
    "platform": "ESP32",
    "cpuType": "ESP32-D0WD-V3",
    "arduino": "Tasmota Arduino v2.0.17",
    "sdk": "4.4.8.240628",
    "freeMem": 187,
    "maxAlloc": 107,
    "freeCaps": 4194,
    "usedApp": 1740,
    "freeApp": 6388,
    "partition": "app0",
    "resetReason": "Software reset CPU / Software reset CPU",
    "psram": false,
    "model": "BBQKees Gateway Model E32 V2 vE32V2.0P3/2024011"
  },
  "network": {
    "network": "WiFi",
    "hostname": "ems-gthp",
    "RSSI": -73,
    "WIFIReconnects": 0,
    "BSSID": "set",
    "TxPowerSetting": 78,
    "staticIP": false,
    "lowBandwidth": false,
    "disableSleep": true,
    "enableMDNS": false,
    "enableCORS": false,
    "APProvisionMode": "disconnected",
    "APSecurity": "wpa2",
    "APSSID": "ems-esp"
  },
  "ntp": {
    "NTPStatus": "connected",
    "enabled": true,
    "server": "time.google.com",
    "tzLabel": "Europe/Copenhagen"
  },
  "mqtt": {
    "MQTTStatus": "connected",
    "MQTTPublishes": 631,
    "MQTTQueued": 0,
    "MQTTPublishFails": 0,
    "MQTTReconnects": 0,
    "enabled": true,
    "clientID": "gthp",
    "keepAlive": 60,
    "cleanSession": false,
    "entityFormat": 4,
    "base": "vk01",
    "discoveryPrefix": "homeassistant",
    "discoveryType": 0,
    "nestedFormat": 2,
    "haEnabled": true,
    "mqttQos": 0,
    "mqttRetain": false,
    "publishTimeHeartbeat": 60,
    "publishTimeBoiler": 10,
    "publishTimeThermostat": 10,
    "publishTimeSolar": 10,
    "publishTimeMixer": 10,
    "publishTimeWater": 10,
    "publishTimeOther": 10,
    "publishTimeSensor": 10,
    "publishSingle": false,
    "publish2command": false,
    "sendResponse": false
  },
  "syslog": {
    "enabled": false
  },
  "sensor": {
    "temperatureSensors": 0,
    "temperatureSensorReads": 0,
    "temperatureSensorFails": 0,
    "analogSensors": 0,
    "analogSensorReads": 0,
    "analogSensorFails": 0
  },
  "api": {
    "APICalls": 0,
    "APIFails": 0
  },
  "bus": {
    "busStatus": "disconnected",
    "busProtocol": "HT3",
    "busTelegramsReceived": 1847,
    "busReads": 629,
    "busWrites": 0,
    "busIncompleteTelegrams": 0,
    "busReadsFailed": 0,
    "busWritesFailed": 0,
    "busRxLineQuality": 100,
    "busTxLineQuality": 100
  },
  "settings": {
    "boardProfile": "E32V2",
    "locale": "en",
    "txMode": 1,
    "emsBusID": 11,
    "showerTimer": false,
    "showerMinDuration": 180,
    "showerAlert": false,
    "hideLed": false,
    "noTokenApi": false,
    "readonlyMode": false,
    "fahrenheit": false,
    "dallasParasite": false,
    "boolFormat": 1,
    "boolDashboard": 1,
    "enumFormat": 1,
    "analogEnabled": true,
    "telnetEnabled": true,
    "maxWebLogBuffer": 25,
    "webLogBuffer": 17,
    "modbusEnabled": false,
    "forceHeatingOff": false,
    "developerMode": false
  },
  "devices": [
    {
      "type": "boiler",
      "name": "CUH/Enviline, Compress 6000AW, Hybrid 3000-7000iAW, SupraEco/Geo 5xx, WLW196i/WSW196i",
      "deviceID": "0x08",
      "productID": 172,
      "brand": "Bosch",
      "version": "03.03",
      "entities": 157,
      "handlersReceived": "0xBF 0xC2 0xD1 0xE3 0xE4 0xE5 0xE9 0x0494 0x0495 0x048F",
      "handlersFetched": "0x14 0xE6 0xEA 0x048D 0x048A 0x04A2 0x0485 0x0486 0x0492 0x0488 0x0484 0x048B 0x0491 0x0499 0x049C 0x049D 0x02CC 0x04A5 0x04AA 0x04A7 0x02D6",
      "handlersPending": "0x10 0x11 0x15 0x1C 0x18 0x19 0x1A 0x35 0x34 0x2A",
      "handlersIgnored": "0xE7 0x02E0 0x02EA 0x061E 0x0240 0x0291 0x029B 0x0489 0x04AC 0xF7 0x2E 0x3B"
    },
    {
      "type": "thermostat",
      "name": "HMI800.2/Rego 3000, UI800, Logamatic BC400",
      "deviceID": "0x10",
      "productID": 253,
      "brand": "",
      "version": "47.07",
      "entities": 71,
      "handlersReceived": "0x06 0x02BA 0x02BB 0x02BC 0x031D 0x0267",
      "handlersFetched": "0x02A5 0x02B9 0x02AF 0x029B 0x0471 0x02CC 0x0467 0x0291 0x0292 0x0293 0x0294 0x02F5 0x023A 0x0240 0xBB 0x023E",
      "handlersPending": "0xA3 0xA2 0x12 0x13 0x02A6 0x02B0 0x029C 0x0472 0x02A7 0x02B1 0x029D 0x0473 0x02A8 0x02B2 0x029E 0x0474 0x02A9 0x02BD 0x02B3 0x029F 0x0475 0x02AA 0x02BE 0x02B4 0x02A0 0x0476 0x02AB 0x02BF 0x02B5 0x02A1 0x0477 0x02AC 0x02C0 0x02B6 0x02A2 0x0478 0x02CE 0x0468 0x02D0 0x0469 0x02D2 0x046A 0x031E",
      "handlersIgnored": "0x0507 0x0508 0x0509 0x050A 0x059D 0xBF"
    }
  ]
}

TO REPRODUCE

Setup the gateway with a valid device with points.
Enable MQTT with the option Topic/Payload Format set to As individual topics.
Subscribe a different device to the main/base topic (/#) to observe.

EXPECTED BEHAVIOUR

I would expect that the option "nested in a single topic" provides a single topic per device (or similar), thus maybe 3-5 topics in total; the online/offline topic, the gateway itself, the heatpump/boiler and the thermostat in a fairly minimal setup. This seems to occur, I see JSON content similar to this when subscribing (abbreviated):

Message 2286 received on vk01/boiler_data at 22:18:
{
    "reset": "",
    "heatingoff": "off",
    "heatingactive": "on",
    "tapwateractive": "off",
    "selflowtemp": 48,
    "heatingpumpmod": 86,
    "outdoortemp": 1.1,
    "curflowtemp": 46.1,
    "rettemp": 37.1,
    "headertemp": 46.1,
    "heatingactivated": "on",
    "heatingtemp": 80,
    "heatingpump": "on",
    "pumpmodmax": 0,
    "pumpmodmin": 0,
    "pumpcharacter": "proportional",
    "selburnpow": 72,
    "curburnpow": 70,
[...]
}

I would then expect the option "as individual topics" to provide each data point in an individual topic, as per the wording. In other words, I'd expect to see stuff like this instead:

vk01/boiler_data/reset "off"
vk01/boiler_data/heatingoff "off"
vk01/boiler_data/curflowtemp 46,1
[...]

ADDITIONAL CONTEXT

The Modbus issue is secondary, but being able to enable that and set that up would likely also solve the problem. At the core; I am trying to integrate the gateway into a BMS, and when subscribing to the MQTT topics, I get them into the system as a very long string. There are tools to process this JSON string and split it up into individual datapoints, but the tool requires an ongoing fee that I'd like to avoid.

[proddy]

The Modbus feature requires more memory, so it is only activated with ESP32 boards equipped with additional PSRAM. I agree, this could be made more clearer in the documentation.

The single topic we'll need to look into.

Remember this project is open source so welcome any PRs to help the community.

[nicoleise]

Thanks Proddy,

The Modbus feature requires more memory, so it is only activated with ESP32 boards equipped with additional PSRAM. I agree, this could be made more clearer in the documentation.

Ah - when searching yesterday, I found a thread where you discussed the memory usage with someone who helped develop the feature (I think), and where it seemed memory usage was very minimal. So I hadn't considered such a limitation.

Suggestion: Using the same criteria which greys out the checkbox, there could simply appear text straight in the UI. Like "The feature is not available on this hardware" or similar.

The single topic we'll need to look into.

Alright, let me know if I can contribute in any meaningful way.

Remember this project is open source so welcome any PRs to help the community.

Absolutely, and I love the product. I use (and prefer) many open source projects, and often "contribute" ideas, feedback, etc. I have no experience with "actual contributions" though. I think I know what a PR is, but no idea of the practicality involved. Also only modest coding skills. Sadly. But if I can help, I don't mind.

[proddy]

I'll update the documentation explaining what MQTT 'nested' does. Publishing each entity in its own MQTT topic would be too much traffic (there could to 300+ publishes every few seconds) and crash the ESP32.

Good idea about mention Modbus not support in the WebUI. If you want to use it see if BBQKees can upgrade your E32 board to the latest V2 which has PSRAM.

[proddy]

I added your suggestion to the WebUI, thanks for that.

[MichaelDvP]

There are different options. The selection nested/individual always shows json and affects the periodic sendings.
nested: topic thermostat_data payload{...,"hc1":{...},"dhw":{...} }
individual: topic: thermostat_data_hc1 payload: {....}
topic: thermostat_data_dhw payload: {....}

There is another option: Publish single value topics on change
This will publish a value, not a json and topic is the entity. e.g.topic thermostat_data/hc1/seltemp payload: 21
But this is only on change to avoid flooding the mqtt queue.

[nicoleise]

The "publish single value topics on change" is disabled if "Enable MQTT Discovery" is enabled. @MichaelDvP is this by design? Also probably nicer to grey it out and put in a message like I did with Modbus in dev-10

Oh yeah. You are right, the option is there for me if I disable discovery. I agree with your suggestion, and then like the Modbus grey out the checkbox and add text like "Publish single value topics on change is only available when discovery is MQTT disabled."

And obviously the same for enable MQTT discovery; grey checkbox and text "MQTT discovery is only available when publish single value topics on change is disabled."

[nicoleise]

yes, the + means there is additional PSRAM on the board, I think the E32 V2 has 4MB (you can check in Status->Hardware). Remember, always make a backup (Settings->Download/Upload)

That worked. In relation to #2151 is it maybe a simple workaround to check the PSRAM total size and if > 4000 add '+' to the download target before the extension? It seems that might catch a lot of the issues? I don't know if it can be that simple or if that breaks other stuff.

Also, my board does indeed have 4 MB. But this means I should be able to enable Modbus, right? It was not possible before I upgraded, as you know. Now on v. 3.7.2 dev 10 the checkbox is not greyed out and if enabled I see parameters appear. So it could potentially have been some other bug.

[nicoleise]

There is another option: Publish single value topics on change This will publish a value, not a json and topic is the entity. e.g.topic thermostat_data/hc1/seltemp payload: 21 But this is only on change to avoid flooding the mqtt queue.

Just wanted to confirm that this did the trick, data is received by the BMS. Wonderful! Thank you so much. :)

[proddy]

That worked. In relation to #2151 is it maybe a simple workaround to check the PSRAM total size and if > 4000 add '+' to the download target before the extension? It seems that might catch a lot of the issues? I don't know if it can be that simple or if that breaks other stuff.

That's how it works now. I think you'll find most of these quirks were resolved in 3.7.2, if not let me know!

[MichaelDvP]

The "publish single value topics on change" is disabled if "Enable MQTT Discovery" is enabled. @MichaelDvP is this by design?

Yes, it's by design. discovery only refers to the json publish.
And on startup the discovery sends a message for every found entity, the single topic also sends on startup every found entiy, that will blow the queue.