Multiple federates

[JuanGRuedaV]

Hello. I have been running a co-simulation whose basic scenario is composed of 4 simulators. When I have tried to increase the number of federates by adding some of these same 4 basic-simulators up to 20-100 federates the simulations get stuck. All federates use the ZMQ core and the whole scenario runs on the same computer. I appreciate any help.

[trevorhardy]

Well, that shouldn't be happening 😉.

Any error messages showing up?

Is there a place where you can post your code and I can look at it? Otherwise, we probably need to start with some of the JSON config files (assuming you're doing external configuration).

You can also try turning up the logging level to "TRACE" which will dump a bunch of stuff into the log files and may shed some light.

Lastly, particularly if the co-simulation hangs, make sure the helics_broker is not running before trying to re-launch the new co-simulation.

[JuanGRuedaV]

https://gitlab.com/JuanGRuedaV/pruebas_helics
Here, you can see my code. The exercise has a composition of 4 federates (hdf5, hhsim, pypower and PV). In the newconfig.json file you can see that I'm running an scenario with 1 federate hdf5, 1 federate pypower, 15 federate hhsim and 15 federate PV which is running well. If i want to keep increasing the number of hhsim and PV federates that's when the co.simulation is getting stuck. I don't know if it could be a problem of pc software capabilities or something that i'm missing in my Json. Thank you.

[trevorhardy]

That link returns a 404 error for me; maybe the code isn't public?

[JuanGRuedaV]

yes, sorry, try again https://gitlab.com/JuanGRuedaV/pruebas_helics

[trevorhardy]

@JuanGRuedaV, could you add a config JSON to your repo for a version that works, just so I have something to compare this non-working one against?

[JuanGRuedaV]

@trevorhardy, one question: I think that the problem is that my pc has 32 cores and I'm with my configuration each federate is using a single core therefore with 202 federates is getteing stuck the simulation. Can you help me to clearify how can I define that many federates use a single same core? thank you

[trevorhardy]

I would be surprised if that was the problem. The operating system handles scheduling compute time for all the processes and our experience is the modern OSes do this reasonably well. For example, on my my computer with 8 cores, my macOS shows about 500 processes/threads running right now (most of them doing very little most of the time) and everything is running fine. I just talked to a guy yesterday who was running 1,000,000 federates on his laptop with HELICS successfully.

Do you have something specific you're seeing that makes you think the core count is a problem? For example, does it work at 30 or 31 federates but not at 33 or 34?

[JuanGRuedaV]

That last part you said was what was happening with v2.6, but know with the updated version seems to be solved. I'll keep updating the code and let you know. Thanks

[JuanGRuedaV]

Hello @trevorhardy. Ialready updated code to version 3.2.1, you can check it in the repository. I could run a simulation with newconfig_100.json working. The issue: with newconfig_500.json is giving me an error : helics.capi.HelicsException: [-101] core lost connection with broker in some of the federates. I appreciate any help, thank you.

[phlptp]

If you are running on the same machine, with significantly more federates that cores, what sometimes happens in the broker gets starved by the operating system for a while and the early federates end up timing out. A couple things you can do are switch to using the the ZMQSS core type. that uses a bit less resources with a bit of performance tradeoff. And increasing the timeout on the federates "--timeout=30s" or something along those lines. That would give a bit more time. Or if you just want to turn off the dead broker detection completely use the "--coreinitstring=--slow_responding" flag on the federate.