How to avoid rejected contributions?

[wanderine]

I seem to have many rejected contributions, how can I reduce that? Increase train_timeout ? It doesn't say why the contributions are rejected.

[YuanTingHsieh]

Thanks for raising an issue.

Can you share your logs here so we know more details?

[wanderine]

By the way, why does the client output not show REJECTED / ACCEPTED ?

Server

2023-08-09 02:58:40,056 - DXOAggregator - WARNING - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius1, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=1c35141e-523f-4559-9599-c0a65a450584]: discarding DXO from berzelius1 at round: 532. Current round is: 533
2023-08-09 02:58:40,056 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius1, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=1c35141e-523f-4559-9599-c0a65a450584]: Contribution from berzelius1 REJECTED by the aggregator.
2023-08-09 02:58:40,056 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius1, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=1c35141e-523f-4559-9599-c0a65a450584]: Result of unknown task train sent to aggregator.
2023-08-09 02:58:40,056 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius1, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=1c35141e-523f-4559-9599-c0a65a450584]: finished processing client result by scatter_and_gather
2023-08-09 02:58:41,987 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius1, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70436e68-e981-4b00-b18b-52c2d3281ce5]: assigned task to client berzelius1: name=train, id=70436e68-e981-4b00-b18b-52c2d3281ce5
2023-08-09 02:58:41,988 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius1, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70436e68-e981-4b00-b18b-52c2d3281ce5]: sent task assignment to client
2023-08-09 02:58:42,394 - FederatedServer - INFO - GetTask: Return task: train to client: berzelius1 (98aaae17-5499-4e8f-b538-361343d1c5f3)
2023-08-09 02:58:50,287 - FederatedServer - INFO - received update from berzelius_berzelius6_0 (18826885 Bytes, 1691542730 seconds)
2023-08-09 02:58:50,528 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got result from client berzelius6 for task: name=train, id=fe3ae6bb-f244-4832-85d9-d1d314fd00fc
2023-08-09 02:58:50,529 - IntimeModelSelector - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=fe3ae6bb-f244-4832-85d9-d1d314fd00fc]: validation metric 0.8704215698772007 from client ?
2023-08-09 02:58:50,529 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=fe3ae6bb-f244-4832-85d9-d1d314fd00fc]: invoking result_received_cb ...
2023-08-09 02:58:50,534 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=fe3ae6bb-f244-4832-85d9-d1d314fd00fc]: Contribution from berzelius6 ACCEPTED by the aggregator.
2023-08-09 02:58:50,534 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=fe3ae6bb-f244-4832-85d9-d1d314fd00fc]: finished processing client result by scatter_and_gather
2023-08-09 02:58:54,983 - FederatedServer - INFO - received update from berzelius_berzelius5_0 (18826885 Bytes, 1691542734 seconds)
2023-08-09 02:58:55,217 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got result from client berzelius5 for task: name=train, id=09416599-ba14-40f7-9f08-adf3b2417d21
2023-08-09 02:58:55,217 - IntimeModelSelector - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=09416599-ba14-40f7-9f08-adf3b2417d21]: validation metric 0.8704215698772007 from client ?
2023-08-09 02:58:55,218 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=09416599-ba14-40f7-9f08-adf3b2417d21]: invoking result_received_cb ...
2023-08-09 02:58:55,228 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=09416599-ba14-40f7-9f08-adf3b2417d21]: Contribution from berzelius5 ACCEPTED by the aggregator.
2023-08-09 02:58:55,228 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=09416599-ba14-40f7-9f08-adf3b2417d21]: finished processing client result by scatter_and_gather
2023-08-09 02:59:00,773 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: task train exit with status TaskCompletionStatus.OK
2023-08-09 02:59:01,257 - FederatedServer - INFO - received update from berzelius_berzelius4_0 (18826885 Bytes, 1691542741 seconds)
2023-08-09 02:59:01,264 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: Start aggregation.
2023-08-09 02:59:01,264 - DXOAggregator - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: aggregating 2 update(s) at round 533
2023-08-09 02:59:01,273 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: End aggregation.
2023-08-09 02:59:01,283 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: Start persist model on server.
2023-08-09 02:59:01,333 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: End persist model on server.
2023-08-09 02:59:01,333 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: Round 533 finished.
2023-08-09 02:59:01,333 - ServerEngine - INFO - Start saving snapshot on server.
2023-08-09 02:59:01,506 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got result from client berzelius4 for task: name=train, id=8a5b90f6-22ea-4b85-8487-2a684bb8a242
2023-08-09 02:59:01,507 - IntimeModelSelector - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=8a5b90f6-22ea-4b85-8487-2a684bb8a242]: validation metric 0.8704215698772007 from client ?
2023-08-09 02:59:01,507 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=8a5b90f6-22ea-4b85-8487-2a684bb8a242]: no standing task found for train:8a5b90f6-22ea-4b85-8487-2a684bb8a242
2023-08-09 02:59:01,507 - DXOAggregator - WARNING - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=8a5b90f6-22ea-4b85-8487-2a684bb8a242]: discarding DXO from berzelius4 at round: 533. Current round is: 534
2023-08-09 02:59:01,508 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=8a5b90f6-22ea-4b85-8487-2a684bb8a242]: Contribution from berzelius4 REJECTED by the aggregator.
2023-08-09 02:59:01,508 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=8a5b90f6-22ea-4b85-8487-2a684bb8a242]: Result of unknown task train sent to aggregator.
2023-08-09 02:59:01,509 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=8a5b90f6-22ea-4b85-8487-2a684bb8a242]: finished processing client result by scatter_and_gather
2023-08-09 02:59:01,834 - ServerEngine - INFO - persist the snapshot to: /tmp/nvflare/snapshot-storage/f83759c6-ca8a-4c47-8b65-b703e06610c7
2023-08-09 02:59:01,838 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: Round 534 started.
2023-08-09 02:59:01,839 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: scheduled task train
2023-08-09 02:59:02,582 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=ffc2c053-8347-4012-a761-781e05ada4f9]: assigned task to client berzelius6: name=train, id=ffc2c053-8347-4012-a761-781e05ada4f9
2023-08-09 02:59:02,583 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=ffc2c053-8347-4012-a761-781e05ada4f9]: sent task assignment to client
2023-08-09 02:59:02,990 - FederatedServer - INFO - GetTask: Return task: train to client: berzelius6 (ba4632ac-efc9-4bc7-87c2-7d24ef5871c9)
2023-08-09 02:59:03,421 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=674c7e89-278f-42fe-84f3-012ea8d0ef9f]: assigned task to client berzelius4: name=train, id=674c7e89-278f-42fe-84f3-012ea8d0ef9f
2023-08-09 02:59:03,422 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius4, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=674c7e89-278f-42fe-84f3-012ea8d0ef9f]: sent task assignment to client
2023-08-09 02:59:03,768 - FederatedServer - INFO - GetTask: Return task: train to client: berzelius4 (d76aac3a-cc66-4887-959b-4bb5952a8345)
2023-08-09 02:59:03,775 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=14b37e16-8ab0-4452-aa03-045fbd53680e]: assigned task to client berzelius5: name=train, id=14b37e16-8ab0-4452-aa03-045fbd53680e
2023-08-09 02:59:03,775 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=14b37e16-8ab0-4452-aa03-045fbd53680e]: sent task assignment to client
2023-08-09 02:59:04,134 - FederatedServer - INFO - GetTask: Return task: train to client: berzelius5 (0a8a7968-1b95-4d84-819f-22d84839735b)
2023-08-09 02:59:42,409 - FederatedServer - INFO - received update from berzelius_berzelius2_0 (18826885 Bytes, 1691542782 seconds)
2023-08-09 02:59:42,663 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got result from client berzelius2 for task: name=train, id=ba433f37-d699-4941-aace-fc272b90cead
2023-08-09 02:59:42,665 - IntimeModelSelector - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ba433f37-d699-4941-aace-fc272b90cead]: validation metric 0.8704215698772007 from client ?
2023-08-09 02:59:42,665 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ba433f37-d699-4941-aace-fc272b90cead]: no standing task found for train:ba433f37-d699-4941-aace-fc272b90cead
2023-08-09 02:59:42,665 - DXOAggregator - WARNING - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ba433f37-d699-4941-aace-fc272b90cead]: discarding DXO from berzelius2 at round: 533. Current round is: 534
2023-08-09 02:59:42,666 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ba433f37-d699-4941-aace-fc272b90cead]: Contribution from berzelius2 REJECTED by the aggregator.
2023-08-09 02:59:42,666 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ba433f37-d699-4941-aace-fc272b90cead]: Result of unknown task train sent to aggregator.
2023-08-09 02:59:42,666 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ba433f37-d699-4941-aace-fc272b90cead]: finished processing client result by scatter_and_gather
2023-08-09 02:59:44,590 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=69d77502-d48e-4e42-8ef6-dd10f057a9b0]: assigned task to client berzelius2: name=train, id=69d77502-d48e-4e42-8ef6-dd10f057a9b0
2023-08-09 02:59:44,591 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius2, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=69d77502-d48e-4e42-8ef6-dd10f057a9b0]: sent task assignment to client
2023-08-09 02:59:45,010 - FederatedServer - INFO - GetTask: Return task: train to client: berzelius2 (c9c157b2-a5be-4baf-9bf7-4fd61aa03416)
2023-08-09 03:00:09,051 - FederatedServer - INFO - received update from berzelius_berzelius6_0 (18826885 Bytes, 1691542809 seconds)
2023-08-09 03:00:09,330 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got result from client berzelius6 for task: name=train, id=ffc2c053-8347-4012-a761-781e05ada4f9
2023-08-09 03:00:09,331 - IntimeModelSelector - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ffc2c053-8347-4012-a761-781e05ada4f9]: validation metric 0.8707036323017544 from client ?
2023-08-09 03:00:09,331 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ffc2c053-8347-4012-a761-781e05ada4f9]: invoking result_received_cb ...
2023-08-09 03:00:09,337 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ffc2c053-8347-4012-a761-781e05ada4f9]: Contribution from berzelius6 ACCEPTED by the aggregator.
2023-08-09 03:00:09,337 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius6, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=ffc2c053-8347-4012-a761-781e05ada4f9]: finished processing client result by scatter_and_gather
2023-08-09 03:00:14,179 - FederatedServer - INFO - received update from berzelius_berzelius5_0 (18826885 Bytes, 1691542814 seconds)
2023-08-09 03:00:14,377 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got result from client berzelius5 for task: name=train, id=14b37e16-8ab0-4452-aa03-045fbd53680e
2023-08-09 03:00:14,378 - IntimeModelSelector - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=14b37e16-8ab0-4452-aa03-045fbd53680e]: validation metric 0.8707036323017544 from client ?
2023-08-09 03:00:14,378 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=14b37e16-8ab0-4452-aa03-045fbd53680e]: invoking result_received_cb ...
2023-08-09 03:00:14,387 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=14b37e16-8ab0-4452-aa03-045fbd53680e]: Contribution from berzelius5 ACCEPTED by the aggregator.
2023-08-09 03:00:14,387 - ServerRunner - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather, peer=berzelius5, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer_rc=OK, task_name=train, task_id=14b37e16-8ab0-4452-aa03-045fbd53680e]: finished processing client result by scatter_and_gather
2023-08-09 03:00:19,214 - FederatedServer - INFO - received update from berzelius_berzelius4_0 (18826885 Bytes, 1691542819 seconds)
2023-08-09 03:00:19,391 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: task train exit with status TaskCompletionStatus.OK
2023-08-09 03:00:19,440 - ScatterAndGather - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: Start aggregation.
2023-08-09 03:00:19,446 - DXOAggregator - INFO - [identity=berzelius, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, wf=scatter_and_gather]: aggregating 2 update(s) at round 534

Client

2023-08-09 02:49:40,359 - LearnerExecutor - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: Client trainer got task: train
2023-08-09 02:50:35,730 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: val_metric_global_model (global_model): 0.8710
2023-08-09 02:50:35,731 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: Current/Total Round: 527/600
2023-08-09 02:50:35,731 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: Client identity: berzelius3
2023-08-09 02:50:35,738 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: Local steps per epoch: 34
2023-08-09 02:50:35,739 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: Local epoch berzelius3: 1/1 (lr=0.0001)
2023-08-09 02:51:22,110 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: Local epochs finished. Returning shareable
2023-08-09 02:51:22,111 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: finished processing task
2023-08-09 02:51:22,113 - FederatedClient - INFO - Starting to push execute result.
2023-08-09 02:51:22,134 - Communicator - INFO - Send submitUpdate to berzelius server
2023-08-09 02:51:24,041 - Communicator - INFO - Received comments: berzelius Received from berzelius3 (18826885 Bytes, 1691542283 seconds). SubmitUpdate time: 1.9064884185791016 seconds
2023-08-09 02:51:24,043 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a]: result sent to server for task: name=train, id=70eee0d3-4a23-44b6-9ccc-f6175ba8201a
2023-08-09 02:51:27,843 - Communicator - INFO - Received from berzelius server (18826464 Bytes). getTask time: 1.7927520275115967 seconds
2023-08-09 02:51:27,845 - FederatedClient - INFO - pull_task completed. Task name:train Status:True
2023-08-09 02:51:27,871 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got task assignment: name=train, id=29992ca0-8229-4808-a818-67217f8913ed
2023-08-09 02:51:27,871 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: invoking task executor <class 'nvflare.app_common.executors.learner_executor.LearnerExecutor'>
2023-08-09 02:51:27,872 - LearnerExecutor - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: Client trainer got task: train
2023-08-09 02:52:22,666 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: val_metric_global_model (global_model): 0.8748
2023-08-09 02:52:22,667 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: Current/Total Round: 529/600
2023-08-09 02:52:22,667 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: Client identity: berzelius3
2023-08-09 02:52:22,674 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: Local steps per epoch: 34
2023-08-09 02:52:22,674 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: Local epoch berzelius3: 1/1 (lr=0.0001)
2023-08-09 02:53:09,889 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: Local epochs finished. Returning shareable
2023-08-09 02:53:09,890 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: finished processing task
2023-08-09 02:53:09,892 - FederatedClient - INFO - Starting to push execute result.
2023-08-09 02:53:09,919 - Communicator - INFO - Send submitUpdate to berzelius server
2023-08-09 02:53:11,853 - Communicator - INFO - Received comments: berzelius Received from berzelius3 (18826885 Bytes, 1691542391 seconds). SubmitUpdate time: 1.9341309070587158 seconds
2023-08-09 02:53:11,857 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=29992ca0-8229-4808-a818-67217f8913ed]: result sent to server for task: name=train, id=29992ca0-8229-4808-a818-67217f8913ed
2023-08-09 02:53:16,001 - Communicator - INFO - Received from berzelius server (18826464 Bytes). getTask time: 2.135878086090088 seconds
2023-08-09 02:53:16,004 - FederatedClient - INFO - pull_task completed. Task name:train Status:True
2023-08-09 02:53:16,032 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got task assignment: name=train, id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44
2023-08-09 02:53:16,033 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: invoking task executor <class 'nvflare.app_common.executors.learner_executor.LearnerExecutor'>
2023-08-09 02:53:16,033 - LearnerExecutor - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: Client trainer got task: train
2023-08-09 02:54:10,894 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: val_metric_global_model (global_model): 0.8694
2023-08-09 02:54:10,895 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: Current/Total Round: 530/600
2023-08-09 02:54:10,895 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: Client identity: berzelius3
2023-08-09 02:54:10,902 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: Local steps per epoch: 34
2023-08-09 02:54:10,903 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: Local epoch berzelius3: 1/1 (lr=0.0001)
2023-08-09 02:54:57,572 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: Local epochs finished. Returning shareable
2023-08-09 02:54:57,572 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: finished processing task
2023-08-09 02:54:57,574 - FederatedClient - INFO - Starting to push execute result.
2023-08-09 02:54:57,602 - Communicator - INFO - Send submitUpdate to berzelius server
2023-08-09 02:54:59,484 - Communicator - INFO - Received comments: berzelius Received from berzelius3 (18826885 Bytes, 1691542499 seconds). SubmitUpdate time: 1.8826227188110352 seconds
2023-08-09 02:54:59,494 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44]: result sent to server for task: name=train, id=b7567ca0-62e0-4937-8d5e-1ab0af8e4f44
2023-08-09 02:55:03,433 - Communicator - INFO - Received from berzelius server (18826464 Bytes). getTask time: 1.9316296577453613 seconds
2023-08-09 02:55:03,435 - FederatedClient - INFO - pull_task completed. Task name:train Status:True
2023-08-09 02:55:03,462 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got task assignment: name=train, id=12470e54-332b-451d-b868-48113b56eeb0
2023-08-09 02:55:03,463 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: invoking task executor <class 'nvflare.app_common.executors.learner_executor.LearnerExecutor'>
2023-08-09 02:55:03,463 - LearnerExecutor - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: Client trainer got task: train
2023-08-09 02:55:58,076 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: val_metric_global_model (global_model): 0.8711
2023-08-09 02:55:58,077 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: Current/Total Round: 531/600
2023-08-09 02:55:58,077 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: Client identity: berzelius3
2023-08-09 02:55:58,084 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: Local steps per epoch: 34
2023-08-09 02:55:58,085 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: Local epoch berzelius3: 1/1 (lr=0.0001)
2023-08-09 02:56:44,890 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: Local epochs finished. Returning shareable
2023-08-09 02:56:44,891 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: finished processing task
2023-08-09 02:56:44,893 - FederatedClient - INFO - Starting to push execute result.
2023-08-09 02:56:44,919 - Communicator - INFO - Send submitUpdate to berzelius server
2023-08-09 02:56:46,822 - Communicator - INFO - Received comments: berzelius Received from berzelius3 (18826885 Bytes, 1691542606 seconds). SubmitUpdate time: 1.9028308391571045 seconds
2023-08-09 02:56:46,828 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=12470e54-332b-451d-b868-48113b56eeb0]: result sent to server for task: name=train, id=12470e54-332b-451d-b868-48113b56eeb0
2023-08-09 02:56:50,625 - Communicator - INFO - Received from berzelius server (18826464 Bytes). getTask time: 1.7910733222961426 seconds
2023-08-09 02:56:50,632 - FederatedClient - INFO - pull_task completed. Task name:train Status:True
2023-08-09 02:56:50,657 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got task assignment: name=train, id=17bf81bf-4b01-4c50-a170-0f78f67751b7
2023-08-09 02:56:50,658 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: invoking task executor <class 'nvflare.app_common.executors.learner_executor.LearnerExecutor'>
2023-08-09 02:56:50,658 - LearnerExecutor - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: Client trainer got task: train
2023-08-09 02:57:45,759 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: val_metric_global_model (global_model): 0.8722
2023-08-09 02:57:45,760 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: Current/Total Round: 533/600
2023-08-09 02:57:45,760 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: Client identity: berzelius3
2023-08-09 02:57:45,767 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: Local steps per epoch: 34
2023-08-09 02:57:45,768 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: Local epoch berzelius3: 1/1 (lr=0.0001)
2023-08-09 02:58:32,528 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: Local epochs finished. Returning shareable
2023-08-09 02:58:32,529 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: finished processing task
2023-08-09 02:58:32,530 - FederatedClient - INFO - Starting to push execute result.
2023-08-09 02:58:32,553 - Communicator - INFO - Send submitUpdate to berzelius server
2023-08-09 02:58:34,362 - Communicator - INFO - Received comments: berzelius Received from berzelius3 (18826885 Bytes, 1691542714 seconds). SubmitUpdate time: 1.8091869354248047 seconds
2023-08-09 02:58:34,364 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=17bf81bf-4b01-4c50-a170-0f78f67751b7]: result sent to server for task: name=train, id=17bf81bf-4b01-4c50-a170-0f78f67751b7
2023-08-09 02:58:38,148 - Communicator - INFO - Received from berzelius server (18826464 Bytes). getTask time: 1.7735710144042969 seconds
2023-08-09 02:58:38,154 - FederatedClient - INFO - pull_task completed. Task name:train Status:True
2023-08-09 02:58:38,181 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7]: got task assignment: name=train, id=e3bbdd15-1438-42f8-bc84-be4508d4b053
2023-08-09 02:58:38,181 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: invoking task executor <class 'nvflare.app_common.executors.learner_executor.LearnerExecutor'>
2023-08-09 02:58:38,181 - LearnerExecutor - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: Client trainer got task: train
2023-08-09 02:59:32,956 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: val_metric_global_model (global_model): 0.8704
2023-08-09 02:59:32,957 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: Current/Total Round: 534/600
2023-08-09 02:59:32,957 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: Client identity: berzelius3
2023-08-09 02:59:32,964 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: Local steps per epoch: 34
2023-08-09 02:59:32,965 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: Local epoch berzelius3: 1/1 (lr=0.0001)
2023-08-09 03:00:19,414 - SupervisedMonaiBratsLearner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: Local epochs finished. Returning shareable
2023-08-09 03:00:19,415 - ClientRunner - INFO - [identity=berzelius3, run=f83759c6-ca8a-4c47-8b65-b703e06610c7, peer=berzelius, peer_run=f83759c6-ca8a-4c47-8b65-b703e06610c7, task_name=train, task_id=e3bbdd15-1438-42f8-bc84-be4508d4b053]: finished processing task

[wanderine]

The problem was that "min_clients" was set to 1, so it continued as soon as 1 client had finished.