Script for Plotting the Prostate Validation Curves

[atnassar]

Hi,
Thank you for sharing the NVFlare repository! The examples are really helpful to understand the FL implementation.

I am currently working on the prostate example. I ran all the experiment in a POC workspace and want to generate the validation curves you show in the results.

I tried modifying the plotting script "plot_tensorboard_events.py" which you provided in the cifar10 example and tailor it to plot the prostate results (the 4 figures shown for the 4 clients: I2CVB, MSD, NCI_ISBI_3T, & NCI_ISBI_Dx for the configurations: central, fedavg, fedprox, & ditto) but unfortunately I could not implement that correctly.

Could you please provide the script you used to plot the results in the prostate example?

Thank you again!

[holgerroth]

@ZiyueXu77 do we have a similar script for prostate as for the cifar10 example?

[atnassar]

@ZiyueXu77 @holgerroth Could you support on this please?

[ZiyueXu77]

Hi @atnassar, thanks for the question! I do have a similar script for prostate, but for now it relies on my other two customized scripts moving/reorganizing the tensorboard records from our computing cluster to local folder. You will need some modifications to assign the correct path to the folders.
Please see below, the code works for folder structure of $root/$client_ID/$algorithm_ID, ($client_ID e.g. MSD, $algorithm_ID e.g. fedavg):

# Copyright (c) 2021, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import json
import os
import glob

import pandas as pd
import tensorflow as tf
import matplotlib.pyplot as plt
import seaborn as sns

client_results_root = "./Record/Prostate/"

# Central vs. FedAvg vs. FedProx vs. Ditto
experiments = {"central": {"run": "central", "tag": "val_acc_global_model"},
               "fedavg": {"run": "fedavg", "tag": "val_acc_global_model"},
               "fedprox": {"run": "fedprox", "tag": "val_acc_global_model"},
               "ditto": {"run": "ditto", "tag": "val_acc_global_model"}
               }
# 4 sites
sites = ["I2CVB", "MSD", "NCI_ISBI_3T", "NCI_ISBI_Dx"]

weight = 0.8

def smooth(scalars, weight):  # Weight between 0 and 1
    last = scalars[0]  # First value in the plot (first timestep)
    smoothed = list()
    for point in scalars:
        smoothed_val = last * weight + (1 - weight) * point  # Calculate smoothed value
        smoothed.append(smoothed_val)                        # Save it
        last = smoothed_val                                  # Anchor the last smoothed value

    return smoothed

def read_eventfile(filepath, tags=["val_acc_global_model"]):
    data = {}
    for summary in tf.compat.v1.train.summary_iterator(filepath):
        for v in summary.summary.value:
            if v.tag in tags:
                # print(v.tag, summary.step, v.simple_value)
                if v.tag in data.keys():
                    data[v.tag].append([summary.step, v.simple_value])
                else:
                    data[v.tag] = [[summary.step, v.simple_value]]
    return data


def add_eventdata(data, config, filepath, tag="val_acc_global_model"):
    event_data = read_eventfile(filepath, tags=[tag])
    assert len(event_data[tag]) > 0, f"No data for key {tag}"

    acc = []
    for e in event_data[tag]:
        # print(e)
        data["Config"].append(config)
        data["Step"].append(e[0])
        acc.append(e[1])

    acc = smooth(acc, weight)
    for entry in acc:
        data["Accuracy"].append(entry)

    print(f"added {len(event_data[tag])} entries for {tag}")


def main():
    fig = plt.figure()
    num_site = len(sites)
    i=1
    # add event files
    for site in sites:
        data = {
            "Config": [],
            "Step": [],
            "Accuracy": []
        }
        for config, exp in experiments.items():
            eventfile = glob.glob(os.path.join(client_results_root, site, exp["run"] + "/events.*"), recursive=True)
            assert len(eventfile) == 1, "No unique event file found!"
            eventfile = eventfile[0]
            print("adding", eventfile)
            add_eventdata(data, config, eventfile, tag=exp["tag"])
        ax = plt.subplot(2,int(num_site/2),i)
        ax.set_title(site)
        sns.lineplot(x="Step", y="Accuracy", hue="Config", data=data)
        i = i + 1
    plt.subplots_adjust(hspace=0.3)
    #fig.tight_layout()
    plt.show()


if __name__ == "__main__":
    main()

[atnassar]

Thank you @ZiyueXu77 !

The folder structure following the steps in the prostate example is as follows:

The root: ./examples/prostate/workspace_prostate/

In the "wokspace_prostate" folder I have the following 7 folders: server, admin, client_I2CVB, client_MSD, client_NCI_ISBI_Dx, client_NCI_ISBI_3T, client_All.

The 4 run folders: run_1, run_2, run_3, run_4 that include the events files for the 4 experiments: Central, Fedavg, Fedprox, Ditto, respectively are showing in the server and clients folders as follows:

server: run_1, run_2, run_3, run_4
client_All: run_1
client_I2CVB: run_1, run_2, run_3, run_4
client_MSD: run_2, run_3, run_4
client_NCI_ISBI_Dx: run_2, run_3, run_4
client_NCI_ISBI_3T: run_2, run_3, run_4

Could you just advise what would be the required modifications in this context?

[ZiyueXu77]

try making the follow changes (didn't test it myself, could have some remaining issues, print the path out and see if it aligns with your folder structure)
this:
eventfile = glob.glob(os.path.join(client_results_root, site, exp["run"] + "/events.*"), recursive=True)
to
eventfile = glob.glob(os.path.join(client_results_root, "client_"+site, exp["run"] + "/events.*"), recursive=True)
and this

experiments = {"central": {"run": "central", "tag": "val_acc_global_model"},
               "fedavg": {"run": "fedavg", "tag": "val_acc_global_model"},
               "fedprox": {"run": "fedprox", "tag": "val_acc_global_model"},
               "ditto": {"run": "ditto", "tag": "val_acc_global_model"}
               }

to

experiments = {"central": {"run": "run_1", "tag": "val_acc_global_model"},
               "fedavg": {"run": "run_2", "tag": "val_acc_global_model"},
               "fedprox": {"run": "run_3", "tag": "val_acc_global_model"},
               "ditto": {"run": "run_4", "tag": "val_acc_global_model"}
               }

[atnassar]

@ZiyueXu77 I got the following error:
Traceback (most recent call last):
File "figs/plot_tensorboard_events_prostate1.py", line 116, in
main()
File "figs/plot_tensorboard_events_prostate1.py", line 102, in main
assert len(eventfile) == 1, "No unique event file found!"
AssertionError: No unique event file found!

[ZiyueXu77]

could you check where the "events.*" files are located, and also check how many tb record? this script works for 1 tb record only

[atnassar]

@ZiyueXu77 I am not sure what a tb is and how to check the number of tb records I have?

The events files are located in the folders "app_client_" under the "run_".
The 4 run folders: run_1, run_2, run_3, run_4 that include the events files for the 4 experiments: Central, Fedavg, Fedprox, Ditto, respectively are showing in the server and clients folders as follows:

server: run_1, run_2, run_3, run_4
client_All: run_1
client_I2CVB: run_1, run_2, run_3, run_4
client_MSD: run_2, run_3, run_4
client_NCI_ISBI_Dx: run_2, run_3, run_4
client_NCI_ISBI_3T: run_2, run_3, run_4

For e.g., the events files for the central experiment are shown under the following routs:
/examples/prostate/workspace_prostate/client_All/run_1/app_client_All/events.out.tfevents.*
/examples/prostate/workspace_prostate/client_I2CVB/run_1/app_client_I2CVB/events.out.tfevents.*

similarly for the other experiments:
the events files for the Fedavg experiment (run_2) are shown in:
/examples/prostate/workspace_prostate/client_I2CVB/run_2/app_client_I2CVB/events.out.tfevents.*
/examples/prostate/workspace_prostate/client_MSD/run_2/app_client_MSD/events.out.tfevents.*
/examples/prostate/workspace_prostate/client_NCI_ISBI_Dx/run_2/app_client_NCI_ISBI_Dx/events.out.tfevents.*
/examples/prostate/workspace_prostate/client_NCI_ISBI_3T/run_2/app_client_NCI_ISBI_3T/events.out.tfevents.*

[ZiyueXu77]

@atnassar I see, then try this
eventfile = glob.glob(os.path.join(client_results_root, "client_"+site, exp["run"], "app_client_"+site, "/events.*"), recursive=True)

[atnassar]

@ZiyueXu77 I am getting the same error:

Traceback (most recent call last):
File "figs/plot_tensorboard_events_prostate1.py", line 116, in
main()
File "figs/plot_tensorboard_events_prostate1.py", line 102, in main
assert len(eventfile) == 1, "No unique event file found!"
AssertionError: No unique event file found!

I believe the part where to append results from multiple events files to a json file similar to the cifar10 plotting script is missing here. Here in the prostate example there are 4 figures, each figure shows 4 curves (central, fedavg, fedprox, ditto) for a client. So, there should be a way to add more than one events file to get that figures.

[ZiyueXu77]

could you add `print(eventfile)` below the line? One run should have only 1 tensorboard event file.

…

On Tue, Mar 22, 2022 at 11:59 AM atnassar ***@***.***> wrote: @ZiyueXu77 <https://github.com/ZiyueXu77> I am getting the same error: Traceback (most recent call last): File "figs/plot_tensorboard_events_prostate1.py", line 116, in main() File "figs/plot_tensorboard_events_prostate1.py", line 102, in main assert len(eventfile) == 1, "No unique event file found!" AssertionError: No unique event file found! I believe the part where to append results from multiple events files to a json file similar to the cifar10 plotting script is missing here. Here in the prostate example there are 4 figures, each figure shows 4 curves (central, fedavg, fedprox, ditto) for a client. So, there should be a way to add more than one events file to get that figures. — Reply to this email directly, view it on GitHub <#319 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARDWATZRM62LWUU43CAB43LVBHU57ANCNFSM5Q73BR3Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[atnassar]

Thank you @ZiyueXu77 , I really appreciate your support! I resolved the issue and it is working now.

I have another question, that need your kind support, about loading the trained local models at the clients and the global model to test them on new testing dataset, that need your usual support.

[ZiyueXu77]

sounds good! the trained models are saved in the run_* folders, they are pytorch models (unet from monai), and please refer to https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/torch/unet_inference_dict.py for using them on testing dataset

[holgerroth]

@atnassar, @ZiyueXu77 I closed this issue #319 and moved to discussions.