Skip to content

Commit

Permalink
add new leaderboards
Browse files Browse the repository at this point in the history
  • Loading branch information
@TsingZ0
TsingZ0 committed Dec 17, 2024
1 parent f529ade commit cc5e7a3
Showing 1 changed file with 244 additions and 0 deletions.
244 changes: 244 additions & 0 deletions docs/benchmark.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ <h1><a href="index.html">PFLlib</a></h1>
<h2>Benchmark & Evaluation Platform</h2>
<p>To integrate all the algorithms, datasets, and scenarios, we standardize the experimental settings and create a <strong>unified evaluation platform</strong> for a fair comparison of these algorithms. Here, we present the benchmark results of <strong>20 algorithms</strong> across two widely-used <em><strong>label skew</strong></em> scenarios. This is just one example. You can obtain different results by resetting all the configurations in <code>main.py</code> in our PFLlib.</p>
<h3>Leaderboard</h3>
<p>Our methods—<a href="https://github.com/TsingZ0/FedCP"><strong>FedCP</strong></a>, <a href="https://github.com/TsingZ0/GPFL"><strong>GPFL</strong></a>, and <a href="https://github.com/TsingZ0/DBE"><strong>FedDBE</strong></a>—lead the way. Notably, <strong>FedDBE</strong> stands out with robust performance across varying data heterogeneity levels.</p>
<h4 style="width:100%; text-align:center;">The test accuracy (%) on the CV and NLP tasks in <em>label skew</em> settings.</h4>
<table border="1" cellpadding="5" cellspacing="0">
<thead>
Expand Down Expand Up @@ -427,6 +428,249 @@ <h4 style="width:100%; text-align:center;">The test accuracy (%) on the CV and N
</tbody>
</table>

<h4 style="width:100%; text-align:center;">The test accuracy (%) under different heterogeneous degrees in the practical <em>label skew</em> setting using the 4-layer CNN.</h4>
<table border="1" cellpadding="5" cellspacing="0">
<thead>
<tr>
<th rowspan="2" colspan="1"><b>Datasets</b></th>
<th colspan="4"><b>Cifar100</b></th>
<th colspan="4"><b>Tiny-ImageNet</b></th>
</tr>
<tr>
<th>β=0.01</th>
<th>β=0.1</th>
<th>β=0.5</th>
<th>β=5</th>
<th>β=0.01</th>
<th>β=0.1</th>
<th>β=0.5</th>
<th>β=5</th>
</tr>
</thead>
<tbody>
<tr>
<td><b>FedAvg</b></td>
<td>23.58±0.43</td>
<td>31.89±0.47</td>
<td>27.99±0.32</td>
<td>35.51±0.33</td>
<td>15.70±0.46</td>
<td>19.46±0.20</td>
<td>21.14±0.47</td>
<td>21.71±0.27</td>
</tr>
<tr>
<td><b>FedProx</b></td>
<td>23.74±0.25</td>
<td>31.99±0.41</td>
<td>35.05±0.20</td>
<td>35.31±0.43</td>
<td>15.66±0.36</td>
<td>19.37±0.22</td>
<td>21.22±0.47</td>
<td>21.69±0.25</td>
</tr>
<tr>
<td><b>FedGen</b></td>
<td>20.89±0.42</td>
<td>30.96±0.54</td>
<td>33.88±0.83</td>
<td>35.64±0.36</td>
<td>15.87±0.23</td>
<td>19.19±0.18</td>
<td>21.06±0.08</td>
<td>21.44±0.51</td>
</tr>
<tr>
<td><b>Per-FedAvg</b></td>
<td>49.25±0.54</td>
<td>44.28±0.33</td>
<td>35.32±0.12</td>
<td>24.94±0.64</td>
<td>39.39±0.30</td>
<td>25.07±0.07</td>
<td>16.36±0.13</td>
<td>12.08±0.27</td>
</tr>
<tr>
<td><b>pFedMe</b></td>
<td>65.28±0.83</td>
<td>47.34±0.46</td>
<td>32.52±1.57</td>
<td>14.70±0.92</td>
<td>41.45±0.14</td>
<td>26.93±0.19</td>
<td>17.48±0.61</td>
<td>4.03±0.50</td>
</tr>
<tr>
<td><b>Ditto</b></td>
<td>73.29±0.49</td>
<td>52.87±0.64</td>
<td>26.28±0.17</td>
<td>35.72±0.21</td>
<td>50.62±0.02</td>
<td>32.15±0.04</td>
<td>18.98±0.05</td>
<td>21.79±0.62</td>
</tr>
<tr>
<td><b>APFL</b></td>
<td>72.63±0.15</td>
<td>46.74±0.60</td>
<td>25.69±0.21</td>
<td>20.76±9.03</td>
<td>49.96±0.04</td>
<td>34.87±0.43</td>
<td>23.31±0.18</td>
<td>16.12±0.10</td>
</tr>
<tr>
<td><b>FedFomo</b></td>
<td>71.11±0.08</td>
<td>45.39±0.45</td>
<td>24.35±0.55</td>
<td>29.77±0.56</td>
<td>46.36±0.54</td>
<td>26.33±0.22</td>
<td>11.59±0.11</td>
<td>14.86±0.55</td>
</tr>
<tr>
<td><b>FedAMP</b></td>
<td>72.78±0.17</td>
<td>47.69±0.49</td>
<td>25.94±0.12</td>
<td>13.71±0.12</td>
<td>48.42±0.06</td>
<td>27.99±0.11</td>
<td>12.48±0.21</td>
<td>5.41±0.14</td>
</tr>
<tr>
<td><b>APPLE</b></td>
<td>71.11±0.10</td>
<td>53.22±0.20</td>
<td>41.81±0.23</td>
<td>32.68±0.28</td>
<td>48.04±0.10</td>
<td>35.04±0.47</td>
<td>24.28±0.21</td>
<td>17.79±0.47</td>
</tr>
<tr>
<td><b>FedALA</b></td>
<td>73.82±0.35</td>
<td>55.92±0.03</td>
<td>44.17±0.51</td>
<td>34.27±0.70</td>
<td>55.75±0.02</td>
<td>40.54±0.02</td>
<td>29.04±0.13</td>
<td>22.12±0.22</td>
</tr>
<tr>
<td><b>FedPer</b></td>
<td>71.09±0.34</td>
<td>49.63±0.54</td>
<td>29.17±0.21</td>
<td>16.09±0.19</td>
<td>51.83±0.22</td>
<td>33.84±0.34</td>
<td>17.31±0.19</td>
<td>9.61±0.06</td>
</tr>
<tr>
<td><b>FedRep</b></td>
<td>74.91±0.16</td>
<td>52.39±0.35</td>
<td>29.74±0.21</td>
<td>14.93±0.12</td>
<td>55.43±0.15</td>
<td>37.27±0.20</td>
<td>16.74±0.09</td>
<td>8.04±0.05</td>
</tr>
<tr>
<td><b>FedRoD</b></td>
<td>67.78±0.55</td>
<td>50.94±0.11</td>
<td>36.29±0.07</td>
<td>25.63±0.74</td>
<td>49.17±0.06</td>
<td>36.43±0.05</td>
<td>23.23±0.11</td>
<td>16.71±0.24</td>
</tr>
<tr>
<td><b>FedBABU</b></td>
<td>73.30±0.32</td>
<td>55.02±0.33</td>
<td>39.35±0.45</td>
<td>27.61±0.51</td>
<td>53.97±0.25</td>
<td>36.82±0.45</td>
<td>23.08±0.20</td>
<td>15.42±0.30</td>
</tr>
<tr>
<td><b>FedCP</b></td>
<td>77.76±0.03</td>
<td>59.56±0.08</td>
<td>41.76±0.15</td>
<td>26.83±0.03</td>
<td>56.31±0.39</td>
<td><b>43.49±0.04</b></td>
<td>28.57±0.07</td>
<td>19.12±0.13</td>
</tr>
<tr>
<td><b>GPFL</b></td>
<td>76.27±0.12</td>
<td>61.86±0.31</td>
<td>43.73±0.42</td>
<td>30.86±0.28</td>
<td><b>57.05±0.41</b></td>
<td>43.37±0.53</td>
<td>26.85±0.14</td>
<td>16.34±0.39</td>
</tr>
<tr>
<td><b>FedDBE</b></td>
<td><b>78.39±0.08</b></td>
<td><b>64.39±0.27</b></td>
<td><b>52.58±0.17</b></td>
<td><b>41.12±0.30</b></td>
<td>54.61±0.09</td>
<td>43.32±0.37</td>
<td><b>33.71±0.15</b></td>
<td><b>26.76±0.11</b></td>
</tr>
<tr>
<td><b>FedDistill</b></td>
<td>74.63±0.14</td>
<td>49.93±0.23</td>
<td>29.32±0.46</td>
<td>15.45±0.44</td>
<td>50.49±0.07</td>
<td>30.02±0.09</td>
<td>14.34±0.06</td>
<td>6.56±0.06</td>
</tr>
<tr>
<td><b>FedProto</b></td>
<td>77.19±0.15</td>
<td>52.70±0.33</td>
<td>32.57±0.23</td>
<td>14.20±3.92</td>
<td>50.65±0.32</td>
<td>31.21±0.16</td>
<td>16.69±0.02</td>
<td>8.92±0.19</td>
</tr>
</tbody>
</table>

<h4 style="width:100%; text-align:center;">The time cost (in minutes) on Tiny-ImageNet using ResNet-18 in the practical <em>label skew</em> setting.</h4>
<table border="1" cellpadding="5" cellspacing="0">
<thead>
Expand Down

0 comments on commit cc5e7a3

Please sign in to comment.