Benchmarking Results for MPI-based federated learning
Please visit the following link to check the latest benchmark experimental results: https://app.wandb.ai/automl/fedml/reports/FedML-Benchmark-Experimental-Results--VmlldzoxODE2NTU FedML white paper (https://arxiv.org/pdf/2007.13518.pdf) also summarizes the dataset list and related benchmarks. We refer the hyper-parameters and reproduce results from many top-tier ML conferences. Please check details of our reference hyperparameters as follows.
Linear Models
| Data | Model | Alg | Partition | #C | #C_p | bs | c_opt | lr | e | #R | acc |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MNIST | LR | FedAvg | Power Law | 1000 | 10 | 10 | SGD | 0.03 | 1 | >100 | >75 |
| Federated EMNIST | LR | FedAvg | Power Law | 200 | 10 | 10 | SGD | 0.003 | 1 | >200 | 10~40 |
| Synthetic(α,β） | LR | FedAvg | Power Law | 30 | 10 | 10 | SGD | 0.01 | 1 | >200 | >60 |
Note: #C stands for client_num_in_total; #C_p stands for client_num_per_round; bs = batch_size; c_opt = client optimizer; e = epoch; #R = number of rounds; acc = accuracy. For Synthetic(α,β), (α,β) is chosen from (0,0), (0.5,0.5), (1,1)
- MNIST – Logistic Regression – FedAvg
- Patition Method: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’
- client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’
- client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation
- lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- epochs: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 9 description
- comm_round: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10
- accuracy: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10
- Federated EMNIST – Logistic Regression-FedAvg
- Patition Method: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’
- client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’
- client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation
- lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- epochs: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 9 description
- comm_round: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10
- accuracy: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10
- Synthetic(α,β) – Logistic Regression -FedAvg
- Patition Method: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.1, ‘Synthetic’
- client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.1, ‘Synthetic’
- client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation
- lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
- epochs: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Hyperparameters & evaluation metrics’
- comm_round: ‘Federated optimization in heterogeneous networks’, page 19, Appendix C.3.2 Figure 6
- accuracy: ‘Federated optimization in heterogeneous networks’, page 19, Appendix C.3.2 Figure 6
Lightweight and shallow neural network models
| Task | Data Set | Model | Alogrithm | Partition Method | Partition Alpha | client_num_in_total | client_num_per_round | batch_size | client_optimizer | lr | wd | epochs | comm_round | accuracy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CV | Federated EMNIST | CNN (2 Conv + 2 FC) | FedAvg | Power Law | 3400 | 10 | 20 | SGD | 0.1 | - | 1 | >1500 | 84.9 | |
| CV | CIFAR-100 | ResNet-18+group normalization | FedAvg | Pachinko Allocation | 100/500(ex/cli) | 500 | 10 | 20 | SGD | 0.1 | - | 1 | >4000 | 44.7 |
| NLP | Shakespeare | RNN (2 LSTM + 1 FC) | FedAvg | realistic patition | 715 | 10 | 4 | SGD | 1 | - | 1 | >1200 | 56.9 | |
| NLP | StackOverflow | RNN (1 LSTM + 2 FC) | FedAvg | Pachinko Allocation | 342477 | 50 | 16 | SGD | pow(10,-0.5) | - | 1 | >1500 | 19.5 |
- Federated EMNIST-CNN-FedAvg (https://openreview.net/pdf?id=LkFG3lB13U5)
- Patition Method: ‘Adaptive federated optimization’ (https://openreview.net/pdf?id=LkFG3lB13U5), page 23, Appendix C.2
- client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2
- client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
- batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7
- client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1
- lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8
- wd (learning rate decay): ‘Adaptive federated optimization’, page34, Appendix E.6, Paragraph 2
- epochs: ‘Adaptive federated optimization’, page34, Appendix E.6, Paragraph 1
- comm_round:‘Adaptive federated optimization’, page28, Appendix E.1, figure 3
- accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1
- CIFAR-100 – ResNet18 -FedAvg
- Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.1, Paragraph 3
- Patition_alpha: ‘Adaptive federated optimization’, page 23, Appendix C.1, Paragraph 2
- client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2
- client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
- batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7
- client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1
- lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8
- epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
- comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1
- accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1
- Shakespeare – RNN – FedAvg
- Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.3
- client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2
- client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
- batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7
- client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1
- lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8
- epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
- comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1
- accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1
- StackOverflow – RNN – FedAvg
- Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.4, Paragraph 2
- client_num_in_total: ‘Adaptive federated optimization’, page 25, Appendix C.4, Paragraph 1
- client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
- batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7
- client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1
- lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8
- epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
- comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1
- accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1
Benchmarking using modern DNNs
| Data | Model | Alg | # C | # C_p | bs | c_opt | lr | wd | e | round | IID acc | non-IID acc |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CIFAR10 | ResNet-56 | FedAvg | 10 | 10 | 64 | SGD | 0.001 | 0.001 | 20 | 100 | 93.19 | 87.12 |
| CIFAR100 | ResNet-56 | FedAvg | 10 | 10 | 64 | SGD | 0.001 | 0.001 | 20 | 100 | 68.91 | 64.70 |
| CINIC10 | ResNet-56 | FedAvg | 10 | 10 | 64 | SGD | 0.001 | 0.001 | 20 | 100 | 82.57 | 73.49 |
| CIFAR10 | MobileNet | FedAvg | 10 | 10 | 64 | SGD | 0.001 | 0.001 | 20 | 100 | 91.12 | 86.32 |
| CIFAR100 | MobileNet | FedAvg | 10 | 10 | 64 | SGD | 0.001 | 0.001 | 20 | 100 | 55.12 | 53.54 |
| CINIC10 | MobileNet | FedAvg | 10 | 10 | 64 | SGD | 0.001 | 0.001 | 20 | 100 | 79.95 | 71.23 |
Note: Non-IID distribution is set using LDA ( LDA = Latent Dirichlet Allocation) with alpha = 0.5; #C stands for client_num_in_total; #C_p stands for client_num_per_round; bs = batch size; c_opt = client optimizer.