Skip to content

Latest commit

 

History

History

models

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
celeba
celeba
 
 
femnist
femnist
 
 
metrics
metrics
 
 
reddit
reddit
 
 
sent140
sent140
 
 
shakespeare
shakespeare
 
 
synthetic
synthetic
 
 
utils
utils
 
 
README.md
README.md
 
 
baseline_constants.py
baseline_constants.py
 
 
client.py
client.py
 
 
main.py
main.py
 
 
model.py
model.py
 
 
server.py
server.py
 
 

README.md

Server-Client Simulations

FEMNIST Classifier Instructions

  • Ensure that the data/femnist/data/train and data/femnist/data/test directories contain data
  • Run python3 main.py -dataset femnist -model cnn
  • For more simulation options and details, see 'Additional Notes' section

Sentiment140 Classifier Instructions

  • Ensure that the data/sent140/data/train and data/sent140/data/test directories contain data
  • If you already have a .txt file containing word embeddings (i.e. 'glove.6B.300d.txt') in the sent140 directory, run sent140/get_embs.py -f fp, where fp is the file path to the .txt file, to generate a .json file for the classifier to load. Otherwise, run ./sent140/get_embs.sh, which downloads the embeddings and creates the .json file.
  • Run python3 main.py -dataset sent140 -model stacked_lstm
  • For more simulation options and details, see 'Additional Notes' section

Shakespeare Classifier Instructions

  • Ensure that the data/shakespeare/data/train and data/shakespeare/data/test directories contain data
  • Run python3 main.py -dataset shakespeare -model stacked_lstm
  • For more simulation options and details, see 'Additional Notes' section

Additional Notes

  • In order to run these reference implementations, the -t sample tag must have been used when running the ./preprocess.sh script for the respective dataset
  • The total number of clients simulated equals the total number of users in the respective dataset's training data
  • For optimal model performance, generate data using arguments similar to those listed in the 'large-sized dataset' option in the respective dataset README file.
  • main.py supports these additional tags:
    • --model: name of model; options are listed the respective dataset folder, for example cnn for femnist; defaults to first model in the respective dataset folder
    • --num_rounds: number of rounds to simulate
    • --eval_every: evaluate every ___ rounds
    • --clients_per_round: number of clients trained per round
    • --batch_size: batch size when clients train on data
    • --num_epochs: number of epochs when clients train on data
    • -t: simulation time: small, medium, or large; greater time corresponds to higher accuracy; for large runs, generate data using arguments similar to those listed in the 'large-sized dataset' option in the respective dataset README file for optimal model performance; default: large
    • -lr: learning rate for local optimizers.
  • After running a classifier, open metrics.ipynb to view systems and statistical metrics from the last run.
  • Metrics generated by models are stored in metrics.json, which contains the following 'key: value' pairs:
    • dataset: name of the dataset
    • num_rounds: number of rounds simulated
    • eval_every: integer x such that model evaluated every x rounds
    • accuracies: length-num_rounds list of (ids, groups, num_samples, accs), where
      • ids is a list of client id strings corresponding to the clients tested in the respective round,
      • groups is a list containing the group names of each of those clients (list of Nones if clients don't have an associated group),
      • num_samples is a list containing the number of test samples for each of those clients, and
      • accs is a list containing accuracy from each client's evaluation
    • client_computations: dictionary with client ids as keys and length-num_rounds lists as values; the elements in the lists are ints representing the number of FLOPs computed by the respective client in the corresponding round
    • bytes_written: dictionary with client ids as keys and length-num_rounds lists as values; the elements in the lists are ints representing the number of bytes written to the server by the resepctive client in the corresponding round
    • bytes_read: dictionary with client ids as keys and length-num_rounds lists as values; the elements in the lists are ints representing the number of bytes read from the server by the resepctive client in the corresponding round