Tensorflow examples of 3 common model aggregation methods in federated learning (FL)

Federated Learning (FL) is an excellent approach to ML that enables multiple devices (such as Internet of Things (IoT) devices) or computers to collaborate when model training is complete without sharing their data.

"Clients" are computers and devices used in FL that may be completely separate from each other and have their own distinct data, which may apply different privacy policies, be owned by different organizations, and cannot be accessed by each other.

Using FL, models can learn from a wider range of data sources without data. The areas where FL is widely used are as follows:

• health care
• Internet of Things (IoT)
• Mobile devices

Since data privacy is a big issue for many applications (such as medical data), FL is mainly used to protect the privacy of customers without sharing their data with any other customers or parties. Clients of FL share their model updates with a central server to aggregate an updated global model. The global model is sent back to the client, which can use it to make predictions or take other actions on the local data.

Key Concepts of FL

Data Privacy: Suitable for applications with sensitive or private data.

Data distribution: Training is distributed across a large number of devices or servers; the model should be able to generalize to new data.

Model aggregation: Models updated across different clients are aggregated to generate a single global model. The model aggregation method is as follows:

• Simple average: average over all clients
• Weighted averaging: Before averaging each model, it is weighted according to the quality of the model, or the amount of its training data.
• Federated Averaging: This is useful in reducing communication overhead and helps improve convergence of global models that account for model updates and differences in local data used.
• Hybrid method: Combining the above multiple model aggregation techniques.

Communication overhead: The transmission of model updates between the client and server needs to consider the communication protocol and the frequency of model updates.

Convergence: A key factor in FL is that the model converges to a good solution with respect to the distributed nature of the data.

Simple steps to achieve FL

1. Define the model architecture
2. Divide the data into client datasets
3. Train the model on the client dataset
4. update the global model
5. Repeat the above learning process

Tensorflow code example

First, let's create a simple server:

 importtensorflowastf
 
 # Set up a server and some client devices
 server=tf.keras.server.Server()
 devices= [tf.keras.server.ClientDevice(worker_id=i) foriinrange(4)]
 
 # Define a simple model and compile it
 inputs=tf.keras.Input(shape=(10,))
 outputs=tf.keras.layers.Dense(2, activation='softmax')(inputs)
 model=tf.keras.Model(inputs=inputs, outputs=outputs)
 model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
 
 # Define a federated dataset and iterate over it
 federated_dataset=tf.keras.experimental.get_federated_dataset(devices, model, x=X, y=y)
 forx, yinfederated_dataset:
     # Train the model on the client data
     model.fit(x, y)

Then we implement the model aggregation step:

1. Simple average

 # Average the updated model weights
 model_weights=model.get_weights()
 fordeviceindevices:
     device_weights=device.get_weights()
     fori, (model_weight, device_weight) inenumerate(zip(model_weights, device_weights)):
         model_weights[i] = (model_weight+device_weight) /len(devices)
 
 # Update the model with the averaged weights
 model.set_weights(model_weights)

2. Weighted average

 # Average the updated model weights using weights based on the quality of the model or the amount of data used to train it
     model_weights=model.get_weights()
     total_weight=0
     fordeviceindevices:
         device_weights=device.get_weights()
         weight=compute_weight(device)  # Replace this with a function that returns the weight for the device
         total_weight+=weight
         fori, (model_weight, device_weight) inenumerate(zip(model_weights, device_weights)):
             model_weights[i] =model_weight+ (device_weight-model_weight) * (weight/total_weight)
 
 # Update the model with the averaged weights    
 model.set_weights(model_weights)

3. Federal average

 # Use federated averaging to aggregate the updated models
 model_weights=model.get_weights()
 client_weights= []
 fordeviceindevices:
     client_weights.append(device.get_weights())
 server_weights=model_weights
 for_inrange(num_rounds):
     fori, deviceinenumerate(devices):
         device.set_weights(server_weights)
         model.fit(x[i], y[i])
         client_weights[i] =model.get_weights()
     server_weights=server.federated_average(client_weights)
 
 # Update the model with the averaged weights
 model.set_weights(server_weights)

The above are the three most basic model aggregation methods in federated learning, I hope it will be helpful to you

https://avoid.overfit.cn/post/d426b291716c48409d3b68704545f6d0

Author: Dr Roushanak Rahmat, PhD