Conference Paper

Felicitas: Federated Learning in Distributed Cross Device Collaborative Frameworks

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Subsequently, many novel FL systems have emerged to adapt to diverse federated training scenarios, such as Horizontal FL (e.g. TFF [1], FedLab [256], Felicitas [267], IBM FL [151]), Vertical FL [238] or both (e.g. FATE [146], FedML [79], PaddleFL [155], Flower [11], FedTree [126], NVFLARE [195]). ...
... In a general FL setting, the server part is the central aggregator installed in a trusted cloud environment, while the client part of software can operate in different operating environments on client devices. The server and clients are connected via Internet and typically with the help of Remote Procedure Call (RPC) interface for coordinating [1,11,79,146,267]. We use four colors to represent the four FL roles and the colors with grid lines indicate non-essential roles. ...
Preprint
Full-text available
Traditional Federated Learning (FL) follows a server-domincated cooperation paradigm which narrows the application scenarios of FL and decreases the enthusiasm of data holders to participate. To fully unleash the potential of FL, we advocate rethinking the design of current FL frameworks and extending it to a more generalized concept: Open Federated Learning Platforms. We propose two reciprocal cooperation frameworks for FL to achieve this: query-based FL and contract-based FL. In this survey, we conduct a comprehensive review of the feasibility of constructing an open FL platform from both technical and legal perspectives. We begin by reviewing the definition of FL and summarizing its inherent limitations, including server-client coupling, low model reusability, and non-public. In the query-based FL platform, which is an open model sharing and reusing platform empowered by the community for model mining, we explore a wide range of valuable topics, including the availability of up-to-date model repositories for model querying, legal compliance analysis between different model licenses, and copyright issues and intellectual property protection in model reusing. In particular, we introduce a novel taxonomy to streamline the analysis of model license compatibility in FL studies that involve batch model reusing methods, including combination, amalgamation, distillation, and generation. This taxonomy provides a systematic framework for identifying the corresponding clauses of licenses and facilitates the identification of potential legal implications and restrictions when reusing models. Through this survey, we uncover the the current dilemmas faced by FL and advocate for the development of sustainable open FL platforms. We aim to provide guidance for establishing such platforms in the future, while identifying potential problems and challenges that need to be addressed.
... For cross-silo setting, clients are typically organizations such as data centers and there are stable network connections between clients. While in the cross-device setting [28,29], clients are typically a large number of IoT and mobile devices and the network connections between clients are poor. In this paper, we mainly focus on cross-device setting and horizontal FL. ...
Article
Full-text available
Federated Learning (FL) has emerged as a promising learning approach for utilizing data distributed across edge devices. However, existing works mainly focus on single-job FL systems. In practice, multiple FL jobs will be submitted simultaneously. How to schedule multiple FL jobs is crucial for client resource utilization and job efficiency. In addition, existing works assume that clients are always available during FL jobs, which is often not a reality since clients could be unavailable for FL jobs due to various reasons. To address these challenges, in this paper, we introduce a novel fault-tolerance multi-job scheduling strategy aimed at optimizing job efficiency and resource utilization. The basic idea of our approach is a redundancy-based fault tolerance mechanism, which is designed to ensure the robustness of FL jobs even with insufficient clients. The mechanism strategically selects clients for redundant model training. Based on the mechanism, the scheduling algorithm prioritizes urgent FL jobs, facilitating their completion and obviating the need for prolonged waiting periods for additional client availability. We conduct extensive experiments to demonstrate the effectiveness of the proposed method, which can significantly outperform other baseline methods.
... Some of them are general-purpose, while others focus on specific classes of algorithms, scenarios, topologies or domains of use. For instance, TensorFlow Federated [5], and PySyft [6]) focus on deep learning, Felicitas [7], and FairFed [8] are frameworks developed with a focus on the cross-device scenario, while Substra [9] implements the decentralized topology. Among general purpose frameworks we can mention Flower [10], FedML [11], and FATE [12], while FeatureCloud [13], and Sfkit [14] are examples of frameworks specifically developed for the biomedical sector that come with user-friendly web interfaces, allowing for practical creation and management of FL consortia. ...
Preprint
We present Flotta, a Federated Learning framework designed to train machine learning models on sensitive data distributed across a multi-party consortium conducting research in contexts requiring high levels of security, such as the biomedical field. Flotta is a Python package, inspired in several aspects by Apache Spark, which provides both flexibility and security and allows conducting research using solely machines internal to the consortium. In this paper, we describe the main components of the framework together with a practical use case to illustrate the framework's capabilities and highlight its security, flexibility and user-friendliness.
... Further, none of the empirical studies reported PETs-embedded mobile software applications. This oversight was surprising since the mobile application domain is a leading software domain [24,66] in which the incorporation of PETs has already proven feasible [23,41,102,123]. ...
Article
Federated learning involves training statistical models over edge devices such as mobile phones such that the training data is kept local. Federated Learning (FL) can serve as an ideal candidate for training spatial temporal models that rely on heterogeneous and potentially massive numbers of participants while preserving the privacy of highly sensitive location data. However, there are unique challenges involved with transitioning existing spatial temporal models to federated learning. In this survey paper, we review the existing literature that has proposed FL-based models for predicting human mobility, traffic prediction, community detection, location-based recommendation systems, and other spatial-temporal tasks. We describe the metrics and datasets these works have been using and create a baseline of these approaches in comparison to the centralized settings. Finally, we discuss the challenges of applying spatial-temporal models in a decentralized setting and by highlighting the gaps in the literature we provide a road map and opportunities for the research community.
Conference Paper
Full-text available
Fairness and robustness are two important concerns for federated learning systems. In this work, we identify that robustness to data and model poisoning attacks and fairness, measured as the uniformity of performance across devices, are competing constraints in statistically heterogeneous networks. To address these constraints, we propose employing a simple, general framework for personalized federated learning, Ditto, that can inherently provide fairness and robustness benefits , and develop a scalable solver for it. Theoretically , we analyze the ability of Ditto to achieve fairness and robustness simultaneously on a class of linear problems. Empirically, across a suite of federated datasets, we show that Ditto not only achieves competitive performance relative to recent personalization methods, but also enables more accurate, robust, and fair models relative to state-of-the-art fair or robust baselines.
Book
Full-text available
The term Federated Learning was coined as recently as 2016 to describe a machine learning setting where multiple entities collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client’s raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective. Since then, the topic has gathered much interest across many different disciplines and the realization that solving many of these interdisciplinary problems likely requires not just machine learning but techniques from distributed optimization, cryptography, security, differential privacy, fairness, compressed sensing, systems, information theory, statistics, and more. This monograph has contributions from leading experts across the disciplines, who describe the latest state-of-the art from their perspective. These contributions have been carefully curated into a comprehensive treatment that enables the reader to understand the work that has been done and get pointers to where effort is required to solve many of the problems before Federated Learning can become a reality in practical systems. Researchers working in the area of distributed systems will find this monograph an enlightening read that may inspire them to work on the many challenging issues that are outlined. This monograph will get the reader up to speed quickly and easily on what is likely to become an increasingly important topic: Federated Learning.
Conference Paper
Full-text available
Fairness has emerged as a critical problem in feder- ated learning (FL). In this work, we identify a cause of unfairness in FL – conflicting gradients with large differences in the magnitudes. To address this issue, we propose the federated fair averaging (FedFV) algorithm to mitigate potential conflicts among clients before averaging their gradients. We first use the cosine similarity to detect gradient con- flicts, and then iteratively eliminate such conflicts by modifying both the direction and the magnitude of the gradients. We further show the theoretical foundation of FedFV to mitigate the issue conflict- ing gradients and converge to Pareto stationary so- lutions. Extensive experiments on a suite of fed- erated datasets confirm that FedFV compares fa- vorably against state-of-the-art methods in terms of fairness, accuracy and efficiency. The source code is available at https://github.com/WwZzz/easyFL.
Article
Full-text available
Federated learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit its popularity, it has been observed that FL yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present clustered FL (CFL), a novel federated multitask learning (FMTL) framework, which exploits geometric properties of the FL loss surface to group the client population into clusters with jointly trainable data distributions. In contrast to existing FMTL approaches, CFL does not require any modifications to the FL communication protocol to be made, is applicable to general nonconvex objectives (in particular, deep neural networks), does not require the number of clusters to be known a priori, and comes with strong mathematical guarantees on the clustering quality. CFL is flexible enough to handle client populations that vary over time and can be implemented in a privacy-preserving way. As clustering is only performed after FL has converged to a stationary point, CFL can be viewed as a postprocessing method that will always achieve greater or equal performance than conventional FL by allowing clients to arrive at more specialized models. We verify our theoretical analysis in experiments with deep convolutional and recurrent neural networks on commonly used FL data sets.
Conference Paper
Full-text available
Deep neural networks are susceptible to various inference attacks as they remember information about their training data. We design white-box inference attacks to perform a comprehensive privacy analysis of deep learning models. We measure the privacy leakage through parameters of fully trained models as well as the parameter updates of models during training. We design inference algorithms for both centralized and federated learning, with respect to passive and active inference attackers, and assuming different adversary prior knowledge. We evaluate our novel white-box membership inference attacks against deep learning algorithms to trace their training data records. We show that a straightforward extension of the known black-box attacks to the white-box setting (through analyzing the outputs of activation functions) is ineffective. We therefore design new algorithms tailored to the white-box setting by exploiting the privacy vulnerabilities of the stochastic gradient descent algorithm, which is the algorithm used to train deep neural networks. We investigate the reasons why deep learning models may leak information about their training data. We then show that even well-generalized models are significantly susceptible to white-box membership inference attacks, by analyzing state-of-the-art pre-trained and publicly available models for the CIFAR dataset. We also show how adversarial participants, in the federated learning setting, can successfully run active membership inference attacks against other participants, even when the global model achieves high prediction accuracies.
Article
We address the problem of federated learning (FL) where users are distributed and partitioned into clusters. This setup captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task), they can leverage the strength in numbers in order to perform more efficient federated learning. For this new framework of clustered federated learning, we propose the Iterative Federated Clustering Algorithm (IFCA), which alternately estimates the cluster identities of the users and optimizes model parameters for the user clusters via gradient descent. We analyze the convergence rate of this algorithm first in a linear model with squared loss and then for generic strongly convex and smooth loss functions. We show that in both settings, with good initialization, IFCA is guaranteed to converge, and discuss the optimality of the statistical error rate. In particular, for the linear model with two clusters, we can guarantee that our algorithm converges as long as the initialization is slightly better than random. When the clustering structure is ambiguous, we propose to train the models by combining IFCA with the weight sharing technique in multi-task learning. In the experiments, we show that our algorithm can succeed even if we relax the requirements on initialization with random initialization and multiple restarts. We also present experimental results showing that our algorithm is efficient in non-convex problems such as neural networks. We demonstrate the benefits of IFCA over the baselines on several clustered FL benchmarks.
Article
Large‐scale data training is vital to the generalization performance of deep learning (DL) models. However, collecting data directly is associated with increased risk of privacy disclosure, particularly in special fields such as healthcare, finance, and genomics. To protect training data privacy, collaborative deep learning (CDL) has been proposed to enable joint training from multiple data owners while providing reliable privacy guarantee. However, recent studies have shown that CDL is vulnerable to several attacks that could reveal sensitive information about the original training data. One of the most powerful attacks benefits from the leakage from gradient sharing during collaborative training process. In this study, we present a new CDL framework, PrivateDL, to effectively protect private training data against leakage from gradient sharing. Unlike conventional training process that trains on private data directly, PrivateDL allows effective transfer of relational knowledge from sensitive data to public data in a privacy‐preserving way, and enables participants to jointly learn local models based on the public data with noise‐preserving labels. This way, PrivateDL establishes a privacy gap between the local models and the private datasets, thereby ensuring privacy against the attacks launched to the local models through gradient sharing. Moreover, we propose a new algorithm called Distributed Aggregation Stochastic Gradient Descent, which is designed to improve the efficiency and accuracy of CDL, especially in the asynchronous training mode. Experimental results demonstrate that PrivateDL preserves data privacy with reasonable performance overhead.
Article
The internet of things (IoT) is transforming major industries including but not limited to healthcare, agriculture, finance, energy, and transportation. IoT platforms are continually improving with innovations such as the amalgamation of software-defined networks (SDN) and network function virtualization (NFV) in the edge-cloud interplay. Deep learning (DL) is becoming popular due to its remarkable accuracy when trained with a massive amount of data, such as generated by IoT. However, DL algorithms tend to leak privacy when trained on highly sensitive crowd-sourced data such as medical data. Existing privacy-preserving DL algorithms rely on the traditional server-centric approaches requiring high processing powers. We propose a new local differentially private (LDP) algorithm named LATENT that redesigns the training process. LATENT enables a data owner to add a randomization layer before data leave the data owners’ devices and reach a potentially untrusted machine learning service. This feature is achieved by splitting the architecture of a convolutional neural network (CNN) into three layers: (1) convolutional module, (2) randomization module, and (3) fully connected module. Hence, the randomization module can operate as an NFV privacy preservation service in an SDN-controlled NFV, making LATENT more practical for IoT-driven cloud-based environments compared to existing approaches. The randomization module employs a newly proposed LDP protocol named utility enhancing randomization, which allows LATENT to maintain high utility compared to existing LDP protocols. Our experimental evaluation of LATENT on convolutional deep neural networks demonstrates excellent accuracy (e.g. 91%-96%) with high model quality even under low privacy budgets (e.g. ε=0.5).
Article
Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.
Conference Paper
Deep Learning has recently become hugely popular in machine learning for its ability to solve end-to-end learning systems, in which the features and the classifiers are learned simultaneously, providing significant improvements in classification accuracy in the presence of highly-structured and large databases. Its success is due to a combination of recent algorithmic breakthroughs, increasingly powerful computers, and access to significant amounts of data. Researchers have also considered privacy implications of deep learning. Models are typically trained in a centralized manner with all the data being processed by the same training algorithm. If the data is a collection of users' private data, including habits, personal pictures, geographical positions, interests, and more, the centralized server will have access to sensitive information that could potentially be mishandled. To tackle this problem, collaborative deep learning models have recently been proposed where parties locally train their deep learning structures and only share a subset of the parameters in the attempt to keep their respective training sets private. Parameters can also be obfuscated via differential privacy (DP) to make information extraction even more challenging, as proposed by Shokri and Shmatikov at CCS'15. Unfortunately, we show that any privacy-preserving collaborative deep learning is susceptible to a powerful attack that we devise in this paper. In particular, we show that a distributed, federated, or decentralized deep learning approach is fundamentally broken and does not protect the training sets of honest participants. The attack we developed exploits the real-time nature of the learning process that allows the adversary to train a Generative Adversarial Network (GAN) that generates prototypical samples of the targeted training set that was meant to be private (the samples generated by the GAN are intended to come from the same distribution as the training data). Interestingly, we show that record-level differential privacy applied to the shared parameters of the model, as suggested in previous work, is ineffective (i.e., record-level DP is not designed to address our attack).
Article
April 8, 2009Groups at MIT and NYU have collected a dataset of millions of tiny colour images from the web. It is, in principle, an excellent dataset for unsupervised training of deep generative models, but previous researchers who have tried this have found it di cult to learn a good set of lters from the images. We show how to train a multi-layer generative model that learns to extract meaningful features which resemble those found in the human visual cortex. Using a novel parallelization algorithm to distribute the work among multiple machines connected on a network, we show how training such a model can be done in reasonable time. A second problematic aspect of the tiny images dataset is that there are no reliable class labels which makes it hard to use for object recognition experiments. We created two sets of reliable labels. The CIFAR-10 set has 6000 examples of each of 10 classes and the CIFAR-100 set has 600 examples of each of 100 non-overlapping classes. Using these labels, we show that object recognition is signi cantly
Article
We present jump consistent hash, a fast, minimal memory, consistent hash algorithm that can be expressed in about 5 lines of code. In comparison to the algorithm of Karger et al., jump consistent hash requires no storage, is faster, and does a better job of evenly dividing the key space among the buckets and of evenly dividing the workload when the number of buckets changes. Its main limitation is that the buckets must be numbered sequentially, which makes it more suitable for data storage applications than for distributed web caching.
Article
We consider an efficient realization of the all-reduce operation with large data sizes in cluster environments, under the assumption that the reduce operator is associative and commutative. We derive a tight lower bound of the amount of data that must be communicated in order to complete this operation and propose a ring-based algorithm that only requires tree connectivity to achieve bandwidth optimality. Unlike the widely used butterfly-like all-reduce algorithm that incurs network contention in SMP/multi-core clusters, the proposed algorithm can achieve contention-free communication in almost all contemporary clusters, including SMP/multi-core clusters and Ethernet switched clusters with multiple switches. We demonstrate that the proposed algorithm is more efficient than other algorithms on clusters with different nodal architectures and networking technologies when the data size is sufficiently large.
An open-source deep learning platform from industrial practice
  • Yanjun Ma
  • Dianhai Yu
  • Tian Wu
  • Haifeng Wang
  • Ma Yanjun
Secure multi-party computation. Manuscript. Preliminary version
  • Oded Goldreich
  • Goldreich Oded
Communication-efficient learning of deep networks from decentralized data
  • Brendan Mcmahan
  • Eider Moore
  • Daniel Ramage
  • Seth Hampson
  • Blaise Aguera Y Arcas
  • McMahan Brendan
Fate: An industrial grade federated learning framework
  • Fate The
  • Authors
  • Authors The FATE
Leaf: A benchmark for federated settings
  • Sebastian Caldas
  • Sai Meher Karthik
  • Peter Duddu
  • Tian Wu
  • Jakub Li
  • H Brendan Konený
  • Virginia Mcmahan
  • Ameet Smith
  • Talwalkar
  • Caldas Sebastian
Fedml: A research library and benchmark for federated machine learning
  • Chaoyang He
  • Songze Li
  • Jinhyun So
  • Xiao Zeng
  • Mi Zhang
  • Hongyi Wang
  • Xiaoyang Wang
  • Praneeth Vepakomma
  • Abhishek Singh
  • Hang Qiu
  • Xinghua Zhu
  • Jianzong Wang
  • Li Shen
  • Peilin Zhao
  • Yan Kang
  • Yang Liu
  • Ramesh Raskar
  • Qiang Yang
  • Murali Annavaram
  • Salman Avestimehr
  • He Chaoyang
Tensorflow federated
  • Alex Ingerman
  • Krzys Ostrowski
  • Ingerman Alex
A generic framework for privacy preserving deep learning
  • Theo Ryffel
  • Andrew Trask
  • Morten Dahl
  • Bobby Wagner
  • Jason Mancuso
  • Daniel Rueckert
  • Jonathan Passerat-Palmbach
  • Ryffel Theo
Adaptive federated optimization
  • Sashank Reddi
  • Zachary Charles
  • Manzil Zaheer
  • Zachary Garrett
  • Keith Rush
  • Jakub Konený
  • Sanjiv Kumar
  • H Brendan Mcmahan
  • Reddi Sashank
Constantin Philippenko and Aymeric Dieuleveut. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees
  • Constantin Philippenko
  • Aymeric Dieuleveut
  • Philippenko Constantin
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • Zhenzhong Lan
  • Mingda Chen
  • Sebastian Goodman
  • Kevin Gimpel
  • Piyush Sharma
  • Radu Soricut
  • Lan Zhenzhong
Tackling the objective inconsistency problem in heterogeneous federated optimization
  • Jianyu Wang
  • Qinghua Liu
  • Hao Liang
  • Gauri Joshi
  • H Vincent Poor
  • Wang Jianyu
Federated optimization in heterogeneous networks
  • Tian Li
  • Anit Kumar Sahu
  • Manzil Zaheer
  • Maziar Sanjabi
  • Ameet Talwalkar
  • Virginia Smith
  • Li Tian
Communication compression for decentralized training
  • Hanlin Tang
  • Shaoduo Gan
  • Ce Zhang
  • Tong Zhang
  • Ji Liu
  • Tang Hanlin
Ensemble Distillation for Robust Model Fusion
  • Tao Lin
  • Lingjing Kong
  • Sebastian U Stich
  • Martin Jaggi
  • Lin Tao
Three approaches for personalization with applications to federated learning
  • Yishay Mansour
  • Mehryar Mohri
  • Jae Ro
  • Ananda Theertha Suresh
  • Mansour Yishay
Personalized federated learning: A meta-learning approach
  • Alireza Fallah
  • Aryan Mokhtari
  • Asuman Ozdaglar
  • Fallah Alireza
Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees
  • Constantin Philippenko
  • Aymeric Dieuleveut
  • Philippenko Constantin