Research On Medical Dialogue Text Generation System Based On Federated Distillation

Artificial intelligence technologies are increasingly intertwined with many aspects of our daily lives.For most machine learning approaches,data is the core element driving their performance.In real-world scenarios,however,data can be scattered across different organizations,and we are often dealing with data that is small and fragmented,especially for the healthcare domain.Medical data is often highly sensitive,so sharing data between healthcare organizations has always been a challenge.For example,different hospitals have previously kept their own medical data locally and do not exchange data with other medical institutions,which may create "data silos" and therefore any one hospital will not have enough data to train a machine learning model with good performance.Second,it is even more difficult to obtain enough data as local laws have been introduced to protect data privacy.In this environment,federated learning has become a popular research area in machine learning today.Federated learning allows multiple clients to collaboratively train a shared model by iteratively aggregating model updates,which can help multiple participants to train a high-performance global model together without exposing their original data.At the same time,however,there are many challenges in federated learning.For example,the heterogeneity of private datasets across devices can lead to a high degree of dispersion in the data training,which hinders the aggregation of global models.In addition,the traditional federation learning training process incurs high communication overhead and can be difficult to train in scenarios where the models are heterogeneous.To solve the above problems,two federation learning algorithms based on bidirectional distillation and client selection are proposed in this paper,and the main research works are as follows:In this paper,we propose a federation learning algorithm based on bidirectional knowledge distillation,which integrates knowledge distillation into the two steps of local model upload and global model download for federation learning.In federated distillation,the system works by aggregating the model output information instead of model parameters or gradients,which will greatly reduce the overall communication overhead and this effect will be more obvious as the model parameters increase.Viewing the client-to-server distillation as the knowledge distillation process of multiple teachers,the global model is the unified student model of knowledge from multiple local teacher models.The server-toclient distillation process is a single teacher with multiple students,distilling knowledge from a single global model back to multiple local models.The experimental results of text generation on multiple medical conversation datasets show that this approach can speed up the convergence while ensuring the model performance.This paper also proposes a dynamic federated distillation algorithm based on client selection.First,an efficient knowledge aggregation mechanism is designed in federation learning.Second,a threshold-based technique is proposed to optimize each client’s local model update option for the problem that there is misleading knowledge.The gain of each client on the global model performance is calculated to decide whether to adopt the local model after knowledge distillation.This method can effectively limit the problem of local model performance degradation due to misleading knowledge,thus enabling efficient knowledge aggregation.Experiments based on multiple datasets and comparative methods show that the method speeds up training convergence over existing benchmarking methods and also supports the case of client model heterogeneity.