| BackgroundThe disease burden of cancer is growing globally and the demand for malignant tumor resection continues to increase,placing a strain on blood supplies worldwide;therefore,transfusion strategies that both ensure surgical safety for patients with cancer and conserve blood resources are urgently needed.Compared to a liberal transfusion strategy,a restrictive transfusion strategy allows patients to receive red blood cell(RBC)transfusion at a lower hemoglobin(HB)level.Up to now,policies for transfusing surgical oncology patients remain unclear owing to insufficient evidence from randomized controlled trials(RCTs).Therefore,transfusion guidelines cannot recommend a restrictive or liberal transfusion strategy and call for more RCTs to provide evidence.Actually,it is ethically unfeasible to conduct RCTs among certain patient subgroups of special clinical concern.Observational studies using daily health care practice data provide a unique opportunity to complement RCTs.However,a large number of RCTs and observational studies conducted on other types of surgical patients are divergent in terms of their objectives and findings and have been ongoing for three decades.For example,many observational studies have shown that perioperative blood transfusions can increase the risk of adverse outcomes,whereas no such effect was confirmed in RCTs.These conflicting findings have led pioneers in the field of blood transfusion strategies to question whether existing observational studies should inform transfusion practice.Bridging the wound that has existed for three decades is challenging,but essential to promote and augment evidence-based transfusion practice.Surgical oncology patients have great individual differences in age,cancer type,anemia,bleeding and so on.Transfusion guidelines also emphasize the importance of"individualized" patient blood management,but the current way is "population-based"and guided by hemoglobin levels,which is difficult to adapt to the needs of clinical decision-making and precision transfusion practice.Meanwhile,transfusion decisions in surgical oncology patients go through intensely changing phases of preoperative preparation-intraoperative shock-postoperative recovery,so perioperative transfusion is a series of decision-making processes rather than a single decision.The single decision point approach used in current transfusion studies is a simplification of the dynamic decision processes and is not conducive to longitudinal planning of transfusion decisions and global optimization of patient outcomes.Therefore,there is a need to explore an optimal dynamic treatment regime for perioperative transfusion in surgical oncology patients,i.e.,to give RBC transfusion volume decisions based on the patient’s current state at each decision point in order to obtain optimal patient outcomes.This coincides with the framework and goals of reinforcement learning,and the successful application of reinforcement learning to optimal drug dosing studies supports the feasibility of the framework to address the dynamic decision problem of perioperative transfusion.ObjectivesPart Ⅰ:Determine a feasible,tailored restrictive transfusion strategy with a possibly low safety threshold for various surgical oncology patient groups.Part Ⅱ:Explore optimal dynamic treatment regimes for individualized dynamic blood management of surgical oncology patients.MethodsThe data used in this study were from a multicenter prospective project "Modern Surgery and Anesthesia Safety Management System Construction and Promotion" in parallel periods(January to June 2015 and January to June 2016).Part Ⅰ:Restrictive RBC transfusion thresholds for surgical patients with cancerA.Study participants1.Base population(n=6055).We defined the base population as follows:(1)age≥18 years;(2)hospital stay≥ 24 hours;(3)the type of solid malignant tumor resection had a relatively high transfusion rate(≥1%).Patients in the base population are totaling with seven types of cancer:colorectal,lung,hepatopancreaticobiliary,urological,brain,oesophageal cancer,and stomach cancer.2.Primary analytic sample(n=1350).Patients in the based population further added the following exclusion criteria:(1)intraoperative bleeding volume≥500 mL;(2)severe anemia:hemoglobin concentration<6 g/dL;(3)irrational transfusion:hemoglobin concentration≥10 g/dL.These exclusions left a total of 1350 patients with stable hemoglobin of[6,10)g/dL as the primary analytic sample.B.Study variablesRBC transfusion information includes RBC transfusion time,RBC transfusion volumes,hemoglobin testing time,and hemoglobin level.The study outcomes included death(in-hospital death or death within 30 days of discharge)and complications(ischemic,infectious,and other complications),which were similar to those used in RCTs;to have enough sample size for the stability of results,we defined an adverse outcome by combining death and all complications.Covariates included demographic characteristics(age,sex,body mass index[BMI]);comorbidities in the medical records or diagnosis during hospitalization(coronary heart disease,diabetes,hypertension,pulmonary disorders);American Society of Anesthesiologists(ASA)score;preoperative laboratory tests(albumin,creatinine,white blood cell count,hemoglobin);intraoperative characteristics(operative time,bleeding volume);postoperative characteristics(admission to the intensive care unit[ICU]after surgery);cancer site;and study site.C.Statistical analysis1.Descriptions of demographic information,surgical information,and blood transfusion information in the base population.To identify between-group differences in patients’ characteristics,we Used the χ2 test or Fisher’s exact test for categorical variables and the t-test or Wilcoxon’s rank-sum test for continuous variables,as appropriate.2.Identification and quantification of heterogeneity.Assess the impact of changes in patient heterogeneity on the association between RBC transfusion and the adverse outcome by restricting the base population to the primary analytic sample and further using inverse probability of treatment weighting propensity score(PS)analysis.Heterogeneity by transfusion status was quantified using the standardized mean difference(SMD),for which≤10%the heterogeneity was considered acceptable.Logistic regression with adjustment for covariates was used to assess the association between RBC transfusion and the adverse outcome.3.Evaluate the safety of a restrictive transfusion strategy in the primary analytic sample by the following steps.(1)Definition of trigger hemoglobin.We defined a trigger hemoglobin level as the nearest measurement before the initial RBC transfusion for transfused patients and the nadir hemoglobin reading during hospitalization for non-transfused patients.Among the changing hemoglobin levels,the choice of the indicative one(last prior to transfusion;nadir without transfusion)was made to maximize the likelihood of reflecting the role of hemoglobin in the transfusion decision.(2)Definition of exposure and control groups.Exposure and control groups were defined on a trigger hemoglobin level range of[X1,X2)g/dL.The exposure group was patients who received one or greater than one allogeneic RBC transfusion during their hospitalization,and the control one was those who did not receive any allogeneic RBC transfusion.Comparing the two groups is analogous to comparing liberal transfusion threshold X2 g/dL and restrictive transfusion threshold X1 g/dL.(3)Assess the association between RBC transfusion and the adverse outcome.For greater flexibility in exploring feasible transfusion thresholds for patients with various types of cancer,a comparison analysis between exposure and control groups was independently performed in the trigger hemoglobin groups[6,8)g/dL and[8,10)g/dL among the total patients with cancer,i.e.,the primary analytic sample.A similar analysis was performed on patients with specific cancer.(4)Test the safety of transfusion thresholds in transfused patients.We compared the adverse outcome following transfusion at lower[6,8)g/dL and higher[8,10)g/dL trigger hemoglobin in transfused patients with the total cancers and with specific cancer.(5)Evaluation of validity.Analyses were conducted for an overall evaluation of result validity.First,the proportions of missing values were compared by transfusion status.Next,verify the stability of hemoglobin level after receiving the RBC transfusion.Part Ⅱ:Optimal dynamic treatment regime for individualized dynamic blood management in surgical oncologyA.Training set and test setA sample for exploring the optimal dynamic treatment regime(DTR)is obtained from patients in the base population with potential transfusion decision needs.This sample is then randomly divided into a training set and a test set in the ratio of 8:2.B.Reinforcement learning(RL)frameworkThe RL agent is equivalent to an artificial intelligence(AI)clinician,who makes a decision on the RBC transfusion volume for transfusion according to the current patient state.The patient will receive the recommended RBC transfusion volume,transfer to the next state and feedback the reward to the RL agent.The RL agent is aimed to learn the optimal policy to make the cumulative reward score as high as possible,i.e.,an AI clinician is aimed to learn the optimal DTRs to minimize the occurrence of adverse outcomes.1.Definition of state spaceThe state space includes variables that did not change over time:sex,age,BMI,ASA score,duration of surgery,comorbidities in the medical records or diagnosis during hospitalization,cancer sites,and hospital;and variables that changed over time:hemoglobin level,intraoperative bleeding volume,and admission to ICU after surgery.2.Definition of action spaceThe action space includes RBC transfusion volume for transfusion.3.Definition of rewardThe reward score represents the patient outcomes and is calculated by the comprehensive complication index,taking a score from 0 to 100.A score of 0 is assigned when the patient outcome is death,and a score of 100 is assigned when there is no death and the number of complication items is zero.C.Learning optimal policy in the training setThe deep Q-network(DQN)algorithm is used to train the RL agent on the training set to learn the optimal Q-value function,by estimating and updating the Q-value function,which can make the final cumulative reward score to be as high as possible to get the optimal policy.DQN is a combination of deep learning and reinforcement learning,which approximates a Q-value function by a deep neural network in a Q-learning framework that can deal with a high-dimensional state space.D.Evaluating optimal policy in the testing setThe optimal policy given is evaluated using the patient treatment trajectories in the testing set generated by the actual human clinician.The optimal policy given by the AI clinician can be considered better than the human clinician’s policy when the average reward of patients whose actual treatment decisions are consistent with the AI clinician’s decisions higher than that of patients whose actual treatment decisions are inconsistent with the AI clinician’s decisions.ResultsPart Ⅰ:Of 6055 participants in the base population(32.27%for colorectal cancer,20.34%for lung cancer,18.31%for stomach cancer,0.37%for hepatopancreaticobiliary cancer,7.69%for brain cancer,7.03%for urological cancer,3.96%for oesophageal cancer),836 patients(13.81%)underwent RBC transfusion with a mean of 3.9 units.Systemic bias was noted between patients who did and did not receive a transfusion,especially for two primary indications for transfusion:Intraoperative bleeding and anemia.Compared with the non-transfused patients,the rate of intraoperative hemorrhage(bleeding volume≥500 mL)was 9.2 times higher(45.33%vs 4.92%,P<0.0001),and the proportion of preoperative hemoglobin<8 g/dL was 4.3 times higher in the transfused patients(18.42%vs 4.33%,P<0.0001).After restricting patients to stable hemoglobin of[6,10)g/dL and performing PS analysis,heterogeneity by transfusion status was greatly reduced(all SMD≤10%).In terms of the association between transfusion and the adverse outcome,there was a significant association in the base population(OR 1.82,95%CI 1.27-2.62);however,transfusion was no longer associated with a risk of the adverse outcome in the primary analytic sample,and the estimated OR had greater precision after PS analysis(original:adjusted OR 1.19,95%CI 0.69-2.06;PS:adjusted OR 1.18,95%CI 0.71-1.98).Overall,transfusion remained unrelated to risk of the adverse outcome in the trigger hemoglobin groups[6,8)g/dL and[8,10)g/dL for total patients with cancer(HB 6-8 g/dL:OR 1.20,95%CI 0.57-2.53;HB 8-10 g/dL:OR 1.00,95%CI 0.44-2.24),especially for patients with colorectal cancer at[6,8)g/dL hemoglobin(HB 6-8 g/dL:OR 0.54,95%CI 0.17-1.68;HB 8-10 g/dL:OR 1.63,95%CI 0.56-4.77),and patients with stomach cancer at[8,10)g/dL hemoglobin(HB 6-8 g/dL:OR 4.35,95%CI 0.63-30.18;HB 8-10 g/dL:OR 0.20,95%CI 0.01-3.01).To further evaluate the safety of the two restrictive transfusion thresholds,we compared patient outcomes in the[6,8)g/dL versus[8,10)g/dL threshold groups in transfused patients.Patients transfused at the hemoglobin[6,8)g/dL versus[8,10)g/dL showed equal outcomes for total cancers(OR 1.08,95%CI 0.44-2.65),especially for colorectal cancer(OR 0.46.95%CI 0.12-1.82).The proportion of missing covariates among all covariates was from 0%to 4.5%.No statistically significant difference in the rate of missing data was found between the transfused and non-transfused groups.Hemoglobin levels in the 8-10 and 6-8 hemoglobin threshold groups remained stable after the first RBC transfusion.Part Ⅱ:Focusing the time window on the more intensive transfusion decision needs from 4 days preoperatively to 4 days postoperatively for a total of 9 days;the number of patients with potential transfusion decision needs in the base population was 431,i.e.,431 nine-stage patient treatment trajectories for the reinforcement learning.The optimal dynamic treatment regime will be learned from 344 patient treatment trajectories in the training set,and its performance will be evaluated among 87 patient treatment trajectories in the testing set.The evaluation results of the optimal policy given by the Al clinician showed that the recommended RBC transfusion volume was consistent with those given by the human clinicians in 82 patient treatment trajectories(94.25%),and their average reward was 98.21 points;the recommended RBC transfusion volume was inconsistent with those given by the human clinicians in rest 5 patient treatment trajectories(5.75%),and their average reward was 87.34 points.This indicated that when human clinicians give decisions that are closer to those of an AI clinician,patients have a higher average reward,i.e.,less occurrence of adverse outcomes.ConclusionThis study explores the safety of perioperative red blood cell transfusion in surgical oncology patients from two perspectives,by traditional statistical methods and emerging machine learning techniques-hemoglobin-oriented"population-based" transfusion thresholds;and patient characteristics-focused"individualized " transfusion dynamic decisions.Current evidence on transfusion practice in surgical oncology patients is limited.This study is the first to find that a restrictive transfusion strategy is feasible in surgical oncology patients by observational data:a restrictive transfusion threshold of 8 g/dL may be feasible for all patients with cancer,with a threshold as low as 6 g/dL for those with colorectal cancer.This study developed a hemoglobin-based design that can break through the limitations of disease categories and provides a reference for investigating transfusion thresholds for multiple cancers simultaneously.The hemoglobin-based design can align the objective of observational studies with that of RCTs,which contributes to areas unexplored by RCTs and allows the evidence to guide transfusion practice with a unified voice,therefore greatly promoting the healthy development of evidence-based surgical transfusion practice research.Reinforcement learning is a viable approach for personalized dynamic blood perioperative transfusion decision-making in surgical oncology.Patients whose human clinician decisions matched the optimal DTR given by the AI clinician had higher reward scores,i.e.,a lower incidence of adverse outcomes.This study is the first attempt to apply reinforcement learning to medical observational data in the field of blood transfusion,providing a new idea for an individualized dynamic perioperative blood management. |
PDF Full Text Request