A Peer-reviewed Expert Counter-evaluation Study Based On Cluster Analysis

Objective:(1)To classify the expert review performance with application of clustering analysis and outliers mining.(2)To explore the identification method of the non-consensus projects and eliminate the non-consensus projects for expert meta-assessment.Methods:(1)Literature research:from 1994,as a starting point,we would to refer to a database of wanfang,CNKI,VIP database and ProQuest foreign language database to complete sorting the existing expert meta-assessment indexes and the identification indexes of the non-consensus projects,and then analyze the meaning and features to revise and select the indexes according to the demand.(2)Expert meeting:we would classify the existing assessment indexes to explore the scientific and feasible evaluation method for meta-assessment and finally determine the indexes for letter of comment.(3)Clustering analysis:after identifying and eliminating the non-consensus projects,we would make comprehensive evaluation for expert’ evaluation results with clustering analysis,and then select typical appraisal behavior with outliers diagnosis method based on clustering analysis.(4)Index calculation,basic statistical description and normality test is completed by SAS 9.13 statistical software,and outliers detection based on clustering analysis by SPSS ClementineResults:(1)The choice of the non-consensus index.From quantitative Angle,the measure index of the degree of non-consensus projects is maximum absolute deviation and coefficient of variation.The greater of the two indexes,the greater of the non-consensus degree is.Average relative deviation and average absolute deviation is to measure experts’non-consensus.The smaller of the average absolute deviation is and the closer to zero the average relative deviation is,the smaller of the non-consensus between all the experts is and vice versa.(2)The choice of the meta-assessment index.This study assumes that average is a true score of the project,so the discrete degree is to reflect the accuracy.Linear correlation coefficient is to reflect correlation.Ratio of discrete rate is to reflect rationality of degree of differentiation.The synthetic discrete rate and differences of consistency will reflect volatility of subjective error.The corrected shooting as a reference index,validates the rationality of the evaluation results.(3)The identification and treatment of non-consensus projects.When the maximum absolute deviation and coefficient of variation were both greater than the 95%percentile of the two index,it would be treat as a non-consensus project.For the project that did not enter the review meeting,we should give priority to most experts’ review opinions in theory,but also have to weigh meta-assessment results the degree of familiar with projects.For the project that entered the review meeting,we should focus on the maximum deviation of expert on what index he varies greatly with other experts.We should also improve the peer review mechanism,such as setting up the separate application,complaints and feedback channel to support significant innovation.(4)empirical research about meta-assessment.We carry out the empirical study with 162 experts who have evaluated more then 5 consensus projects.Experts were divided into two categories automatically.The first kind of expert’s meta-assessment results and the shooting percentage is much better than the second,which illustrated the effectiveness of the indexes and methods.When experts’ evaluation accuracy is high,the correlation and differentiation in general will be better.Volatility of subjective error will be smaller,if there is a special case,the expert is probably outliers.Assume that there are five percent of outliers in each type,then there are five outliers in the better kind,and there are three outliers in the ordinary kind according to the outliers mining.Outliers are better or worse experts relative to the same type because of some abnormal meta-assessment indexes.We would summarize the characteristics of typical experts’ review with graphics respectively.Conclusion:The subjective differences or systematic error may be one of the important reasons that generate the non-consensus on the project.This study combined maximum absolute deviation and coefficient of variation from quantitative perspective to identify and eliminate the non-consensus projects to enhance the rationality for meta-assessment.Exploring outliers based on clustering analysis can classify experts’ review level and filter out typical experts to analyze characteristics.It plays a positive role in improving the expert review behavior and achieving the optimal adjustment of experts.It is beneficial to expand the study methods for meta-assessment.