Research On Academic Quality Assessment And Its Evaluation System Of Open Access Journals In Biomedicine

BackgroundBecause of soaring prices of traditional academic journals as well as the library's subscription of a serious shortage of funds,has led to a global "Serials Crisis".In order to solve the crisis,a large-scale "open access"(OA) movement has been launched in the international publishing industries,academic circles,library and information sectors, government agencies.Open Access journals(OA journals),as one of OA publishing strategies,came into being.OA journals are a great vitality,full of vigor and vitality of the new periodicals and are showing a good momentum of development.OA journals are a great vitality,full of vigor and vitality of the new journals,showing a good momentum of development.They have been included and reported by more and more traditional abstracts/indexing tools; received recognition and supported by more and more publishers;more and more obtained the widespread approval of scientific researchers and readers.They are playing an increasingly important in the academic exchange system.However,under the premise of maintaining fast and low in cost,the most concern by academic circles,publishing industries,library and information sectors,extensive authors and readers,is whether biomedical OA journals can really carry out high-quality peer review,and maintain editorial integrity and high quality.Its academic quality and quality evaluation issues are always the point of debate and the focus of research.Nowadays,Academic quality of biomedical OA journals is evaluated more from the perspective of citations,and less from the perspective of web impact,academic performance,and much less from the synthetical evaluation index system and models.So, it is necessary to study Academic Quality Assessment and its Evaluation System of biomedical OA Journals,which is also of great theoretical and practical importance.Objective1.To analyze the overall quality,academic impact and discipline differences of biomedical OA journals,to compare deeply the advantages of biomedical OA journals with Non-OA journals,and to explore the impact of the free status on the academic quality of biomedical OA journals.2.To explore the scientific,applied,operational webometrics indexes of biomedical OA journals,to select the main indexes and factors that can reflect the web impact of biomedical OA journals,and to explore the feasibility,scientificalness and applicability of Webometrics indexes used in the evaluation of web impact of OA journals by block analysis,correlation analysis,principal components analysis.3.To analyze the distribution characteristics of h-index of biomedical OA journals, to probe into the application of h-index in the academic performance evaluation;to investigate the relationship between h-index and the index of traditional Bibliometrics,to make a deeply comparison about the consistency and difference when the indexes evaluate journals' academic performance,so as to find the advantages and disadvantages of h-index;To investigate the significance and effect of h-index,g-index and hc-index in academic performance evaluation,and then to evaluate their feasibility and applicability; to compare the consistency and difference of h-index between WOS and Google Scholar.4.Based on analyzing current evaluation systems and evaluation indexes of academic journals,the evaluation index framework of biomedical OA journals was proposed from the connotation of academic quality,evaluation indexes selected and their weight estimated,and then construct a set of synthetical evaluation index system of academic quality of biomedical OA journals,which are scientificalness,rationality and maneuverability,and to build a kind of scientific and effective synthetical evaluation model of academic quality of biomedical OA journals.To verify the scientificity, rationality,and feasibility what built above by empirical research.Methods1.The method of literature survey.To grasp the status of development and trends currently on which this paper studies,through investigation and analysis of numerous related research works all over the world.2.The method of online investigation.More than 400 kinds of biomedical OA journals and its related information and data of evaluating index were collected from OA journals' sites,search engines such as Google Scholar,Alltheweb and AltaVista, databases such as Web of Science,JCR and Ulrich's Periodicals Directory,and software such as PoP,etc.3.Questionnaire method.After designing two rounds of questionnaires about synthetical evaluation index of academic of quality of biomedical OA journals,to carry out two rounds of expert investigation,then to select synthetical evaluation indexes and to estimate weight.4.Comparative analysis.Compared with Non-OA journals' academic quality,it finds the advantages of biomedical OA journals.After comparing different rankings of biomedical OA journals,it explores the advantages and disadvantages,scientificalness and applicability of each evaluating method.5.Some theories and methods as Bibliometrics,Webometrics and h-index.The academic impact,web impact and academic performance of biomedical OA journals were evaluated from different perspectives using these theories and methods.6.Statistical analysis.It studies the theories and methods of academic quality evaluation of biomedical OA journals by using the statistical analysis soft SPSS15.0,and some methods such as statistical description,correlation analysis,multiple linear regression analysis,principal components analysis,cluster analysis and many kinds of synthetical evaluation methods.7.Technique and methods of Database.The biomedical OA journals Basic Database was built by using database technology of SQL Server and data were processed and analyzed using SQL.8.Attribute mathematics theory,connection mathematics theory and the method of attribute measure.The synthetical evaluation model of academic quality of biomedical OA journals was built Using these theory above.9.Empirical research.We verify the scientificity,rationality,and feasibility of the synthetical evaluation's index system and model of academic of quality of biomedical OA journals by empirical research.Results1.The academic impact of 483 kinds of biomedical OA journals were evaluated using the indexes such as impact factors(IF),immediacy index(ImInd) percentile rank,the average IF,the average ImInd,the average annual article quantity,and the average cited frequency,etc.The results are as follows:(1) On the whole,the average IF percentile of OA journals is 61.26,median is 70.42, the average immediacy index percentile is 61.75,and median is 68.47.(2) From the average IF and the average ImInd of OA journals,there are three kinds of OA journals whose average IF and the average immediacy index are obviously higher than Non-OA journals',namely Biomedicine,Humanities & Social Sciences and General Science.(3) From the average IF percentile and the average ImInd percentile of each subject, Biomedicine and General Science are all above the 60^th;Chemistry and Chemical engineering & Material science,Engineering and Technology,Agriculture and Food Science,Physics and Astronomy,Mathematics and Statistics are under the 50^th;Earth and Environmental Sciences,Humanities & Social Sciences are between the 50^th and the 60^th.(4) From the comparison of the advantages of academic quality between biomedical OA journals and Non-OA journals,biomedical OA journals show greater advantage than Non-OA journals in terms of the average total cited frequency,the cited frequency every article,the average IF,the average ImInd,the average annual article quantity,besides the average cited half-life,and the average odds ratios are 3.15,0.97,0.77,1.13 and 1.10 respectively.However,as time goes on,the advantages of biomedical OA journals decline day by day.There are obviously discipline differences in the academic quality advantage compared OA journals with Non-OA journals.(5) From the Free State impact on the academic quality of biomedical OA journals, whatever the average IF or the average ImInd,partly OA journals is greater than completely OA journals from 2001 to 2007(IF:3.228＞1.394;ImInd:0.542＞0.292).But their rising tendency are opposite,the rising tendency of the average IF or the average ImInd(namely the slope of the fitting line) of biomedical completely OA journals is clearly higher than partly OA journals(IF:0.122＞0.028;ImInd:0.03＞0.026).2.The web impact of 483 kinds of biomedical OA journals were evaluated using 11 indexes,namely the web literature volume,web pages,internal links number,web citations volume,web impact factor,total external web impact factor,total links number, external links number,IP visits,PV views and per capita page views.The results are as follows:(1) The web literature volume is between 0 and 1000 based on the biomedical OA journals from 2003 to 2007,the annual web literature volume is around 100.Among them there are 303 kinds of OA journals' web literature volume between 100 and 1000,which is 62.73%of the total(483),and the median is 502.But the amount of collection in WOS is about twice as the amount of web literature volume and the average the amount of collection in WOS exceeds the web literature volume nearly by 800 for each journal.The distribution of biomedical OA journals' web pages shows the discrete tendency and they are completely different.(2) The web citation volume is between 100 and 100000,accounted for 93.99%.The total number and the mean of web quotations(6760055 and 13995.9731 respectively) are both higher than that of WOS(5897869 and 12210.9089 respectively).(3) The total links number of journals between 0 and 100 is fewer,and the kind number is 52,accounted for 10.77%.But the number of journals' extemal links number and internal links number are in the interval which is 176 and 131 respectively,accounted for 36.44%and 27.12%respectively.There are 431 kinds of journal whose total links number is greater than 100,accounted for 89.23%,and 307 kinds whose external links number is greater than 100,accounted for 63.56%,and 352 kinds whose internal links number is greater than 100,accounted for 72.88%.(4) The distribution of biomedical OA journals websites' external impact factor is centralized,the average external impact factor is just 0.17,and its standard deviation is smaller.The total impact factor is much different,and its range is 171.60,its standard deviation is 12.08.(5) The IP visits and PV views are mainly below 1000,taking share of the total number of journal at 95.86%and 87.37%respectively.The journal's IP visits and PV views are between 0 and 100 is more,127 kinds and 102 kinds respectively,accounted for 28.04%and 21.12%respectively.Per capita page views are mainly between 0 and 3, accounted for 93.17%,and its mean is 2.042,its standard deviation is 0.7602. (6) The correlation between 9 indexes(namely the web citations volume,web literature volume,web pages,total links number,external links number,internal links number,external web impact factor,IP visits and PV views) and the IF of journals are low to intermediate,and the Pearson correlation coefficient are 0.550,0.243,0.243,0.232, 0.238,0.230,0.152,0.173 and 0.128 respectively.The values of P of significance test are all less than 0.01(bilateral).The total web impact factor(total WIF),Per capita page views and the IF of journals has no obvious correlation(P＞0.05),and the Pearson correlation coefficient are 0.045 and 0.039 respectively.(7) Put 10 indexes together into 4 principal constituents,their eigenvalues are all greater than 1,their cumulated variance contribution ratio amounts to 89.619%.3.The academic performance of 483 kinds of biomedical OA journals were evaluated and compared using h-index,h-like index and h_GS-index and traditional bibliometrics indexes.The results are as follows:(1) The distribution frequency of h-index is right-oblique,and h-index of most biomedical OA journals are between 5 and 50.The h-index of 96 kinds journals is greater than 50,accounted for 19.88%.The h-index lacks of uniqueness,that is to say,only 14 kinds journals where one h-index is corresponding to one journal,and one h-index is corresponding to many journal in others,for instance,there are 10 kinds journals when h-index is 5,20 kinds journals when h-index is 7,and 23 kinds journals when h-index is 14.And the lower h-index is,the more number of OA journals are.(2) There is a positive correlation(P＜0.01) between overall h-index and cited frequency of biomedical OA journals each year from 2003 to 2007,the Pearson correlation coefficient are 0.786,0.775,0.769,0.750,0.751 and 0.777 respectively,and significant correlation exists based on the level of 0.01;And there is a positive correlation (P＜0.01) between overall h-index and article quantity,the Pearson correlation coefficient are 0.517,0.545,0.545,0.537,0.539 and 0.551 respectively;there is also a positive correlation(P＜0.01) between overall h-index and average citation rates,the Pearson correlation coefficient is 0.620;there is a positive correlation(P＜0.01) between overall h-index and IF,the Pearson correlation coefficient is 0.678.(3) The results based on the clustering of traditional Bibliometrics indexes mainly concentrate on grade D,accounted for around 93%,and the total of others(grade A,B,C)is just accounted for 7%.When based on the clustering of h-index,the results are more scientific and rational,and the journals number of grade A,B,C,D are accounted for0.4%,7.7%,29%and 62.9%respectively,which are more realistic.(4) The h-index ranking has a positive correlation(P＜0.01) with the cited frequency ranking,the article quantity ranking,the average citation rates,and the IF ranking respectively,the Pearson correlation coefficient are 0.979,0.764,0.830 and 0.882 respectively.So h-index is much related to cited frequency,IF and the average citation rates.(5) The top 6 high-ranks are all review journals according to the total values of relative h-index of five years.Secondly,some small but excellent Non-Review journals' ranks are all on the rise,especially biomedical OA journals founded in recent years.The relative h-indexes of most journals have digressive tendency over time(increase gradually when back to time),and the relative h-indexes of a few journals fluctuate wildly.g-index is higher than h-index of biomedical OA journals,and the rank of journals according to g-index keeps in step with that of h-index.And g-index also solves the problems about h-index's lacking of discrimination and sensitivity.hc-index better reflect the current impact and the level of activity of biomedical OA journals.(6) In the 483 kinds of biomedical OA journals,the h-index of 330 kinds of them in Google Scholar is higher than that in WOS,and increases by an average of 6.58.The h-index is not changed only for 43 kinds of biomedical OA journals.The h-index of 110 kinds of them in Google Scholar is lower than that in WOS,and decreases by an average of 6.53.As a whole,h-index in Google Scholar is higher than that in WOS,and increases by an average of 3.01.The h_GS has a significant positive correlation(p＜0.01) with h_WOS,and the Pearson correlation coefficient is 0.967.The h_GS and h_WOS also have a significant positive correlation(p＜0.01) with IF of 2007,and the Pearson correlation coefficient are 0.647 and 0.678 respectively,and The difference is smaller.4.The evaluation system of academic quality of biomedical OA journals has been built from the synthetical evaluation index system of academic quality and synthetical evaluation model.The results are as follows:(1)20 evaluation indexes of academic quality of biomedical OA journals have been screened out from 5 aspects such as academic content,academic impact,network richness,web impact and academic performance,and the index weight has been set up by using Saaty's method.Thus a set of synthetical evaluation index system of academic quality of biomedical OA journals has been built,which includes 2 levels and 20 indexes.The index system not only includes dose-effect indexes such as the extent of dispatch countries or regions,the volume of being included by authoritative data bank, the web literature volume,internal link number,IP visits,etc,but also includes quality efficiency indexes such as IF,ImInd and total cited frequency,etc,and more includes some academic performance indexes such as h-index,g-index,etc.The index system covers 5 indexes in the first level(academic content,academic impact,network richness,web impact and academic performance),and 20 indexes in the second level.The indexes not only come from biomedical OA journals itself,such as IF, but also come from biomedical OA journals websites,such as external web impact factor, and more come from performance indexes integrated by both,such as h-index.All the data of the indexes are abundantly available,quantifiable and standardized.So the system is comprehensive,systematic,maneuverable and applicable.(2) Combining organically the theory of attribute mathematics with connection mathematics,a synthetical evaluation model of academic quality of biomedical OA journals is proposed based on attribute mathematics and connection mathematics.The single index classification criteria of 20 indexes and 100 attribute measure functions of single index have been built too.Its implementation procedure has been elaborated on, and the model is practical and feasible is proved by empirical research.(3) Using the index system and the evaluation model,10 kinds of biomedical OA journals are selected by stratified random sampling method,and then are synthetically evaluated and graded using attribute measure of single index,synthetical attribute measure of multi-index and attribute connection mathematics,and the findings are compared with the one of weighted rank sum ratio and weighted TOPSIS.The results have good consistency,and compare with the results of IF,the evaluation system to reveal more information of system structure.So the results are scientific,rational and feasible.Conclusion1.Through the evaluation research of academic impact of biomedical OA journals, the conclusions are as follows:(1) In general,the academic impact of biomedical OA journals is above the average, and gradually raised,some biomedical OA journals which have greater influence come into being.(2) There are discipline differences in biomedical OA journals' academic impact, main show is:there are three kinds of biomedical OA journals whose academic impact are obviously higher than Non-OA journals',namely Biomedicine,Humanities &Social Sciences and General Science,but there are not much different in the academic impact between biomedical OA journals and Non-OA journals compared the subject of Chemistry and Chemical engineering & Material science,Engineering and Technology, Agriculture and Food Science,Earth and Environmental Sciences,Physics and Astronomy,Mathematics and Statistics.In terms of specific subject,there are many differences in the academic impact among OA journals.(3) Biomedical OA journals show greater advantage than Non-OA journals,that is, biomedical OA journals have the positive promotion for academic quality of journals.As time goes on,the advantages of biomedical OA journals decline day by day,besides, there are obviously discipline differences.(4) The quality and academic impact of completely OA journals are lower than partly OA journals',but the growth rate in quality and academic impact is higher than partly OA journals'.So,it indicates that the completely OA publishing model has better academic impact than the partly OA publishing model.2.Through the evaluation research of web impact of biomedical OA journals,the conclusions are as follows:(1) Currently,the network degree of OA journals document is on the low side (mainly are incomplete OA journals),or has not been included and showed by search engine completely(full OA journals).And the construction scale of OA journals website is irregularly.(2) Biomedical OA journals can get more citations since it has the advantage of open access and can get many citations which are not included in ISI;thereby it can acquire more web citations.So,web citations are more applicable to the evaluation of web impact of OA journals.(3) Biomedical OA journals website can bring about greater web impact for its OA, and can get more link numbers and greater total web impact,but the external web impact and the completeness of inner structure remain to be further improved.(4) The IP visits,PV views and per capita page views are on the low side. Biomedical OA journals must strengthen its academic quality construction and network construction so as to increase biomedical OA journals' visits,views and stickiness of website.At the same time,we should strengthen propaganda to impel researchers to make use of these journals frequently which has access to information from them or publish his research findings on these journals.(5) The correlation between 9 indexes(namely the web citations volume,web literature volume,web pages,total links number,external links number,internal links number,external web impact factor,IP visits and PV views) and the IF of journals are low to intermediate,so they could be the evaluation indexes of academic quality of OA journals.But the total web impact factor and per capita page views are not related directly to academic quality of biomedical OA journals.(6) Among the web citation volume,web literature volume,web pages,total links number,external links number,internal links number,external web impact factor,total web impact factor,IP visits and PV views,one or more weak correlations exists,and between per capita page views and any other Webometrics indexes there is no correlation, besides,its dipartite degree is lower,therefore,per capita page views cannot be the evaluation index of academic quality of biomedical OA journals.(7) We summarize how all the indexes impact on the evaluation from four aspects, and the findings are below:the determinants of the first principal component are web pages,links number,external links number and internal links number,IP visits and PV views is in the second,the third one are web citation volume and web literature volume, and external WIF and total WIF is at last.Each principal component is the linear combination of each evaluation index,and it could not evaluate journals in some aspects by using one principal component.We should rank and evaluate journals based on contribution ratio of each principal component and the composite scores.3.Through the evaluation research of academic performance of biomedical OA journals,the conclusions are as follows:(1) In terms of h-index,the whole academic quality of OA journals is above the average currently.h-index is applicable to the evaluation of academic quality of OA journals,but lacking of discrimination and sensitivity.So we should add other indexes to discriminate when the h-index is equal used in the evaluation.(2) h-index,which can make up for the disadvantages of indexes of traditional Bibliometrics,has a good correlation with indexes of traditional Bibliometrics,and can be combined with traditional Bibliometrics,so as to mutual complement,and to make a objective and fair judgment on academic quality of OA journals from different perspectives.(3) The h-like index relative can make up for the disadvantages of h-index to a certain extent.,and the relative h-index improves the evaluation of review journals and the small but excellent journals,which can reveal the stability and activeness of academic quality of OA journals and it can be used with h-index.g-index has the similar effects as h-index when evaluating the academic quality of OA journals,and it can solve the problems on h-index's lacking of discrimination and sensitivity,he index better reflect current impacts of OA Journals than h index and it can be used as an important supplement to the h-index evaluation.(4) There are significant differences between h_GS and h_WOS.The h_GS can provide more accurate and more comprehensive information than that h_WOS can provide,so it can be the substitution or supplement to h_WOS.So when to evaluate academic quality of OA journals by using h-index,we should take various factors which have effects on it into account,such as collection scope,document type,quantity,search mechanism and so on of statistical source databases. 4.Through the evaluation research of Evaluation System of biomedical OA journals,the conclusions are as follows:(1) A set of synthetical evaluation index system of academic of quality of biomedical OA journals has been built,which includes 5 indexes in the first level and 20 indexes in the second level.And the system is scientific,rational,comprehensive, systematic,maneuverable and applicable.(2) The theories of attribute mathematics and connection mathematics are applied to the comprehensive evaluation of academic of quality of biomedical OA journals,and a synthetical evaluation model of academic quality of biomedical OA journals has been built,which is based on attributive mathematics and connection mathematics.The model is objective,effective and practical,and it provides a new way to comprehensively assess the classification of academic quality of biomedical OA journals.(3) Using the index system and the evaluation model,the classification criteria of 20 indexes of academic quality of biomedical OA journals and 100 attribute measure functions of single index have been built.We select 10 kinds of OA journals by using stratified random sampling,then to make comprehensive evaluation and grade using attribute measure of single index,synthetical attribute measure of multi-index and attribute connection mathematics,and the findings are compared with the one of weighted rank sum ratio and weighted TOPSIS.The results have good consistency,and compare with the results of IF,the evaluation system to reveal more information of system structure.So the results are more objective,scientific,and comprehensive. Therefore,the evaluation system is scientific,rational and feasible.