Metamorphic Testing Of Code Search Engines And Sentiment Analysis Systems

Now,code search engines are widely used by software developers.However,due to the huge amount of data being processed as well as the lack of complete specifications,the testing of code search engines faces the oracle problem.Meanwhile,because of the intensive reliance on the code search engines,it is also crucial to investigate their quality of search results to help users choose their code search engines.And with the rapid development and wide application of natural language processing(NLP)technology,sentiment analysis has attracted vast attention because of its potentiality for application in industry and other fields.But the accuracy of NLP models is difficult to measure,and there is oracle problem in the evaluation of NLP models.And different people have different purposes when they use the sentiment alalysis system,it is of great significance to propose an appropriate mechanism to provide guidance to users.Because of the oracle problem in the testing of code search engines and sentiment analysis systems,we propose to apply metamorphic testing(MT)to these systems,and we also put forward the corresponding metamorphic relation(MR)to test the systems under test from two perspectives of software verification and validation.First of all,we identify five MRs for code search engines.Four MRs are defined based on the partial specifications of code search engines and thus are used for the purpose of verification,and the one MR identified from the users' perspective can be used to conduct validation.Secondly,in the testing of sentiment analysis system,we construct thirteen MRs to test the sentiment analysis systems which are sentence-level,and six MRs to the validation of systems that support the sentiment analysis of document-level.Finally,we apply these MRs in five famous code search engines and four sentiment analysis systems,and illustrate the effectiveness of metamorphic testing in evaluating software quality through violation rate of MR.The experimental results show that the abnormal behaviors of these code search engines have been successfully detected by the MRs we proposed,the results of these MRs based on the purpose of software validation can also help users understand the working mechanism of the software better and guide users to choose the appropriate system based on their own needs.The main contributions of this paper are as follows:(1)The metamorphic testing is applied to the testing of code search engine.Four MRs applied in code search engines are proposed in combination with partial requirement specifications.The results show that metamorphic testing can indeed find that there exist some defects in code search engines;(2)One MR is proposed from the perspective of user.The experimental results show that the code search engine has limitations in meeting the needs of users.According to these experimental results,users can choose the appropriate code search engine based on their own needs;(3)Nineteen MRs are proposed for the sentiment analysis system.The experimental results show that the expression of sentiment has a certain impact on the sentiment analysis ability of these systems,and these experimental results can give guiding opinions for users to choose the appropriate sentiment analysis systems.