| With the increasing popularity of microblogging services like Twitter, researchers discovered a rich medium for tackling real-life problems like event detection. However, event detection in Twitter is often obstructed by the lack of public evaluation mechanisms such as test collections (set of tweets, labels, and queries to measure the effectiveness of an information retrieval system). The problem is more evident when non-English languages, e.g., Arabic, are concerned. With the recent surge of significant events in the Arab world, news agencies and decision makers rely on Twitters microblogging service to obtain recent information on events. In this thesis, we address the problem of building a test collection of Arabic tweets (named EveTAR) for the task of event detection.;To build EveTAR, we first adopted an adequate definition of an event, which is a significant occurrence that takes place at a certain time. An occurrence is significant if there are news articles about it. We collected Arabic tweets using Twitter's streaming API. Then, we identified a set of events from the Arabic data collection using Wikipedias current events portal. Corresponding tweets were extracted by querying the Arabic data collection with a set of manually-constructed queries. To obtain relevance judgments for those tweets, we leveraged CrowdFlower's crowdsourcing platform.;Over a period of 4 weeks, we crawled over 590M tweets, from which we identified 66 events that cover 8 different categories and gathered more than 134k relevance judgments. Each event contains an average of 779 relevant tweets. Over all events, we got an average Kappa of 0.6, which is a substantially acceptable value. EveTAR was used to evaluate three state-of-the-art event detection algorithms. The best performing algorithms achieved 0.60 in F1 measure and 0.80 in both precision and recall. We plan to make our test collection available for research, including events description, manually-crafted queries to extract potentially-relevant tweets, and all judgments per tweet. EveTAR is the rst Arabic test collection built from scratch for the task of event detection. Additionally, we show in our experiments that it supports other tasks like ad-hoc search. |
