Framework for Crawling and Local Event Detection Using Twitter Data

Twitter is a popular social media service, with millions of registered users as of December 2010. Twitter hosts substantial amounts of user-contributed data of realworld events. Twitter's core functions represent a simple social awareness stream model. Twitter users share information about upcoming events, the events the users are attending and events being broadcasted. The users also specify their location in their profile on Twitter. We can programmatically collect data from Twitter using their API and detect top terms and events in the data. Researchers can use this program to collect any kind of data from social networks easily. Journalists can get a real time the list of events detected by this method.;In this thesis, we propose a solution to tackle the problem above. We wrote scripts that collected Twitter data through Twitter API. The scripts collect data according to user location and by search keywords. We built a web interface that provides mechanism to manage the collection of data. The web interface allows addition of new locations and keywords to the data collection. We collected Twitter data for important locations across the United States of America and the world using these tools. We use two approaches to detect trends in the data. In the first approach, we detected spikes in data by looking at overall rate of tweets at each location over a period of time. In the second approach, we indexed the data according to location and time of the day. Then, we identified trends in the indexed data by ranking the terms according to spikes in term frequency. Using our framework, we can detect the top events and trends for a given time period and location according to Twitter data.