| Internet has penetrated through all respects of human life, becoming a most important way for us to retrieve imformation. Within these years, computer techniques and internet applications has brought the rapid growth of data, it’s been paid more and more attention that how to effectively get high-value, understandable knowledge to help mankind make tough decisions with low cost.The information generated by large-scale internet applications is proved to be with vast amount but merely fine quality and not helpful without timely analyzing. Disperse business requirements result to decentralized storage in hardware as well as software, bringing more problems in data join and migration. Data cleansing is a helpful technique in organizing, redundancy remova and quality raise, it’s been even curcial nowadays.Recommender system is an effective technical schema to help people get information they need. With users’ everyday behavior taken into calculation and information filtering, it brings win-win to both users and information providers. With the growing of large-scale web applications, bigdata has a key impact to recommender techniques.This thesis discusses the foundamental elements of data cleansing and recommender systems and development of them both at home and abroad. Focusing on the automation of data cleansing and big-data challenge of recommender systems, this essay mainly discuss:1, Theories and applications of various classic recommender algorithms, comparing their performance via precision and recall rate on a real-world dataset. Then a time-weight based collaborative filtering technique is proposed.2, Fundamental theories, exsiting forms of data quality problems and procedures of data cleansing. Then based on real-world dataset example, automated data cleansing is implemented using an ETL framework called Scriptella.3, A mapreduce implementation based on Apache hadoop and test of the time-weight collaborative filtering algorithm has been achived. |
