Performance Analysis & Tuning For Enterprise Data Warehouse Based On Big Data

With the continually rising of informational trend, information is becoming more and more important to our daily lives. Among those information, some are structured ones such as order and customers information while others are non-structured ones including audios, videos or pictures. The amount of those data is not only huge but also complex. So a new word was created for them: the big data. This is the same situation inside enterprise. When informationization just kick-off several decades years ago, the internal enterprise data is not so complex that IT system can handle them.While very soon the people found that those internal & external data are becoming more and more complex. It is getting more difficult in dealing with those data with traditional IT systems and databases efficiently. Then EDW comes out. EDW also push the development of big data theories and technologies forward. It is totally different with traditional database, EDW reflects the full-view and the dynamic trend of the data, and it is also the basic framework of Business Intelligence(BI). With the increase of the data inside EDW, the performance issue comes out with high priority to the enterprise leaders. Most performance issues occurred during the process of data extraction and data mining. This article will emphasis on the testing and analyzing the performance issue as soon as possible and fix those issues right before the system Move-To-Production.The article starts with the latest and hottest big data, based on the understanding of big data background, current situation and the future trends, import EDW as the foundation of study of the article. This article analyzed the history, structures and future application. Meanwhile, this article also systematically elaborated the basic theories and methodologies of traditional IT system’s performance analysis and tuning. These basic theories and methodologies are also the foundation of the EDW performance analysis and tuning. Although there are some differences between them, there are still many common ideas and technologies. The article also took the latest Vertica database as an example, discussed the performance analysis and tuning strategies and skills of EDW system. Finally, this article explained the whole process, technical details, tuning result and some lessons and learnt of a real EDW analysis and tuning project in detail which provided some valuable evidence and informative practical methodologies.