| In to day’s Big dat a and mo bile int ernet era, ent erpr ises have mo re and mo re dat a so urces, increasing ly wit h large amount dat a. It fo llo wed by t hat t he co mp lexit y o f t he dat a pro cessing is increasing. When pro cessing dat a, work flo w schedu ling has a sig nificant imp act o n t he o verall per fo r mance and reso urce ut ilizat io n. This paper realized a d ist r ibut ed schedu ling syst em which u se Zoo Keeper, Red is, HDFS and ot her t echno lo g ies. The syst em co mes fro m need s o f real pro ject s fro m Co mpany X.The main co nt ent o f t his paper has t he fo llo wing t wo aspect s: 1) exp lo r ing t he co re need s o f t he indust r y’s abo ut wo rk flo w schedu ling, su mmar izing t hem and gu id ing t he develo p ment o f co re fu nct io ns fo r Bu ffalo schedu ling syst em. 2) based o n t he previo us research o f t ask sched u ling, t he appro pr iat e imp lement at io n is pro po sed, and t he hardware reso urce can be effect ively ut ilized.The main co ntr ibut io ns o f t his paper are:1) realizing t he core schedu ling lo g ic in schedu ling syst em t hat can suppo rt co mp lex dat a pro cessing wit h DAG work flow. Pr io r it y can be set fo r each t ask when schedu ling, so syst em has an unified schedu ling acco rd ing to t he pr io r it y order. 2) desig ning t he o verall archit ect ure fo r t he schedu ling syst em, so t hat schedu ling syst em it self can be do ne wit h t he bu siness co de co mp let ely separat ed wit h p lat for m co de. The d irect benefit s it brings is p lat for m can suppo rt a var iet y o f co mput ing framewo rks, such as Map Red uce, Spark and Stor m, and so o n. 3) opt imiz ing t he per fo rmance o f schedu ling syst em, support ing a large nu mber o f hig h co ncurrent t asks as p lanned. When t he t ask is t he amo u nt o f blo wo ut, yo u can ext end t he schedu ling syst em wit h new server d ynamic ally, t hereby increasing t ask t hro ughput in ensur ing t he no n-sto p sit uat io n. 4) pro vid ing a user-fr iend ly co nt ro l syst em includ ing t he t ask mo nit or ing, ser vice mo nit or ing, and lo g viewer and ot her basic fu nct io ns.The act ual e ffect o f t he pro ject sho ws t hat t he schedu ling syst em ca n co mp let e Co mpany X schedu ling needs fo r dat a pro cessing t asks, and suppo rt o ne hu ndred t ho usand hig h co ncurrent schedu ling t asks. Current ly, t he syst em ha s been ru nning o n t he line fo r mo nt hs, and has beco me an impo rt ant co mpo nent o f big dat a p lat for m so lut io ns fo r ot her co mp anies which have similar needs. |
PDF Full Text Request