A Survey on Job Scheduling in Big Data

Senthilkumar, M.; Ilango, P.

doi:10.1515/cait-2016-0033

Abstract

Big Data Applications with Scheduling becomes an active research area in last three years. The Hadoop framework becomes very popular and most used frameworks in a distributed data processing. Hadoop is also open source software that allows the user to effectively utilize the hardware. Various scheduling algorithms of the MapReduce model using Hadoop vary with design and behavior, and are used for handling many issues like data locality, awareness with resource, energy and time. This paper gives the outline of job scheduling, classification of the scheduler, and comparison of different existing algorithms with advantages, drawbacks, limitations. In this paper, we discussed various tools and frameworks used for monitoring and the ways to improve the performance in MapReduce. This paper helps the beginners and researchers in understanding the scheduling mechanisms used in Big Data.

References

1. Abraham, A., R. Buyya, B. Nath. Nature’s Heuristics for Scheduling Jobs on Computational Grids. - In: Proc. of IEEE International Conference on Advanced Computing and Communications, 2000, pp. 1-8.
Search in Google Scholar
2. Ada, R., R. Kaur. - International Journal of Advanced Research in Computer Science and Software Engineering. - Ijarcsse, Vol. 3, 2013, No 3, pp. 665-668.
Search in Google Scholar
3. Assunção, M. D., R. N. Calheiros, S. Bianchi, M. A. S. Netto, R. Buyya. Big Data Computing and Clouds: Trends and Future Directions. - Journal of Parallel and Distributed Computing, 2015, 79-80, pp. 3-15.10.1016/j.jpdc.2014.08.003
Search in Google Scholar
4. Bardhan, S., D. Menasc. The Anatomy of Mapreduce Jobs, Scheduling, and Performance Challenges. - In: Proc. of Computer Measurement Group, 2013.
Search in Google Scholar
5. Bardhan, S., D. Menascé. Queuing Network Models to Predict the Completion Time of the Map Phase of Map Reduce Jobs, 2012.
Search in Google Scholar
6. Bu, Y., B. Howe, M. D. Ernst. Ha Loop : Efficient Iterative Data Processing on Large Clusters. - Proceedings of the VLDB Endowment, Vol. 3, 2010, No 1-2, pp. 285-296.10.14778/1920841.1920881
Search in Google Scholar
7. Casavant, T. L., J. G. Kuhl. A Taxonomy of Scheduling in General-Purpose Distributed Computing Systems. - IEEE Transactions on Software Engineering, Vol. 14, 1988, No 2, pp. 141-154.10.1109/32.4634
Search in Google Scholar
8. Coulouris, G., J. Dollimore, T. Kindberg. Distributed Systems: Concepts and Design. - Computer, Vol. 4, 2012.
Search in Google Scholar
9. Dabhi, V. K., H. B. Prajapati. Soft Computing Based Intelligent Grid Architecture. - In: Proc. of International Conference on Computer and Communication Engineering (ICCCE’08), Global Links for Human Development, 13-15 May 2008, pp. 574-577.
Search in Google Scholar
10. Davis, R. I., A. Burns. A Survey of Hard Real-Time Scheduling for Multiprocessor Systems. - ACM Computing Surveys, Vol. 43, 2011, No 4, pp. 1-44.10.1145/1978802.1978814
Search in Google Scholar
11. Dittrich, J., J.-A. Quiané-Ruiz, A. Jindal, Y. Kargin, V. Sett y, J. Schad. Hadoop++: Makinga Yellow Elephant Run Likea Cheetah (Without it Even Noticing). - Proceedings of the VLDB Endowment, Vol. 3, 2010, No 1-2, pp. 515-529.10.14778/1920841.1920908
Search in Google Scholar
12. Dong, F., S. G. Akl. Scheduling Algorithms for Grid Computing : State of the Art and Open Problems. - Components, Vol. 202, 2006, No 4, pp. 1-55.
Search in Google Scholar
13. Doulkeridis, C., K. Nørvåg. A Survey of Large-Scale Analytical Query Processing in Map Reduce. - VLDB Journal, Vol. 23, 2014, No 3, pp. 355-380.10.1007/s00778-013-0319-9
Search in Google Scholar
14. Ekanayake, J., H. Li, B. Zhang, T. Gunarathne, S. Bae, J. Qiu, G. Fox. Twister : A Runtime for Iterative Map Reduce. - In: Proc. of 19th ACM International Symposium on High Performance Distributed Computing, HPDC’10, 2010, pp. 810-818.
Search in Google Scholar
15. Hadoop Fair Scheduler Design Document, 2010, pp. 1-11
Search in Google Scholar
16. He, C., Y. Lu, D. Swanson. Matchmaking: A New Map Reduce Scheduling Technique. - In: Proc. of 3rd IEEE International Conference on Cloud Computing Technology and Science, Cloud Com’2011, 2011, pp. 40-47.
Search in Google Scholar
17. Herodotou, H., S. Babu. Profiling, What-if Analysis, and Cost-Based Optimization of Map Reduce Programs. - PVLDB: Proceedings of the VLDB Endowment, Vol. 4, 2011, No 11, pp. 1111-1122.10.14778/3402707.3402746
Search in Google Scholar
18. Herodotou, H., F. Dong, S. Babu. Mapreduce Programming and Costbased Optimization? Crossing this Chasm with Starfish. - Proceedings of the VLDB Endowment, Vol. 4, 2011, No 12, pp. 1446-1449.10.14778/3402755.3402792
Search in Google Scholar
19. Jahani, E., M. J. Cafarella, C. Ré. Automatic Optimization for Map Reduce Programs. - Proceedings of the VLDB Endowment, Vol. 4, 2011, No 6, pp. 385-396.10.14778/1978665.1978670
Search in Google Scholar
20. Gautam, J. V., H. B. Prajapati, V. K. Dabhi, S. Chaudhary. A Survey on Job Scheduling Algorithms in Big Data Processing. - In: Proc. of IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT’15), Coimbatore, 2015, pp. 1-11.10.1109/ICECCT.2015.7226035
Search in Google Scholar
21. Kc, K., K. Anyanwu. Scheduling Hadoop Jobs to Meet Deadlines. - In: Proc. of 2nd IEEE International Conference on Cloud Computing Technology and Science (Cloud Com’10), 2010, pp. 388-392.10.1109/CloudCom.2010.97
Search in Google Scholar
22. Kwon, Y., M. Balazinska, B. Howe, J. Rolia. Skew Tune: Mitigating Skew in Mapreduce Applications. - In: Proc. of 2012 ACM SIGMOD International Conference on Management of Data, 2012, pp. 25-36.
Search in Google Scholar
23. Lee, K. H., Y. J. Lee, H. Choi, Y. D. Chung, B. Moon. Parallel Data Processing with Map Reduce: A Survey. - SIGMOD Record, Vol. 40, 2011, No 4, pp. 11-20.10.1145/2094114.2094118
Search in Google Scholar
24. Liao, Xinyi, et al. An Enforcement of Real Time Scheduling in Spark Streaming. - In: Proc. of Green Computing Conference and Sustainable Computing Conference IEEE, 2015, pp. 1-6.
Search in Google Scholar
25. Lim, H., H. Herodotou, S. Babu. Stubby: A Transformation-Based Optimizer for Map Reduce Workflows. - Proceedings of the VLDB Endowment, Vol. 5, 2012, No 11, pp. 1196-1207.10.14778/2350229.2350239
Search in Google Scholar
26. Lin, Y., D. Agrawa l, C. Chen, B. C. Ooi, S. Wu. Llama: Leveraging Columnar Storage for Scalable Join Processing in the Map Reduce Framework. - In: Proc. of 2011 International Conference on Management of Data (SIGMOD’11), 2011, pp. 961-972.
Search in Google Scholar
27. Prajapati, H. B., V. A. Shah. Scheduling in Grid Computing Environment. - In: Proc. of 4th International Conference on Advanced Computing & Communication Technologies, 2014, pp. 315-324.10.1109/ACCT.2014.32
Search in Google Scholar
28. Rao, B. T., L. S. S. Reddy. Survey on Improved Scheduling in Hadoop Map Reduce in Cloud Environments. - International Journal of Computer Applications, Vol. 34, 2012, No 9, pp. 29-33.
Search in Google Scholar
29. Rasooli, A., D. G. Down. A Hybrid Scheduling Approach for Scalable Heterogeneous Hadoop Systems. - In: Proc. of 2012 SC Companion: High Performance Computing, Networking Storage and Analysis (SCC’2012), 2012, pp. 1284-1291.
Search in Google Scholar
30. Sakr, S., A. Liu, A. G. Fayoumi. The Family of Map Reduce and Large-Scale Data Processing Systems. - ACM Computing Surveys, Vol. 46, 2013, No 1, pp. 1-44.10.1145/2522968.2522979
Search in Google Scholar
31. International Journal of Advanced Research, Vol. 3, 2013, No 5, pp. 875-878.
Search in Google Scholar
32. Suthaharan, S. Big Data Classification: Problems and Challenges in Network Intrusion Prediction with Machine Learning. - Performance Evaluation Review, Vol. 41, 2014, No 4, pp. 70-73.10.1145/2627534.2627557
Search in Google Scholar
33. Verma, A., L. Cherkasova, R. Campbell. ARIA: Automatic Resource Inference and Allocation for Map Reduce Environments. - In: Proc. of 8th ACM International Conference on Autonomic Computing, 2011, pp. 235-244.
Search in Google Scholar
34. Wang, C. Journal of Computers, Vol. 8, 2013, No 3.
Search in Google Scholar
35. Wolf, J., A. Balmin, D. Rajan, K. Hildrum, R. Khandekar, S. Parekh, R. Vernica. CIRCUMFLEX: A Scheduling Optimizer for Map Reduce Workloads with Shared Scans. - ACM SIGOPS Operating Systems Review, Vol. 46, 2012, No 1.10.1145/2146382.2146388
Search in Google Scholar
36. Yang, Zhiw Ei, et al. Adaptive Task Scheduling Strategy for Heterogeneous Spark Cluster. - Computer Engineering, 2016.
Search in Google Scholar
37. Yong, M., N. Garegrat, S. Mohan. Towardsa Resource Aware Scheduler in hadoop. - In: Proc. of ICWS, 2009, pp. 1-10.
Search in Google Scholar
38. Yoo, D., K. M. Sim. A Comparative Review of Job Scheduling for Map Reduce. - In: IEEE International Conference on Cloud Computing and Intelligence Systems, 2011, pp. 353-358.10.1109/CCIS.2011.6045089
Search in Google Scholar
39. Zaharia, M., D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, I. Stoica. Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling. - In: Proc. of 5th European Conference on Computer Systems, 2010, pp. 265-278.
Search in Google Scholar
40. Zaharia, M., A. Konwinski, A. Joseph, R. Katz, I. Stoica. Improving Map Reduce Performance in Heterogeneous Environments. - Osdi, 2008, pp. 29-42.
Search in Google Scholar
41. Zhang, Y., Q. Gao, L. Gao, C. Wang. i Map Reduce: A Distributed Computing Framework for Iterative Computation. - Journal of Grid Computing, Vol. 10, 2012, No 1, pp. 47-68. 10.1007/s10723-012-9204-9
Search in Google Scholar

A Survey on Job Scheduling in Big Data

Abstract

Paradigm

My account