Performance evaluation of MapReduce using full virtualisation on a departmental cloud

González-Vélez, Horacio; Kontagora, Maryam

doi:10.2478/v10006-011-0020-3

Abstract

This work analyses the performance of Hadoop, an implementation of the MapReduce programming model for distributed parallel computing, executing on a virtualisation environment comprised of 1+16 nodes running the VMWare workstation software. A set of experiments using the standard Hadoop benchmarks has been designed in order to determine whether or not significant reductions in the execution time of computations are experienced when using Hadoop on this virtualisation platform on a departmental cloud. Our findings indicate that a significant decrease in computing times is observed under these conditions. They also highlight how overheads and virtualisation in a distributed environment hinder the possibility of achieving the maximum (peak) performance.

References

Anon, E.A. (1998). A measure of transaction processing power, in M. Stonebraker and J.M. Hellerstein (Eds.), Readings in Database Systems, 3rd Edn., Morgan Kaufmann, San Francisco, CA, pp. 609-621.
Search in Google Scholar
Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R.H., Konwinski, A., Lee, G., Patterson, D.A., Rabkin, A., Stoica, I. and Zaharia, M. (2010). A view of cloud computing, Communications of the ACM53(4): 50-58.10.1145/1721654.1721672
Search in Google Scholar
Bacci, B., Danelutto, M., Pelagatti, S. and Vanneschi, M. (1999). SkIE: A heterogeneous environment for HPC applications, Parallel Computing25(13): 1827-1852.10.1016/S0167-8191(99)00072-1
Search in Google Scholar
Beaumont, O., Casanova, H., Legrand, A., Robert, Y. and Yang, Y. (2005). Scheduling divisible loads on star and tree networks: Results and open problems, IEEE Transactions on Parallel and Distributed Systems16(3): 207-218.10.1109/TPDS.2005.35
Search in Google Scholar
Buono, D., Danelutto, M. and Lametti, S. (2010). Map, reduce and MapReduce, the skeleton way, Procedia Computer Science1(1): 2089-2097.10.1016/j.procs.2010.04.234
Search in Google Scholar
Buyya, R., Yeo, C.S., Venugopal, S., Broberg, J. and Brandic, I. (2009). Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility, Future Generation Computer Systems—The International Journal of Grid Computing: Theory Methods and Applications25(6): 599-616.10.1016/j.future.2008.12.001
Search in Google Scholar
Buzen, J.P. and Gagliardi, U.O. (1973). The evolution of virtual machine architecture, Proceedings of the National Computer Conference and Exposition, AFIPS '73, ACM, New York, NY, pp. 291-299.
Search in Google Scholar
Cole, M. (1989). Algorithmic Skeletons: Structured Management of Parallel Computation, Pitman/MIT Press, London.
Search in Google Scholar
Cole, M. (2004). Bringing skeletons out of the closet: A pragmatic manifesto for skeletal parallel programming, Parallel Computing30(3): 389-406.10.1016/j.parco.2003.12.002
Search in Google Scholar
Danelutto, M. (2004). Adaptive task farm implementation strategies, 12th Euromicro Workshop on Parallel, Distributed and Network-Based Processing, PDP 2004, IEEE, La Coruña, pp. 416-423.
Search in Google Scholar
Dean, J. and Ghemawat, S. (2004). MapReduce: Simplified data processing on large clusters, Proceedings of the 6th conference on Symposium on Operating Systems Design & Implementation OSDI'04, Vol. 6, USENIX, San Francisco, CA, pp. 137-150.
Search in Google Scholar
Dean, J. and Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters, Communications of the ACM51(1): 107-113.10.1145/1327452.1327492
Search in Google Scholar
González-Vélez, H. (2006). Self-adaptive skeletal task farm for computational grids, Parallel Computing32(7-8): 479-490.10.1016/j.parco.2006.07.002
Search in Google Scholar
González-Vélez, H. and Cole, M. (2010a). Adaptive statistical scheduling of divisible workloads in heterogeneous systems, Journal of Scheduling13(4): 427-441.10.1007/s10951-009-0138-4
Search in Google Scholar
González-Vélez, H. and Cole, M. (2010b). Adaptive structured parallelism for distributed heterogeneous architectures: A methodological approach with pipelines and farms, Concurrency and Computation: Practice and Experience22(15): 2073-2094.10.1002/cpe.1549
Search in Google Scholar
González-Vélez, H. and Leyton, M. (2010). A survey of algorithmic skeleton frameworks: High-level structured parallel programming enablers, Software: Practice and Experience40(12): 1135-1160.10.1002/spe.1026
Search in Google Scholar
Ibrahim, S., Jin, H., Lu, L., Qi, L., Wu, S. and Shi, X. (2009). Evaluating MapReduce on virtual machines: The Hadoop case, in M. Jaatun, G. Zhao, and C. Rong (Eds.) Cloud-Com 2009, Lecture Notes in Computer Science, Vol. 5931, Springer-Verlag, Berlin/Heidelberg, pp. 519-528.10.1007/978-3-642-10665-1_47
Search in Google Scholar
Kontagora, M. and González-Vélez, H. (2010). Benchmarking a MapReduce environment on a full virtualisation platform, in L. Barolli, F. Xhafa, S. Vitabile and H.-H. Hsu (Eds.), CISIS 2010, The Fourth International Conference on Complex, Intelligent and Software Intensive Systems, Krakow, Poland, 15-18 February 2010, IEEE Computer Society, Washington, DC, pp. 433-438.10.1109/CISIS.2010.45
Search in Google Scholar
Kuchen, H. and Striegnitz, J. (2005). Features from functional programming for a C++ skeleton library, Concurrency and Computation: Practice and Experience17(7-8): 739-756.10.1002/cpe.844
Search in Google Scholar
Mesghouni, K., Hammadi, S. and Borne, P. (2004). Evolutionary algorithms for job-shop scheduling, International Journal of Applied Mathematics and Computer Science14(1): 91-103.
Search in Google Scholar
Nagarajan, A.B., Mueller, F., Engelmann, C. and Scott, S.L. (2007). Proactive fault tolerance for HPC with Xen virtualization, in B. J. Smith (Ed.), Proceedings of the 21th Annual International Conference on Supercomputing, ICS 2007, Seattle, Washington, USA, June 17-21, 2007, ACM, New York, NY, pp. 23-32.10.1145/1274971.1274978
Search in Google Scholar
Nokia Research Center (2009). Disco, Manual version 0.2.3, Nokia Research Center http://discoproject.org
Search in Google Scholar
Pisoni, A. (2007). Skynet, Manual version 0.9.3Geni.com,skynet.rubyforge.org
Search in Google Scholar
Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G. and Kozyrakis, C. (2007). Evaluating MapReduce for multi-core and multiprocessor systems, 13th International Conference on High-Performance Computer Architecture (HPCA-13 2007), Phoenix, AZ, USA, pp. 13-24.
Search in Google Scholar
Robertazzi, T.G. (2003). Ten reasons to use divisible load theory, Computer36(5): 63-68.10.1109/MC.2003.1198238
Search in Google Scholar
Sandholm, T. and Lai, K. (2009). MapReduce optimization using regulated dynamic prioritization, in J.R. Douceur, A.G. Greenberg, T. Bonald, J. Nieh (Eds.), Proceedings of the Eleventh International Joint Conference on Measurement and Modeling of Computer Systems, SIGMETRICS/Performance 2009, Seattle, WA, USA, June 15-19, 2009, ACM, New York, NY, pp. 299-310.10.1145/1555349.1555384
Search in Google Scholar
The Apache Software Foundation (2008). Hadoop MapReduce tutorial, Manual version 0.15, Hadoop Project http://hadoop.apache.org
Search in Google Scholar
VMware (2007). Understanding full virtualization, paravirtualization, and hardware assist, White Paper Revision: 20070911, VMware, Inc., Palo Alto, CA.
Search in Google Scholar
Whitaker, A., Shaw, M. and Gribble, S.D. (2002). Scale and performance in the Denali isolation kernel, ACM SIGOPS Operating Systems Review36(SI): 195-209.10.1145/844128.844147
Search in Google Scholar
Youseff, L., Wolski, R., Gorda, B. and Krintz, C. (2006). Paravirtualization for HPC systems, in G. Min, B. Di Martino, L.T. Yang, M. Guo and Gudula Rünger (Eds.), Frontiers of High Performance Computing and Networking—ISPA 2006 International Workshops, Sorrento, Italy, December 4-7, 2006, Lecture Notes in Computer Science, Vol. 4331, Springer-Verlag, Berlin/Heidelberg, pp. 474-486.10.2172/894791
Search in Google Scholar
Zaharia, M., Konwinski, A., Joseph, A., Katz, R. and Stoica, I. (2008). Improving MapReduce performance in heterogeneous environments, in R. Draves and R. van Renesse (Eds.), 8th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2008, December 8-10, 2008, San Diego, California, USA, USENIX Association, Berkeley, CA.
Search in Google Scholar

Performance evaluation of MapReduce using full virtualisation on a departmental cloud

Abstract

Paradigm

My account