E-ETL: Framework for Managing Evolving ETL Workflows

Wojciechowski, Artur

doi:10.2478/fcds-2013-0005

Abstract

Data warehouses integrate external data sources (EDSs), which very often change their data structures (schemas). In many cases, such changes cause an erroneous execution of an already deployed ETL workow. Structural changes of EDSs are frequent, therefore an automatic reparation of an ETL workow, after such changes, is of a high importance. This paper presents a framework, called E-ETL, for handling the evolution of an ETL layer. Detection of changes in EDSs causes a repa- ration of the fragment of ETL workow which interacts with the changed EDSs. The proposed framework was developed as a module external to a standard commercial or open-source ETL engine, accessing the engine by means of API. The innovation of this framework consists in: (1) the algorithms for semi-automatic reparation of an ETL workow and (2) its ability to interact with various ETL engines that provide API.

References

[1] G. Papastefanatos, P. Vassiliadis, A. Simitsis, T. Sellis, and Y. Vassiliou, \Rule- based Management of Schema Changes at ETL sources," in Proc. of Conf. Ad- vances in Databases and Information Systems Workshops (ADBIS), pp. 55{62, Springer, LNCS 5968, 2010.10.1007/978-3-642-12082-4_8
Search in Google Scholar
[2] E. A. Rundensteiner, A. Koeller, X. Zhang, A. J. Lee, A. Nica, A. Van Wyk, and Y. Lee, \Evolvable View Environment (EVE): Non-Equivalent View Maintenance under Schema Changes," in Proc. of ACM Int. Conf. on Management of Data (SIGMOD), pp. 553{555, ACM Press, 1999.10.1145/304181.304574
Search in Google Scholar
[3] A. Wojciechowski, \E-ETL: Framework For Managing Evolving ETL Processes," in Proc. of Conf. Advances in Databases and Information Systems Workshops (ADBIS), vol. 185 of Advances in Intelligent Systems and Computing, pp. 441{ 449, Springer, 2013.10.1007/978-3-642-32518-2_42
Search in Google Scholar
[4] R. Wrembel and B. Bebel, \The Framework for Detecting and Propagating Changes from Data Sources Structure into a Data Warehouse," Foundations of Computing & Decision Sciences, vol. 30, no. 4, pp. 361{372, 2005.
Search in Google Scholar
[5] A. Wojciechowski and R. Wrembel, \Research Problems of the ETL Technol- ogy.," Foundations of Computing and Decision Sciences, vol. 35, no. 5, pp. 283{ 306, 2010.
Search in Google Scholar
[6] R. Wrembel, \On handling the evolution of external data sources in a data ware- house architecture," in Data Mining and Database Technologies: Innovative Ap- proaches (D. Taniar and L. Chen, eds.), IGI Group, 2011.
Search in Google Scholar
[7] G. Papastefanatos, P. Vassiliadis, A. Simitsis, and Y. Vassiliou, \Policy- Regulated Management of ETL Evolution.," J. Data Semantics, pp. 147{177, 2009.10.1007/978-3-642-03098-7_6
Search in Google Scholar
[8] A.Wojciechowski, \E-ETL: Framework For Managing Evolving ETL Processes.," in Proc. of Ph.D. Students in Information and Knowledge Management Work- shop (PIKM), pp. 59{66, ACM Press, 2011.10.1145/2065003.2065016
Search in Google Scholar
[9] J. Eder, C. Koncilia, and T. Morzy, \The COMET Metamodel for Temporal Data Warehouses," in Proc. of Int. Conf. on Advanced Information Systems Engineering (CAiSE), pp. 83{99, Springer-Verlag, 2002.10.1007/3-540-47961-9_9
Search in Google Scholar
[10] E. A. Rundensteiner, A. Koeller, and X. Zhang, \Maintaining data warehouses over changing information sources," Communications of the ACM, vol. 43, no. 6, pp. 57{62, 2000.10.1145/336460.336475
Search in Google Scholar

E-ETL: Framework for Managing Evolving ETL Workflows

Abstract

Paradigm

My account