Skip to main content
Have a personal or library account? Click to login
Data Engineering Best Practices Cover

Data Engineering Best Practices

Architect robust and cost-effective data solutions in the cloud era

Publisher:Packt Publishing Limited
By:  and    
Paid access
|Mar 2025
Product purchase options
This title is not yet available for download
Description

Explore modern data engineering techniques and best practices to build scalable, efficient, and future-proof data processing systems across cloud platforms

Key Features

  • Architect and engineer optimized data solutions in the cloud with best practices for performance and cost-effectiveness
  • Explore design patterns and use cases to balance roles, technology choices, and processes for a future-proof design
  • Learn from experts to avoid common pitfalls in data engineering projects
  • Purchase of the print or Kindle book includes a free PDF eBook

Book Description

Revolutionize your approach to data processing in the fast-paced business landscape with this essential guide to data engineering. Discover the power of scalable, efficient, and secure data solutions through expert guidance on data engineering principles and techniques. Written by two industry experts with over 60 years of combined experience, it offers deep insights into best practices, architecture, agile processes, and cloud-based pipelines.
You’ll start by defining the challenges data engineers face and understand how this agile and future-proof comprehensive data solution architecture addresses them. As you explore the extensive toolkit, mastering the capabilities of various instruments, you’ll gain the knowledge needed for independent research. Covering everything you need, right from data engineering fundamentals, the guide uses real-world examples to illustrate potential solutions. It elevates your skills to architect scalable data systems, implement agile development processes, and design cloud-based data pipelines. The book further equips you with the knowledge to harness serverless computing and microservices to build resilient data applications.
By the end, you'll be armed with the expertise to design and deliver high-performance data engineering solutions that are not only robust, efficient, and secure but also future-ready.

What you will learn

  • Architect scalable data solutions within a well-architected framework
  • Implement agile software development processes tailored to your organization's needs
  • Design cloud-based data pipelines for analytics, machine learning, and AI-ready data products
  • Optimize data engineering capabilities to ensure performance and long-term business value
  • Apply best practices for data security, privacy, and compliance
  • Harness serverless computing and microservices to build resilient, scalable, and trustworthy data pipelines

Who this book is for

If you are a data engineer, ETL developer, or big data engineer who wants to master the principles and techniques of data engineering, this book is for you. A basic understanding of data engineering concepts, ETL processes, and big data technologies is expected. This book is also for professionals who want to explore advanced data engineering practices, including scalable data solutions, agile software development, and cloud-based data processing pipelines.

Table of Contents

  1. Overview of the Business Problem Statement
  2. A Data Engineer's Journey – Background Challenges
  3. A Data Engineer's Journey – IT's Vision and Mission
  4. Architecture Principles
  5. Architecture Framework – Conceptual Architecture Best Practices
  6. Architecture Framework – Logical Architecture Best Practices
  7. Architecture Framework – Physical Architecture Best Practices
  8. Software Engineering Best Practice Considerations
  9. Key Considerations for Agile SDLC Best Practices
  10. Key Considerations for Quality Testing Best Practices
  11. Key Considerations for IT Operational Service Best Practices
  12. Key Considerations for Data Service Best Practices
  13. Key Considerations for Management Best Practices
  14. Key Considerations for Data Delivery Best Practices
  15. Other Considerations – Measures, Calculations, Restatements, and Data Science Best Practices
  16. Machine Learning Pipeline Best Practices and Processes
  17. Takeaway Summary – Putting It All Together
  18. Appendix and Use Cases
https://github.com/PacktPublishing/Data-Engineering-Best-Practices
PDF ISBN: 978-1-80324-736-6
Publisher: Packt Publishing Limited
Copyright owner: © 2024 Packt Publishing Limited
Publication date: 2025
Language: English
Pages: 550
Related subjects:
Computer sciences,
Computer sciences, other
BISAC subjects:
COM021040,
COM062000,
COM018000
Thema subjects:
UND,
UNA,
UYZM,
UNC,
GPH

People also read