Data Ingestion with Python Cookbook

A practical guide to ingesting, monitoring, and identifying errors in the data ingestion process

Publisher:Packt Publishing Limited

By: Gláucia Esppenchutz

Paid access

|May 2024

E-Book €23.99Institutions €123.95

Description

Deploy your data ingestion pipeline, orchestrate, and monitor efficiently to prevent loss of data and quality

Key Features

Harness best practices to create a Python and PySpark data ingestion pipeline
Seamlessly automate and orchestrate your data pipelines using Apache Airflow
Build a monitoring framework by integrating the concept of data observability into your pipelines

Book Description

Data Ingestion with Python Cookbook offers a practical approach to designing and implementing data ingestion pipelines. It presents real-world examples with the most widely recognized open source tools on the market to answer commonly asked questions and overcome challenges.
You’ll be introduced to designing and working with or without data schemas, as well as creating monitored pipelines with Airflow and data observability principles, all while following industry best practices. The book also addresses challenges associated with reading different data sources and data formats. As you progress through the book, you’ll gain a broader understanding of error logging best practices, troubleshooting techniques, data orchestration, monitoring, and storing logs for further consultation.
By the end of the book, you’ll have a fully automated set that enables you to start ingesting and monitoring your data pipeline effortlessly, facilitating seamless integration with subsequent stages of the ETL process.

What you will learn

Implement data observability using monitoring tools
Automate your data ingestion pipeline
Read analytical and partitioned data, whether schema or non-schema based
Debug and prevent data loss through efficient data monitoring and logging
Establish data access policies using a data governance framework
Construct a data orchestration framework to improve data quality

Who this book is for

This book is for data engineers and data enthusiasts seeking a comprehensive understanding of the data ingestion process using popular tools in the open source community. For more advanced learners, this book takes on the theoretical pillars of data governance while providing practical examples of real-world scenarios commonly encountered by data engineers.

Introduction to Data Ingestion
Principals of Data Access – Accessing your Data
Data Discovery – Understanding Our Data Before Ingesting It
Reading CSV and JSON Files and Solving Problems
Ingesting Data from Structured and Unstructured Databases
Using PySpark with Deﬁned and Non-Deﬁned Schemas
Ingesting Analytical Data
Designing Monitored Data Workflows
Putting Everything Together with Airﬂow
Logging and Monitoring Your Data Ingest in Airflow
Automating Your Data Ingestion Pipelines
Using Data Observability for Debugging, Error Handling, and Preventing Downtime

Table of contents

Authors

Metrics

PDF ISBN: 978-1-83763-309-8

Publisher: Packt Publishing Limited

Publication date: 2024

Language: English

Pages: 414

Related subjects:

Computer sciences,

Databases and data mining

Data Ingestion with Python Cookbook

Key Features

Book Description

What you will learn

Who this book is for

Table of Contents

People also read

Publications carousel

Data Engineering with Python

Modern Data Architectures with Python

Data Analysis with Python

Exploratory Data Analysis with Python Cookbook

Applied Data Science with Python and Jupyter

Data Engineering with Databricks Cookbook

Practical Data Science with Python

Building Data Integration Solutions

Practical Data Science Cookbook, Second Edition

Cleaning Data for Effective Data Science

Paradigm

My account