![[EN] Defense Against the Dark Arts of Data Pipelines #1: Fundamentals of Data Pipeline failure](https://prod-assets.cosmic.aws.dev/a/3EFw8eidGYbG0HDXFxEjpU3tnod/S3-F.webp)
[EN] Defense Against the Dark Arts of Data Pipelines #1: Fundamentals of Data Pipeline failure
Providing the fundamentals of why a data pipeline can fail and how we should understand this in order to define criteria for addressing a solution.
A data pipeline is an analytical solution that companies use to generate the information and knowledge needed to make business decisions and improve the efficiency of their operations. Therefore, the data team must ensure the proper functioning of the data pipeline to ensure that information is always available to the client, who could be a marketing team leader or the CEO.
This can then be viewed as follows:
Data pipeline is working correctly 🟰 Required information is always available 🟰 Customer is happy and empowered
As a data engineer, ensuring the data pipeline functions correctly will always be part of your daily duties. To achieve this, you must protect the data pipeline from the dark elements that could cause it to fail, but the question one might ask is:
🤔💭 What are the elements that cause a data pipeline to fail?
This article, which is part of a series, will seek to provide references to the elements that cause a data pipeline to fail, in order to apply an efficient solution for it; which will possibly be another article.
Those involved in the failure
To answer what elements can cause its failure, we need to examine the data pipeline from the perspective of its components in its daily operation. In my personal opinion, these are the most common elements found in a data pipeline:
- Data: Records of important events at a specific point in time, structured to be captured by the data pipeline.
- Processing Logic: Indicates how the data pipeline should process the data and generate the expected result for the client. This logic may be divided into tasks.
- Hardware and Software Services: Which software and hardware services are used to fully implement the data pipeline.
- Orchestration: How a group of tasks within a data pipeline should execute correctly, defining the exact sequence of tasks according to the expected behaviors.
- Integrations: How the data pipeline interacts with external elements such as APIs, queues, databases, or other data pipelines.
- Client Expectations: The client's expectations regarding the results that the data pipeline must deliver. This applies to business clients, as well as suppliers and teams collaborating throughout the data pipeline lifecycle.
- Deployment: How a data pipeline is moved to production, synchronized with the data and operational environment.
Since they are building blocks of a data pipeline, the collective interaction of each element among them could be the cause of the failure.
⛓️ A data pipeline is only as strong as its weakest link.
The failure caused by the absence of a mechanism in the data pipeline solution that allows it to manage the root cause of the failure , which is directly related to a previously listed element. Examples of failure es associated with each element as the root cause are listed below:
| Associated data pipeline element | Example of a failure |
|---|---|
| Data | Unsocialized changes to the data structure. Unsocialized changes to data types. |
| Processing Logic | Functions with behaviors that do not operate as expected. |
| Hardware and Software Services | Data volumes exceeding hardware capacity. Network disconnection. |
| Orchestration | Incorrect task management according to the intended logic. Non-intelligent retry mechanisms. |
| Integrations | Access credentials and permissions not accepted. |
| Client Expectations | Non-acceptance of the generated results. Delivery times not accepted. |
| Deployment | Incorrect application of versioning of a data pipeline. |
From these examples, we can derive two important truths that you should keep in mind:
1️⃣ Many failures can be resolved through good practices and self-management.
2️⃣ Some failures are not entirely within your control because they are dependent on external factors.
Topics like self-healing and resilience by design are technical approaches that help implement the first truth, and they largely depend on the internal data team. The second truth, however, is always a matter of management between vendors to ensure everything is in order according to the agreed-upon terms.
Types of failures and who resolves them
Failures can be categorized based on the most significant effect they have directly on the data pipeline. Although there are many ways to classify them, here I share a list that helps me with this:
| Categories | Behavior | Effect |
|---|---|---|
| Hard failure | The data pipeline fails to complete its execution due to an internal crashing. | The data is not delivered or generated |
| Silent failure | The data pipeline works well but there are errors in the data. | The data is delivered, but it is incorrect. |
| Partial failure | Part of the data pipeline works, but another part doesn't. | Some data is generated and delivered, but other data is not generated or delivered. |
| Performance failure | A data pipeline is experiencing performance issues. | The data is not delivered in a timely manner. |
| Cascading failure | The data pipeline has failures because it depends on external components to function properly. | The data is not delivered or generated, or it is incorrect. |
A data pipeline can have one or more failure categories in a single incident. For example, data pipeline B executed successfully but generated incorrect data. Upon investigation, it was found that it depended on data pipeline A, which produced incorrect data. In this failure scenario, we can observe both a cascading failure and a silent failure.
To resolve a failure, the people involved in its creation are usually part of the process: the data team, providers, and business representatives, also known as business clients. In one way or another, they all participate in the creation and operation of a data pipeline, so they can contribute and help understand the situation in order to find a solution when necessary.
Each involved team has a specific role:
- The business representative/client verifies that everything is in order, with the results as expected and meeting the requirements requested by the data team.
- The data team reviews the solutions to ensure they function correctly and meet the requirements requested by the business.
- The vendor guarantees that the delivered devices, on which the data pipeline relies, function correctly for the data team and/or business.
The efficiency of a data team depends on their mastery of less common data management topics—such as metadata, governance, and data quality—as key areas for quickly understanding, as much as possible, the situation in order to technically manage the mechanisms and environment needed to provide a solution to data pipeline failures. For example, by understanding what operational metadata the data pipeline generated and the results of data quality control mechanisms, we can identify which actors contributed to the failure and how they are responsible for it.
🏗️ The data team must master data management in order to efficiently guide and support the handling of data pipeline failures.
Team coordination and alignment around artifacts, the implementation of improvements and best practices in solutions, as well as planning and addressing potential incidents and scenarios for the data pipeline, are elements that, when absent from the data pipeline lifecycle, can lead to failures due to the lack of prevention of possible situations.
Conclusion
The perception of the client and the company when a failure occurs can affect trust in your data pipeline; therefore, knowing who, when, and why it failed helps provide confidence through the data team’s full understanding and control of the situation.
Regardless of the speed, veracity, variety, or value of the data your data pipeline processes, there will always be failures due to some issue that occurs in the pipeline. If I had to add a fundamental truth to remember, it would be the following:
✨ There will always be unexpected failures in your data pipeline; think about and manage expected and/or planned failures. ✨
As a data engineer, one must be prepared to address this truth in a pragmatic way within the solutions they implement.
In the next lesson, we will talk about resilience and how to incorporate it into data pipelines.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article