| title | description | ms.service | ms.subservice | author | ms.author | ms.reviewer | ms.topic | ms.date |
|---|---|---|---|---|---|---|---|---|
Pipeline failure and error message |
Understand how pipeline failure status and error message are determined |
data-factory |
orchestration |
chez-charlie |
chez |
jburchel |
tutorial |
11/08/2021 |
[!INCLUDEappliesto-adf-asa-md]
Azure Data Factory and Synapse Pipeline orchestration allows conditional logic and enables user to take different based upon outcomes of a previous activity. Using different paths allow users to build robust pipelines and incorporates error handling in ETL/ELT logic. In total, we allow four conditional paths,
- Upon Success (default pass)
- Upon Failure
- Upon Completion
- Upon Skip
:::image type="content" source="media/tutorial-pipeline-failure-error-handling/pipeline-error-1-four-branches.png" alt-text="Screenshot showing the four branches out of an activity.":::
In this approach, customer defines the business logic, and only defines the Upon Failure path to catch any error from previous activity. This approach renders pipeline succeeds, if Upon Failure path succeeds.
:::image type="content" source="media/tutorial-pipeline-failure-error-handling/pipeline-error-2-try-catch-definition.png" alt-text="Screenshot showing definition and outcome of a try catch block.":::
In this approach, customer defines the business logic, and defines both the Upon Failure and Upon Success paths. This approach renders pipeline fails, even if Upon Failure path succeeds.
:::image type="content" source="media/tutorial-pipeline-failure-error-handling/pipeline-error-3-do-if-else-definition.png" alt-text="Screenshot showing definition and outcome of do if else block.":::
In this approach, customer defines the business logic, and defines both the Upon Failure path, and Upon Success path, with a dummy Upon Skipped activity attached. This approach renders pipeline succeeds, if Upon Failure path succeeds.
:::image type="content" source="media/tutorial-pipeline-failure-error-handling/pipeline-error-4-do-if-skip-else-definition.png" alt-text="Screenshot showing definition and outcome of do if skip else block.":::
| Approach | Defines | When activity succeeds, overall pipeline shows | When activity fails, overall pipeline shows |
|---|---|---|---|
| Try-Catch | Only Upon Failure path | Success | Success |
| Do-If-Else | Upon Failure path + Upon Success paths | Success | Failure |
| Do-If-Skip-Else | Upon Failure path + Upon Success path (with a Dummy Upon Skip at the end) | Success | Success |
Different error handling mechanisms will lead to different status for the pipeline: while some pipelines fail, others succeed. We determine pipeline success and failures as follows:
- Evaluate outcome for all leaves activities. If a leaf activity was skipped, we evaluate its parent activity instead
- Pipeline result is success if and only if all nodes evaluated succeed
Assuming Upon Failure activity and Dummy Upon Failure activity succeed,
-
In Try-Catch approach,
- When previous activity succeeds: node Upon Failure is skipped and its parent node succeeds; overall pipeline succeeds
- When previous activity fails: node Upon Failure is enacted; overall pipeline succeeds
-
In Do-If-Else approach,
- When previous activity succeeds: node Upon Success succeeds and node Upon Failure is skipped (and its parent node succeeds); overall pipeline succeeds
- When previous activity fails: node Upon Success is skipped and its parent node failed; overall pipeline fails
-
In Do-If-Skip-Else approach,
- When previous activity succeeds: node Dummy Upon Skip is skipped and its parent node Upon Success succeeds; the other node activity, Upon Failure, is skipped and its parent node succeeds; overall pipeline succeeds
- When previous activity fails: node Upon Failure succeeds and Dummy Upon Skip succeeds; overall pipeline succeeds