Data Factory Pipeline Tasks

Introduction

A Data Factory pipeline is a logical grouping of activities that together perform a task. For example, a pipeline could contain a set of activities to perform a specific task, and then kick off a mapping data flow to analyze the same. The task that is performed can be a success or failure depending on the actions involved in the Data Factory run.

The significance of Data Factory Pipeline automated tasks is that a failed Data Factory run can be rerun at some point in time which might result in a successful run.

Thus, Data Factory Pipeline automated tasks aid in the process of rerunning the failed runs.

Rerun the failed pipeline runs

The Automated task can be configured to rerun only the runs that are required by using the following filters:

Failure Occurrence: The Data Factory Pipeline runs failed in the past x hours (specified in the automated task creation) will be chosen for rerun.

Users are also privileged to Include the previous reruns, Rerun the ignored runs of previous run, and can opt to Run task once immediately after saving, which runs the created automated task on the go after being saved.

The below illustration is an example of rerunning pipeline runs failed in last 12 hours:

Notifications

The final section of the Data Factory Pipeline Automated task configuration blade includes the notification configuration section, where users can configure the desired Notification channels and email address(es) to receive notifications individually or for a group.

Multiple email addresses can also be provided so that a group of users can get notifications and stay connected.

By checking one or both the options under Advanced settings, users can authorize the task notifications to be received in specific situations:

• Send alerts only when the automated task fails - By enabling this checkbox, the user will receive notifications only if the task fails.

• Send alerts only when the completed count is greater than 0 - By enabling this checkbox, the user will receive notifications only when the task completion count is greater than 0.

Upon checking both checkboxes, the user will be notified if the task fails (or) the completion count exceeds 0.