| title | description | ms.author | author | ms.service | ms.subservice | ms.topic | ms.custom | ms.date | ||
|---|---|---|---|---|---|---|---|---|---|---|
Push Data Factory lineage data to Microsoft Purview |
Learn about how to push Data Factory lineage data to Microsoft Purview |
jianleishen |
jianleishen |
data-factory |
tutorials |
tutorial |
|
08/10/2021 |
[!INCLUDEappliesto-adf-xxx-md]
In this tutorial, you'll use the Data Factory user interface (UI) to create a pipeline that run activities and report lineage data to Microsoft Purview account. Then you can view all the lineage information in your Microsoft Purview account.
Currently, lineage is supported for Copy, Data Flow, and Execute SSIS activities. Learn more details on the supported capabilities from Supported Azure Data Factory activities.
- Azure subscription. If you don't have an Azure subscription, create a free Azure account before you begin.
- Azure Data Factory. If you don't have an Azure Data Factory, see Create an Azure Data Factory.
- Microsoft Purview account. The Microsoft Purview account captures all lineage data generated by data factory. If you don't have a Microsoft Purview account, see Create a Microsoft Purview.
You can establish the connection between Data Factory and Microsoft Purview account by following the steps in Connect Data Factory to Microsoft Purview.
You can create pipelines, Copy activities and Dataflow activities in Data Factory. You don't need any additional configuration for lineage data capture. The lineage data will automatically be captured during the activities execution.
:::image type="content" source="./media/data-factory-purview/adf-activities-for-lineage.png" alt-text="Screenshot of Copy and Dataflow activity.":::
:::image type="content" source="./media/data-factory-purview/ssis-activities-for-lineage.png" alt-text="Screenshot of Execute SSIS Package activity.":::
Learn more about how to create Copy, Data Flow and Execute SSIS activities from Copy data from Azure Blob storage to a database in Azure SQL Database by using Azure Data Factory, Transform data using mapping data flows and Run SSIS Packages in Azure.
After you run the pipeline, in the pipeline monitoring view, you can check the lineage reporting status by clicking the following Lineage status button. The same information is also available in the activity output JSON -> reportLineageToPurvew section.
:::image type="content" source="./media/data-factory-purview/monitor-lineage-reporting-status.png" alt-text="Monitor the lineage reporting status in pipeline monitoring view.":::
On Microsoft Purview UI, you can browse assets and choose type "Azure Data Factory". You can also search the Data Catalog using keywords.
:::image type="content" source="./media/data-factory-purview/view-dataset.png" alt-text="Screenshot of browing assets in Microsoft Purview." lightbox="./media/data-factory-purview/view-dataset.png":::
On the activity asset, click the Lineage tab, you can see all the lineage information.
-
Copy activity:
:::image type="content" source="./media/data-factory-purview/copy-lineage.png" alt-text="Screenshot of the Copy activity lineage in Microsoft Purview." lightbox="./media/data-factory-purview/copy-lineage.png":::
-
Data Flow activity:
:::image type="content" source="./media/data-factory-purview/dataflow-lineage.png" alt-text="Screenshot of the Data Flow lineage in Microsoft Purview." lightbox="./media/data-factory-purview/dataflow-lineage.png":::
[!NOTE] For the lineage of Dataflow activity, we only support source and sink. The lineage for Dataflow transformation is not supported yet.
-
Execute SSIS Package activity:
:::image type="content" source="./media/data-factory-purview/ssis-lineage.png" alt-text="Screenshot of the Execute SSIS lineage in Microsoft Purview." lightbox="./media/data-factory-purview/ssis-lineage.png":::
[!NOTE] For the lineage of Execute SSIS Package activity, we only support source and destination. The lineage for transformation is not supported yet.