Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit b3e70ee

Browse files
committedMar 3, 2022
Final draft
1 parent c4f18c2 commit b3e70ee

File tree

3 files changed

+45
-40
lines changed

3 files changed

+45
-40
lines changed
 

‎articles/sentinel/configure-data-transformation.md

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,12 +28,33 @@ Before you start configuring DCRs for data transformation:
2828

2929
- **Verify data connector support**. Make sure that your data connectors are supported for data transformation.
3030

31-
In our [data connector reference](data-connectors-reference.md) article, check the section for your data connector to understand the types of DCRs and data transformation supported.
31+
In our [data connector reference](data-connectors-reference.md) article, check the section for your data connector to understand which types of DCRs are supported. Continue in this article to understand how the DCR type you select affects the rest of the ingestion and transformation process.
32+
33+
## Determine your requirements
34+
35+
| If you are ingesting | Ingestion-time transformation is... | Use this DCR type |
36+
| -------------------- | ---------------------------- | ----------------- |
37+
| **Custom data** through <br>the **DCR-based API** | <li>Required<li>Included in the DCR that defines the data model | Standard DCR |
38+
| **Built-in data types** <br>(Syslog, CommonSecurityLog, WindowsEvent, SecurityEvent) <br>using the **Azure Monitor Agent (AMA)** | <li>Optional<li>If desired, included in the DCR that defines the AMA configuration | Standard DCR |
39+
| **Built-in data types** <br>(Syslog, CommonSecurityLog, WindowsEvent, SecurityEvent) <br>using the legacy **Log Analytics Agent (MMA)** | <li>Optional<li>If desired, added to the DCR attached to the Workspace where this data is being ingested | Workspace transformation DCR |
40+
| **Built-in data types** <br>from most other sources | <li>Optional<li>If desired, added to the DCR attached to the Workspace where this data is being ingested | Workspace transformation DCR |
41+
| | |
42+
43+
3244

3345
## Configure your data transformation
3446

3547
Use the following procedures from the Log Analytics and Azure Monitor documentation to configure your data transformation DCRs:
3648

49+
[Direct ingestion through the DCR-based Custom Logs API](../azure-monitor/logs/custom-logs-overview.md):
50+
- Walk through a tutorial for [ingesting custom logs using the Azure portal](../azure-monitor/logs/tutorial-custom-logs.md).
51+
- Walk through a tutorial for [ingesting custom logs using Azure Resource Manager (ARM) templates and REST API](../azure-monitor/logs/tutorial-custom-logs-api.md).
52+
53+
[Ingestion-time data transformation](../azure-monitor/logs/ingestion-time-transformations.md):
54+
- Walk through a tutorial for [configuring ingestion-time transformation using the Azure portal](../azure-monitor/logs/tutorial-ingestion-time-transformations.md).
55+
- Walk through a tutorial for [configuring ingestion-time transformation using Azure Resource Manager (ARM) templates and REST API](../azure-monitor/logs/tutorial-ingestion-time-transformations-api.md).-
56+
57+
[More on data collection rules](../azure-monitor/essentials/data-collection-rule-overview.md):
3758
- [Structure of a data collection rule in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-structure.md)
3859
- [Data collection rule transformations in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-transformations.md)
3960

@@ -51,15 +72,15 @@ Use one of the following methods:
5172

5273
After you've verified that your data is properly ingested to the new table, you can delete the legacy table, as well as your legacy, custom data connector.
5374

54-
- Continue using the custom table created by your custom data connector. You might use this option if you have a lot of custom security content created for your existing table. In such cases, consult [AMA migration for Microsoft Sentinel](ama-migrate.md).
75+
- Continue using the custom table created by your custom data connector. You might use this option if you have a lot of custom security content created for your existing table. In such cases, see [Migrate from Data Collector API and custom fields-enabled tables to DCR-based custom logs](../azure-monitor/logs/custom-logs-migrate.md) in the Azure Monitor documentation.
5576

5677
## Next steps
5778

5879
For more information about data transformation and DCRs, see:
5980

6081
- [Custom data ingestion and transformation in Microsoft Sentinel (preview)](data-transformation.md)
6182
- [Ingestion-time transformations in Azure Monitor Logs (preview)](../azure-monitor/logs/ingestion-time-transformations.md)
62-
- [Data collection rule transformations in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-transformations.md)
6383
- [Custom logs API in Azure Monitor Logs (Preview)](../azure-monitor/logs/custom-logs-overview.md)
84+
- [Data collection rule transformations in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-transformations.md)
6485
- [Structure of a data collection rule in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-structure.md)
6586
- [Configure data collection for the Azure Monitor agent](../azure-monitor/agents/data-collection-rule-azure-monitor-agent.md)

‎articles/sentinel/data-transformation.md

Lines changed: 21 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@ Log Analytics' custom data ingestion process gives you a high level of control o
1515

1616
Microsoft Sentinel gives you two tools to control this process:
1717

18-
- The [**custom logs API**](../azure-monitor/logs/custom-logs-overview.md) allows you to send custom-format logs from any data source to your Log Analytics workspace, and store those logs either in certain specific standard tables, or in custom-formatted tables that you create. You have full control over the creation of these custom tables, down to specifying the column names and types. You create [**Data collection rules (DCRs)**](../azure-monitor/essentials/data-collection-rule-overview.md) to define and configure these workflows.
18+
- The [**custom logs API**](../azure-monitor/logs/custom-logs-overview.md) allows you to send custom-format logs from any data source to your Log Analytics workspace, and store those logs either in certain specific standard tables, or in custom-formatted tables that you create. You have full control over the creation of these custom tables, down to specifying the column names and types. You create [**Data collection rules (DCRs)**](../azure-monitor/essentials/data-collection-rule-overview.md) to define, configure, and apply transformations to these data flows.
1919

20-
- [**Ingestion-time data transformation**](../azure-monitor/logs/ingestion-time-transformations.md) for both standard and custom logs uses DCRs to apply basic KQL queries. These transformations can filter out irrelevant data, enrich existing data with analytics or external data, or mask sensitive or personal information.
20+
- [**Ingestion-time data transformation**](../azure-monitor/logs/ingestion-time-transformations.md) uses DCRs to apply basic KQL queries to incoming standard logs (and certain types of custom logs) before they're stored in your workspace. These transformations can filter out irrelevant data, enrich existing data with analytics or external data, or mask sensitive or personal information.
2121

2222
These two tools will be explained in more detail below.
2323

@@ -40,19 +40,15 @@ Ingestion-time transformation also lets you improve analytics by enriching your
4040

4141
For example, you could add extra information such as external HR data, an expanded event description, or classifications that depend on the user, location, or activity type.
4242

43-
### Obfuscation
43+
### Masking
4444

4545
Ingestion-time transformations can also be used to mask or remove personal information. For example, you might use data transformation to mask all but the last digits of a social security number or credit card number, or you could replace other types of personal data with nonsense, standard text, or dummy data. Mask your personal information at ingestion time to increase security across your network.
4646

47-
48-
### USE CASE FOR CUSTOM LOGS API?
49-
50-
5147
## Data ingestion flow in Microsoft Sentinel
5248

5349
The following image shows where ingestion-time data transformation enters the data ingestion flow into Microsoft Sentinel.
5450

55-
Microsoft Sentinel data connectors collect data into the Log Analytics workspace, where it's processed using some combination of hardcoded workflows, ingestion-time transformations, and the custom logs API, and then stored in either standard or custom tables.
51+
Microsoft Sentinel collects data into the Log Analytics workspace from multiple sources. Data from built-in data connectors is processed in Log Analytics using some combination of hardcoded workflows and ingestion-time transformations, and data ingested directly into the custom logs API endpoint is , and then stored in either standard or custom tables.
5652

5753
:::image type="content" source="media/data-transformation/data-transformation-architecture.png" alt-text="Diagram of the Microsoft Sentinel data transformation architecture.":::
5854

@@ -62,37 +58,29 @@ In Log Analytics, data collection rules (DCRs) determine the data flow for diffe
6258

6359
Support for DCRs in Microsoft Sentinel includes:
6460

65-
- *Standard DCRs*, currently supported only for AMA-based connectors and the [custom logs workflow](../azure-monitor/logs/custom-logs-overview.md).
61+
- *Standard DCRs*, currently supported only for AMA-based connectors and workflows using the new [custom logs API](../azure-monitor/logs/custom-logs-overview.md).
6662

67-
Each connector or log source workflow gets its own standard DCR.
63+
Each connector or log source workflow can have its own dedicated *standard DCR*, though multiple connectors or sources can share a common *standard DCR* as well.
6864

6965
- *Workspace transformation DCRs*, for workflows that don't currently support standard DCRs.
7066

71-
A single *workspace transformation DCR* serves all the supported workflows in a workspace that aren't served by standard DCRs. A workspace can have only one *workspace transformation DCR*, but that DCR contains separate transformations for each input stream. Also, workspace transformation DCRs are supported only for a [specific set of tables](../azure-monitor/logs/tables-feature-support.md).
67+
A single *workspace transformation DCR* serves all the supported workflows in a workspace that aren't served by standard DCRs. A workspace can have only one *workspace transformation DCR*, but that DCR contains separate transformations for each input stream. Also, *workspace transformation DCR*s are supported only for a [specific set of tables](../azure-monitor/logs/tables-feature-support.md).
7268

73-
Microsoft Sentinel's support for ingestion-time transformation depends on the type of data connector you're using. For more in-depth information on custom logs, ingestion-time transformation, and data collection rules, see the following articles in the Azure Monitor documentation:
74-
75-
- [Ingestion-time transformations in Azure Monitor Logs (preview)](../azure-monitor/logs/ingestion-time-transformations.md)
76-
- [Custom logs API in Azure Monitor Logs (Preview)](../azure-monitor/logs/custom-logs-overview.md)
77-
- [Data collection rules in Azure Monitor](../azure-monitor/essentials/data-collection-rule-overview.md)
78-
- [Data collection rule transformations in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-transformations.md)
79-
- [Structure of a data collection rule in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-structure.md)
80-
- [Configure data collection for the Azure Monitor agent](../azure-monitor/agents/data-collection-rule-azure-monitor-agent.md)
69+
Microsoft Sentinel's support for ingestion-time transformation depends on the type of data connector you're using. For more in-depth information on custom logs, ingestion-time transformation, and data collection rules, see the articles linked in the [Next steps](#next-steps) section at the end of this article.
8170

8271
### DCR support for Microsoft Sentinel data connectors
8372

8473
The following table describes DCR support for Microsoft Sentinel data connector types:
8574

8675
| Data connector type | DCR support |
8776
| ------------------- | ----------- |
88-
| [**AMA standard logs**](connect-azure-windows-microsoft-services.md?tabs=AMA#windows-agent-based-connections), such as: <li>[Windows Security Events via AMA](data-connectors-reference.md#windows-security-events-via-ama)<li>[Windows Forwarded Events](data-connectors-reference.md#windows-forwarded-events-preview)<li>[CEF data](connect-common-event-format.md)<li>[Syslog data](connect-syslog.md) | Standard DCRs |
77+
| **Direct ingestion via [Custom Logs API](../azure-monitor/logs/custom-logs-overview.md)** | Standard DCRs |
78+
| [**AMA standard logs**](connect-azure-windows-microsoft-services.md?tabs=AMA#windows-agent-based-connections), such as: <li>[Windows Security Events via AMA](data-connectors-reference.md#windows-security-events-via-ama)<li>[Windows Forwarded Events](data-connectors-reference.md#windows-forwarded-events-preview)<li>[CEF data](connect-common-event-format.md)<li>[Syslog data](connect-syslog.md) | Standard DCRs |
8979
| [**MMA standard logs**](connect-azure-windows-microsoft-services.md?tabs=LAA#windows-agent-based-connections), such as <li>[Syslog data](connect-syslog.md)<li>[CommonSecurityLog](connect-azure-windows-microsoft-services.md) | Workspace transformation DCRs |
90-
| [**Diagnostic settings-based connections**](connect-azure-windows-microsoft-services.md#diagnostic-settings-based-connections) | Workspace transformation DCRs, based on the [supported output tables](../azure-monitor/logs/tables-feature-support.md) for specific data connectors |
91-
| **Built-in, service-to-service data connectors**, such as:<li>[Microsoft Office 365](connect-azure-windows-microsoft-services.md#api-based-connections)<li>[Azure Active Directory](connect-azure-active-directory.md)<li>[Amazon S3](connect-aws.md) | Workspace transformation DCRs, based on the [supported output tables](../azure-monitor/logs/tables-feature-support.md) for specific data connectors |
92-
| **Custom or [direct API](connect-rest-api-template.md)-based data connectors** | Standard DCRs |
93-
| **Built-in, API-based data connectors**, such as: <li>[Codeless data connectors](create-codeless-connector.md)<li>[Azure Functions-based data connectors](connect-azure-functions-template.md) | Not currently supported |
94-
| **[Logstash](connect-logstash.md)-based data connectors** | Not currently supported |
95-
| | |
80+
| [**Diagnostic settings-based connections**](connect-azure-windows-microsoft-services.md#diagnostic-settings-based-connections) | Workspace transformation DCRs, based on the [supported output tables](../azure-monitor/logs/tables-feature-support.md) for specific data connectors |
81+
| **Built-in, service-to-service data connectors**, such as:<li>[Microsoft Office 365](connect-azure-windows-microsoft-services.md#api-based-connections)<li>[Azure Active Directory](connect-azure-active-directory.md)<li>[Amazon S3](connect-aws.md) | Workspace transformation DCRs, based on the [supported output tables](../azure-monitor/logs/tables-feature-support.md) for specific data connectors |
82+
| **Built-in, API-based data connectors**, such as: <li>[Codeless data connectors](create-codeless-connector.md)<li>[Azure Functions-based data connectors](connect-azure-functions-template.md) | Not currently supported |
83+
| | |
9684

9785
## Data transformation support for custom data connectors
9886

@@ -113,10 +101,6 @@ Ingestion-time data transformation currently has the following known issues for
113101

114102
There can only be one workspace transformation DCR for an entire workspace. Within that DCR, each table can use a separate input stream with its own transformation. However, if you have two different MMA-based data connectors sending data to the *Syslog* table, they will both have to use the same input stream configuration in the DCR.
115103

116-
- *Workspace transformation DCRs* can be created using the Log Analytics portal, but once created, they can be edited *only* using the Log Analytics API or ARM template.
117-
118-
For example, if you create a DCR to transform Syslog data for a specific MMA-based data connector, and later you want to modify it to support another MMA-based data connector, you'll need to use API calls to make that modification.
119-
120104
- The following configurations are supported only via API:
121105

122106
- Standard DCRs for AMA-based connectors like [Windows Security Events](data-connectors-reference.md#windows-security-events-via-ama) and [Windows Forwarded Events](data-connectors-reference.md#windows-forwarded-events-preview).
@@ -129,15 +113,15 @@ Ingestion-time data transformation currently has the following known issues for
129113

130114
## Next steps
131115

132-
For more in-depth information on ingestion-time transformation and data collection rules, see the following articles in the Azure Monitor documentation:
133-
134-
- [Ingestion-time transformations in Azure Monitor Logs (preview)](../azure-monitor/logs/ingestion-time-transformations.md)
135-
- [Data collection rule transformations in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-transformations.md)
136-
- [Custom logs API in Azure Monitor Logs (Preview)](../azure-monitor/logs/custom-logs-overview.md)
137-
- [Structure of a data collection rule in Azure Monitor (preview)](../azure-monitor/essentials/data-collection-rule-structure.md)
138-
- [Configure data collection for the Azure Monitor agent](../azure-monitor/agents/data-collection-rule-azure-monitor-agent.md)
116+
[Get started configuring ingestion-time data transformation in Microsoft Sentinel](configure-data-transformation.md).
139117

140118
Learn more about Microsoft Sentinel data connector types. For more information, see:
141119

142120
- [Microsoft Sentinel data connectors](connect-data-sources.md)
143121
- [Find your Microsoft Sentinel data connector](data-connectors-reference.md)
122+
123+
For more in-depth information on ingestion-time transformation, the Custom Logs API, and data collection rules, see the following articles in the Azure Monitor documentation:
124+
125+
- [Ingestion-time transformations in Azure Monitor Logs (preview)](../azure-monitor/logs/ingestion-time-transformations.md)
126+
- [Custom logs API in Azure Monitor Logs (Preview)](../azure-monitor/logs/custom-logs-overview.md)
127+
- [Data collection rules in Azure Monitor](../azure-monitor/essentials/data-collection-rule-overview.md)
Loading

0 commit comments

Comments
 (0)
Please sign in to comment.