Skip to content

Files

Latest commit

oshezafoshezaf
fixed examples
May 26, 2022
896c077 · May 26, 2022

History

History
335 lines (248 loc) · 34.3 KB

dns-normalization-schema.md

File metadata and controls

335 lines (248 loc) · 34.3 KB
title description author ms.topic ms.date ms.author
The Advanced Security Information Model (ASIM) DNS normalization schema reference (Public preview) | Microsoft Docs
This article describes the Microsoft Sentinel DNS normalization schema.
oshezaf
reference
11/09/2021
ofshezaf

The Advanced Security Information Model (ASIM) DNS normalization schema reference (Public preview)

[!INCLUDE Banner for top of topics]

The DNS information model is used to describe events reported by a DNS server or a DNS security system, and is used by Microsoft Sentinel to enable source-agnostic analytics.

For more information, see Normalization and the Advanced Security Information Model (ASIM).

Important

The DNS normalization schema is currently in PREVIEW. This feature is provided without a service level agreement, and is not recommended for production workloads.

The Azure Preview Supplemental Terms include additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Schema overview

The ASIM DNS schema represents DNS protocol activity. Both DNS servers and devices sending DNS requests to a DNS server log DNS activity. The DNS protocol activity includes DNS queries, DNS server updates, and DNS bulk data transfers. Since the schema represents protocol activity, it's governed by RFCs and officially assigned parameter lists, which are referenced in this article when appropriate. The DNS schema doesn't represent DNS server audit events.

The most important activity reported by DNS servers is a DNS query, for which the EventType field is set to Query.

The most important fields in a DNS event are:

  • DnsQuery, which reports the domain name for which the query was issued.

  • The SrcIpAddr (aliased to IpAddr), which represents the IP address from which the request was generated. DNS servers typically provide the SrcIpAddr field, but DNS clients sometimes don't provide this field and only provide the SrcHostname field.

  • EventResultDetails, which reports whether the request was successful and if not, why.

  • When available, DnsResponseName, which holds the answer provided by the server to the query. ASIM doesn't require parsing the response, and its format varies between sources.

    To use this field in source-agnostic content, search the content with the has or contains operators.

DNS events collected on client device may also include User and Process information.

Guidelines for collecting DNS events

DNS is a unique protocol in that it may cross a large number of computers. Also, since DNS uses UDP, requests and responses are de-coupled and aren't directly related to each other.

The following image shows a simplified DNS request flow, including four segments. A real-world request can be more complex, with more segments involved.

:::image type="content" source="media/normalization/dns-request-flow.png" alt-text="Simplified DNS request flow.":::

Since request and response segments aren't directly connected to each other in the DNS request flow, full logging can result in significant duplication.

The most valuable segment to log is the response to the client. The response provides the domain name queries, the lookup result, and the IP address of the client. While many DNS systems log only this segment, there is value in logging the other parts. For example, a DNS cache poisoning attack often takes advantage of fake responses from an upstream server.

If your data source supports full DNS logging and you've chosen to log multiple segments, adjust your queries to prevent data duplication in Microsoft Sentinel.

For example, you might modify your query with the following normalization:

_Im_Dns | where SrcIpAddr != "127.0.0.1" and EventSubType == "response"

Parsers

For more information about ASIM parsers, see the ASIM parsers overview.

Unifying parsers

To use parsers that unify all ASIM out-of-the-box parsers, and ensure that your analysis runs across all the configured sources, use the _Im_Dns filtering parser or the _ASim_Dns parameter-less parser. You can also use workspace deployed ImDns and ASimDns parsers.

Out-of-the-box, source-specific parsers

For the list of the DNS parsers Microsoft Sentinel provides out-of-the-box refer to the ASIM parsers list

Add your own normalized parsers

When implementing custom parsers for the Dns information model, name your KQL functions in the following format:

  • vimDns<vendor><Product> for parametrized parsers
  • ASimDns<vendor><Product> for regular parsers

Filtering parser parameters

The im and vim* parsers support filtering parameters. While these parsers are optional, they can improve your query performance.

The following filtering parameters are available:

Name Type Description
starttime datetime Filter only DNS queries that ran at or after this time.
endtime datetime Filter only DNS queries that finished running at or before this time.
srcipaddr string Filter only DNS queries from this source IP address.
domain_has_any dynamic Filter only DNS queries where the domain (or query) has any of the listed domain names, including as part of the event domain. The length of the list is limited to 10,000 items.
responsecodename string Filter only DNS queries for which the response code name matches the provided value.
For example: NXDOMAIN
response_has_ipv4 string Filter only DNS queries in which the response field includes the provided IP address or IP address prefix. Use this parameter when you want to filter on a single IP address or prefix.

Results aren't returned for sources that don't provide a response.
response_has_any_prefix dynamic Filter only DNS queries in which the response field includes any of the listed IP addresses or IP address prefixes. Prefixes should end with a ., for example: 10.0..

Use this parameter when you want to filter on a list of IP addresses or prefixes.

Results aren't returned for sources that don't provide a response. The length of the list is limited to 10,000 items.
eventtype string Filter only DNS queries of the specified type. If no value is specified, only lookup queries are returned.

For example, to filter only DNS queries from the last day that failed to resolve the domain name, use:

_Im_Dns (responsecodename = 'NXDOMAIN', starttime = ago(1d), endtime=now())

To filter only DNS queries for a specified list of domain names, use:

let torProxies=dynamic(["tor2web.org", "tor2web.com", "torlink.co"]);
_Im_Dns (domain_has_any = torProxies)

Tip

To pass a literal list to parameters that expect a dynamic value, explicitly use a dynamic literal. For example: dynamic(['192.168.','10.']).

Normalized content

For a full list of analytics rules that use normalized DNS events, see DNS query security content.

Schema details

The DNS information model is aligned with the OSSEM DNS entity schema.

For more information, see the Internet Assigned Numbers Authority (IANA) DNS parameter reference.

Common ASIM fields

Important

Fields common to all schemas are described in detail in the ASIM Common Fields article.

Common fields with specific guidelines

The following list mentions fields that have specific guidelines for DNS events:

Field Class Type Description
EventType Mandatory Enumerated Indicates the operation reported by the record.

For DNS records, this value would be the DNS op code.

Example: lookup
EventSubType Optional Enumerated Either request or response.

For most sources, only the responses are logged, and therefore the value is often response.
EventResultDetails Mandatory Enumerated For DNS events, this field provides the DNS response code.

Note: IANA doesn't define the case for the values, so analytics must normalize the case. If the source provides only a numerical response code and not a response code name, the parser must include a lookup table to enrich with this value.

If this record represents a request and not a response, set to NA.

Example: NXDOMAIN
EventSchemaVersion Mandatory String The version of the schema documented here is 0.1.3.
EventSchema Mandatory String The name of the schema documented here is Dns.
Dvc fields - - For DNS events, device fields refer to the system that reports the DNS event.

All common fields

Fields that appear in the table below are common to all ASIM schemas. Any guideline specified above overrides the general guidelines for the field. For example, a field might be optional in general, but mandatory for a specific schema. For further details on each field, refer to the ASIM Common Fields article.

Class Fields
Mandatory - EventCount
- EventStartTime
- EventEndTime
- EventType
- EventResult
- EventProduct
- EventVendor
- EventSchema
- EventSchemaVersion
- Dvc
Recommended - EventResultDetails
- EventSeverity
- DvcIpAddr
- DvcHostname
- DvcDomain
- DvcDomainType
- DvcFQDN
- DvcId
- DvcIdType
- DvcAction
Optional - EventMessage
- EventSubType
- EventOriginalUid
- EventOriginalType
- EventOriginalSubType
- EventOriginalResultDetails
- EventOriginalSeverity
- EventProductVersion
- EventReportUrl
- DvcMacAddr
- DvcOs
- DvcOsVersion
- DvcOriginalAction
- DvcInterface
- AdditionalFields
- DvcDescription

DNS-specific fields

The fields listed in this section are specific to DNS events, although many are similar to fields in other schemas and therefore follow the same naming convention.

Field Class Type Notes
Src Recommended String A unique identifier of the source device.

This field can alias the SrcDvcId, SrcHostname, or SrcIpAddr fields.

Example: 192.168.12.1
SrcIpAddr Recommended IP Address The IP address of the client that sent the DNS request. For a recursive DNS request, this value would typically be the reporting device, and in most cases set to 127.0.0.1.

Example: 192.168.12.1
SrcPortNumber Optional Integer Source port of the DNS query.

Example: 54312
IpAddr Alias Alias to SrcIpAddr
SrcGeoCountry Optional Country The country associated with the source IP address.

Example: USA
SrcGeoRegion Optional Region The region within a country associated with the source IP address.

Example: Vermont
SrcGeoCity Optional City The city associated with the source IP address.

Example: Burlington
SrcGeoLatitude Optional Latitude The latitude of the geographical coordinate associated with the source IP address.

Example: 44.475833
SrcGeoLongitude Optional Longitude The longitude of the geographical coordinate associated with the source IP address.

Example: 73.211944
SrcRiskLevel Optional Integer The risk level associated with the source. The value should be adjusted to a range of 0 to 100, with 0 for benign and 100 for a high risk.

Example: 90
SrcHostname Recommended String The source device hostname, excluding domain information.

Example: DESKTOP-1282V4D
Hostname Alias Alias to SrcHostname
SrcDomain Recommended String The domain of the source device.

Example: Contoso
SrcDomainType Recommended Enumerated The type of SrcDomain, if known. Possible values include:
- Windows (such as: contoso)
- FQDN (such as: microsoft.com)

Required if SrcDomain is used.
SrcFQDN Optional String The source device hostname, including domain information when available.

Note: This field supports both traditional FQDN format and Windows domain\hostname format. The SrcDomainType field reflects the format used.

Example: Contoso\DESKTOP-1282V4D
SrcDvcId Optional String The ID of the source device as reported in the record.

For example: ac7e9755-8eae-4ffc-8a02-50ed7a2216c3
SrcDvcIdType Optional Enumerated The type of SrcDvcId, if known. Possible values include:
- AzureResourceId
- MDEid

If multiple IDs are available, use the first one from the list, and store the others in the SrcDvcAzureResourceId and SrcDvcMDEid, respectively.

Note: This field is required if SrcDvcId is used.
SrcDeviceType Optional Enumerated The type of the source device. Possible values include:
- Computer
- Mobile Device
- IOT Device
- Other
SrcUserId Optional String A machine-readable, alphanumeric, unique representation of the source user. Format and supported types include:
- SID (Windows): S-1-5-21-1377283216-344919071-3415362939-500
- UID (Linux): 4578
- AADID (Azure Active Directory): 9267d02c-5f76-40a9-a9eb-b686f3ca47aa
- OktaId: 00urjk4znu3BcncfY0h7
- AWSId: 72643944673

Store the ID type in the SrcUserIdType field.

If other IDs are available, we recommend that you normalize the field names to SrcUserSid, SrcUserUid, SrcUserAadId, SrcUserOktaId and UserAwsId, respectively. For more information, see The User entity.

Example: S-1-12
SrcUserIdType Optional Enumerated The type of the ID stored in the SrcUserId field. Supported values include: SID, UIS, AADID, OktaId, and AWSId.
SrcUsername Optional String The Source username, including domain information when available. Use one of the following formats and in the following order of priority:
- Upn/Email: johndow@contoso.com
- Windows: Contoso\johndow
- DN: CN=Jeff Smith,OU=Sales,DC=Fabrikam,DC=COM
- Simple: johndow. Use the Simple form only if domain information is not available.

Store the Username type in the SrcUsernameType field. If other IDs are available, we recommend that you normalize the field names to SrcUserUpn, SrcUserWindows and SrcUserDn.

For more information, see The User entity.

Example: AlbertE
User Alias Alias to SrcUsername
SrcUsernameType Optional Enumerated Specifies the type of the username stored in the SrcUsername field. Supported values are: UPN, Windows, DN, and Simple. For more information, see The User entity.

Example: Windows
SrcUserType Optional Enumerated The type of Actor. Allowed values are:
- Regular
- Machine
- Admin
- System
- Application
- Service Principal
- Other

Note: The value may be provided in the source record using different terms, which should be normalized to these values. Store the original value in the EventOriginalUserType field.
SrcOriginalUserType Optional String The original source user type, if provided by the source.
SrcUserDomain Optional String This field is kept for backward compatibility only. ASIM requires domain information, if available, to be part of the SrcUsername field.
SrcProcessName Optional String The file name of the process that initiated the DNS request. This name is typically considered to be the process name.

Example: C:\Windows\explorer.exe
Process Alias Alias to the SrcProcessName

Example: C:\Windows\System32\rundll32.exe
SrcProcessId Optional String The process ID (PID) of the process that initiated the DNS request.

Example: 48610176

Note: The type is defined as string to support varying systems, but on Windows and Linux this value must be numeric.

If you are using a Windows or Linux machine and used a different type, make sure to convert the values. For example, if you used a hexadecimal value, convert it to a decimal value.
SrcProcessGuid Optional String A generated unique identifier (GUID) of the process that initiated the DNS request.

Example: EF3BD0BD-2B74-60C5-AF5C-010000001E00
Dst Recommended String A unique identifier of the server that received the DNS request.

This field may alias the DstDvcId, DstHostname, or DstIpAddr fields.

Example: 192.168.12.1
DstIpAddr Optional IP Address The IP address of the server that received the DNS request. For a regular DNS request, this value would typically be the reporting device, and in most cases set to 127.0.0.1.

Example: 127.0.0.1
DstGeoCountry Optional Country The country associated with the destination IP address. For more information, see Logical types.

Example: USA
DstGeoRegion Optional Region The region, or state, within a country associated with the destination IP address. For more information, see Logical types.

Example: Vermont
DstGeoCity Optional City The city associated with the destination IP address. For more information, see Logical types.

Example: Burlington
DstGeoLatitude Optional Latitude The latitude of the geographical coordinate associated with the destination IP address. For more information, see Logical types.

Example: 44.475833
DstGeoLongitude Optional Longitude The longitude of the geographical coordinate associated with the destination IP address. For more information, see Logical types.

Example: 73.211944
DstcRiskLevel Optional Integer The risk level associated with the destination. The value should be adjusted to a range of 0 to 100, which 0 being benign and 100 being a high risk.

Example: 90
DstPortNumber Optional Integer Destination Port number.

Example: 53
DstHostname Optional String The destination device hostname, excluding domain information. If no device name is available, store the relevant IP address in this field.

Example: DESKTOP-1282V4D

Note: This value is mandatory if DstIpAddr is specified.
DstDomain Optional String The domain of the destination device.

Example: Contoso
DstDomainType Optional Enumerated The type of DstDomain, if known. Possible values include:
- Windows (contoso\mypc)
- FQDN (docs.microsoft.com)

Required if DstDomain is used.
DstFQDN Optional String The destination device hostname, including domain information when available.

Example: Contoso\DESKTOP-1282V4D

Note: This field supports both traditional FQDN format and Windows domain\hostname format. The DstDomainType reflects the format used.
DstDvcId Optional String The ID of the destination device as reported in the record.

Example: ac7e9755-8eae-4ffc-8a02-50ed7a2216c3
DstDvcIdType Optional Enumerated The type of DstDvcId, if known. Possible values include:
- AzureResourceId
- MDEidIf

If multiple IDs are available, use the first one from the list above, and store the others in the DstDvcAzureResourceId or DstDvcMDEid fields, respectively.

Required if DstDeviceId is used.
DstDeviceType Optional Enumerated The type of the destination device. Possible values include:
- Computer
- Mobile Device
- IOT Device
- Other
DnsQuery Mandatory FQDN The domain that the request tries to resolve.

Note: Some sources send the query in different formats. For example, in the DNS protocol itself, the query includes a dot (.)at the end, which must be removed.

While the DNS protocol allows for multiple queries in a single request, this scenario is rare, if it's found at all. If the request has multiple queries, store the first one in this field, and then and optionally keep the rest in the AdditionalFields field.

Example: www.malicious.com
Domain Alias Alias to DnsQuery.
DnsQueryType Optional Integer The DNS Resource Record Type codes.

Example: 28
DnsQueryTypeName Recommended Enumerated The DNS Resource Record Type names.

Note: IANA doesn't define the case for the values, so analytics must normalize the case as needed. If the source provides only a numerical query type code and not a query type name, the parser must include a lookup table to enrich with this value.

Example: AAAA
DnsResponseName Optional String The content of the response, as included in the record.

The DNS response data is inconsistent across reporting devices, is complex to parse, and has less value for source-agnostic analytics. Therefore the information model doesn't require parsing and normalization, and Microsoft Sentinel uses an auxiliary function to provide response information. For more information, see Handling DNS response.
DnsResponseCodeName Alias Alias to EventResultDetails
DnsResponseCode Optional Integer The DNS numerical response code.

Example: 3
TransactionIdHex Recommended String The DNS query unique ID as assigned by the DNS client, in hexadecimal format. Note that this value is part of the DNS protocol and different from DnsSessionId, the network layer session ID, typically assigned by the reporting device.
NetworkProtocol Optional Enumerated The transport protocol used by the network resolution event. The value can be UDP or TCP, and is most commonly set to UDP for DNS.

Example: UDP
DnsQueryClass Optional Integer The DNS class ID.

In practice, only the IN class (ID 1) is used, and therefore this field is less valuable.
DnsQueryClassName Optional String The DNS class name.

In practice, only the IN class (ID 1) is used, and therefore this field is less valuable.

Example: IN
DnsFlags Optional List of strings The flags field, as provided by the reporting device. If flag information is provided in multiple fields, concatenate them with comma as a separator.

Since DNS flags are complex to parse and are less often used by analytics, parsing and normalization aren't required. Microsoft Sentinel can use an auxiliary function to provide flags information. For more information, see Handling DNS response.

Example: ["DR"]
UrlCategory Optional String A DNS event source may also look up the category of the requested Domains. The field is called UrlCategory to align with the Microsoft Sentinel network schema.

DomainCategory is added as an alias that's fitting to DNS.

Example: Educational \\ Phishing
DomainCategory Optional Alias Alias to UrlCategory.
ThreatCategory Optional String If a DNS event source also provides DNS security, it may also evaluate the DNS event. For example, it can search for the IP address or domain in a threat intelligence database, and assign the domain or IP address with a Threat Category.
DnsNetworkDuration Optional Integer The amount of time, in milliseconds, for the completion of DNS request.

Example: 1500
Duration Alias Alias to DnsNetworkDuration
DnsFlagsAuthenticated Optional Boolean The DNS AD flag, which is related to DNSSEC, indicates in a response that all data included in the answer and authority sections of the response have been verified by the server according to the policies of that server. see RFC 3655 Section 6.1 for more information.
DnsFlagsAuthoritative Optional Boolean The DNS AA flag indicates whether the response from the server was authoritative
DnsFlagsCheckingDisabled Optional Boolean The DNS CD flag, which is related to DNSSEC, indicates in a query that non-verified data is acceptable to the system sending the query. see RFC 3655 Section 6.1 for more information.
DnsFlagsRecursionAvailable Optional Boolean The DNS RA flag indicates in a response that that server supports recursive queries.
DnsFlagsRecursionDesired Optional Boolean The DNS RD flag indicates in a request that that client would like the server to use recursive queries.
DnsFlagsTruncated Optional Boolean The DNS TC flag indicates that a response was truncated as it exceeded the maximum response size.
DnsFlagsZ Optional Boolean The DNS Z flag is a deprecated DNS flag, which might be reported by older DNS systems.
DnsSessionId Optional string The DNS session identifier as reported by the reporting device. Note that this value is different from TransactionIdHex, the DNS query unique ID as assigned by the DNS client.

Example: EB4BFA28-2EAD-4EF7-BC8A-51DF4FDF5B55
SessionId Alias String Alias to DnsSessionId

Deprecated aliases

The following fields are aliases that are maintained for backwards compatibility. They were removed from the schema on December 31st, 2021.

  • Query (alias to DnsQuery)
  • QueryType (alias to DnsQueryType)
  • QueryTypeName (alias to DnsQueryTypeName)
  • ResponseName (alias to DnsReasponseName)
  • ResponseCodeName (alias to DnsResponseCodeName)
  • ResponseCode (alias to DnsResponseCode)
  • QueryClass (alias to DnsQueryClass)
  • QueryClassName (alias to DnsQueryClassName)
  • Flags (alias to DnsFlags)

Schema updates

The changes in version 0.1.2 of the schema are:

  • Added the field EventSchema.
  • Added dedicated flag field which augments the combined Flags field: DnsFlagsAuthoritative, DnsFlagsCheckingDisabled, DnsFlagsRecursionAvailable, DnsFlagsRecursionDesired, DnsFlagsTruncated, and DnsFlagsZ.

The changes in version 0.1.3 of the schema are:

  • The schema now explicitly documents Src*, Dst*, Process* and User* fields.
  • Added more Dvc* fields to match the latest common fields definition.
  • Added Src and Dst as aliases to a leading identifier for the source and destination systems.
  • Added optional DnsNetworkDuration and Duration, an alias to it.
  • Added optional Geo Location and Risk Level fields.

Source-specific discrepancies

The goal of normalizing is to ensure that all sources provide consistent telemetry. A source that doesn't provide the required telemetry, such as mandatory schema fields, cannot be normalized. However, sources that typically provide all required telemetry, even if there are some discrepancies, can be normalized. Discrepancies may affect the completeness of query results.

The following table lists known discrepancies:

Source Discrepancies
Microsoft DNS Server Collected using the DNS connector and the Log Analytics Agent The connector doesn't provide the mandatory DnsQuery field for original event ID 264 (Response to a dynamic update). The data is available at the source, but not forwarded by the connector.
Corelight Zeek Corelight Zeek may not provide the mandatory DnsQuery field. We have observed such behavior in certain cases in which the DNS response code name is NXDOMAIN.

Handling DNS response

In most cases, logged DNS events don't include response information, which may be large and detailed. If your record includes more response information, store it in the ResponseName field as it appears in the record.

You can also provide an extra KQL function called _imDNS<vendor>Response_, which takes the unparsed response as input and returns dynamic value with the following structure:

[
    {
        "part": "answer"
        "query": "yahoo.com."
        "TTL": 1782
        "Class": "IN"
        "Type": "A"
        "Response": "74.6.231.21"
    }
    {
        "part": "authority"
        "query": "yahoo.com."
        "TTL": 113066
        "Class": "IN"
        "Type": "NS"
        "Response": "ns5.yahoo.com"
    }
    ...
]

The fields in each dictionary in the dynamic value correspond to the fields in each DNS response. The part entry should include either answer, authority, or additional to reflect the part in the response that the dictionary belongs to.

Tip

To ensure optimal performance, call the imDNS<vendor>Response function only when needed, and only after an initial filtering to ensure better performance.

Handling DNS flags

Parsing and normalization aren't required for flag data. Instead, store the flag data provided by the reporting device in the Flags field. If determining the value of individual flags is straight forward, you can also use the dedicated flags fields.

You can also provide an extra KQL function called _imDNS<vendor>Flags_, which takes the unparsed response, or dedicated flag fields, as input and returns a dynamic list, with Boolean values that represent each flag in the following order:

  • Authenticated (AD)
  • Authoritative (AA)
  • Checking Disabled (CD)
  • Recursion Available (RA)
  • Recursion Desired (RD)
  • Truncated (TC)
  • Z

Next steps

For more information, see: