Skip to content

Files

Latest commit

v-hearyav-hearya
Broken link fixed
May 27, 2022
06e9f29 · May 27, 2022

History

History
148 lines (115 loc) · 8.56 KB

merge.md

File metadata and controls

148 lines (115 loc) · 8.56 KB
title description author ms.author ms.service ms.custom ms.topic ms.reviewer ms.date
Merge partitions in Azure Cosmos DB (preview)
Learn more about the merge partitions capability in Azure Cosmos DB
seesharprun
sidandrews
cosmos-db
event-tier1-build-2022
conceptual
dech
05/09/2022

Merge partitions in Azure Cosmos DB (preview)

[!INCLUDEappliesto-sql-mongodb-api]

Merging partitions in Azure Cosmos DB (preview) allows you to reduce the number of physical partitions used for your container. With merge, containers that are fragmented in throughput (have low RU/s per partition) or storage (have low storage per partition) can have their physical partitions reworked. If a container's throughput has been scaled up and needs to be scaled back down, merge can help resolve throughput fragmentation issues. For the same amount of provisioned RU/s, having fewer physical partitions means each physical partition gets more of the overall RU/s. Minimizing partitions reduces the chance of rate limiting if a large quantity of data is removed from a container. Merge can help clear out unused or empty partitions, effectively resolving storage fragmentation problems.

Getting started

To get started using merge, enroll in the preview by submitting a request for the Azure Cosmos DB Partition Merge feature via the Preview Features page in your Azure Subscription overview page.

  • Before submitting your request, verify that your Azure Cosmos DB account(s) meet all the preview eligibility criteria.
  • The Azure Cosmos DB team will review your request and contact you via email to confirm which account(s) in the subscription you want to enroll in the preview.

Merging physical partitions

In PowerShell, when the flag -WhatIf is passed in, Azure Cosmos DB will run a simulation and return the expected result of the merge, but won't run the merge itself. When the flag isn't passed in, the merge will execute against the resource. When finished, the command will output the current amount of storage in KB per physical partition post-merge.

Tip

Before running a merge, it's recommended to set your provisioned RU/s (either manual RU/s or autoscale max RU/s) as close as possible to your desired steady state RU/s post-merge, to help ensure the system calculates an efficient partition layout.

PowerShell

// Add the preview extension
Install-Module -Name Az.CosmosDB -AllowPrerelease -Force

// SQL API
Invoke-AzCosmosDBSqlContainerMerge `
    -ResourceGroupName "<resource-group-name>" `
    -AccountName "<cosmos-account-name>" `
    -DatabaseName "<cosmos-database-name>" `
    -Name "<cosmos-container-name>" `
    -WhatIf

// API for MongoDB
Invoke-AzCosmosDBMongoDBCollectionMerge `
    -ResourceGroupName "<resource-group-name>" `
    -AccountName "<cosmos-account-name>" `
    -DatabaseName "<cosmos-database-name>" `
    -Name "<cosmos-collection-name>" `
    -WhatIf

Azure CLI

// Add the preview extension
az extension add --name cosmosdb-preview

// SQL API
az cosmosdb sql container merge \
    --resource-group '<resource-group-name>' \
    --account-name '<cosmos-account-name>' \
    --database-name '<cosmos-database-name>' \
    --name '<cosmos-container-name>'

// API for MongoDB
az cosmosdb mongodb collection merge \
    --resource-group '<resource-group-name>' \
    --account-name '<cosmos-account-name>' \
    --database-name '<cosmos-database-name>' \
    --name '<cosmos-collection-name>'

Monitor merge operations

Partition merge is a long-running operation and there's no SLA on how long it takes to complete. The time depends on the amount of data in the container and the number of physical partitions. It's recommended to allow at least 5-6 hours for merge to complete.

While partition merge is running on your container, it isn't possible to change the throughput or any container settings (TTL, indexing policy, unique keys, etc.). Wait until the merge operation completes before changing your container settings.

You can track whether merge is still in progress by checking the Activity Log and filtering for the events Merge the physical partitions of a MongoDB collection or Merge the physical partitions of a SQL container.

Limitations

Preview eligibility criteria

To enroll in the preview, your Cosmos account must meet all the following criteria:

  • Your Cosmos account uses SQL API or API for MongoDB with version >=3.6.
  • Your Cosmos account is using provisioned throughput (manual or autoscale). Merge doesn't apply to serverless accounts.
    • Currently, merge isn't supported for shared throughput databases. You may enroll an account that has both shared throughput databases and containers with dedicated throughput (manual or autoscale).
    • However, only the containers with dedicated throughput will be able to be merged.
  • Your Cosmos account is a single-write region account (merge isn't currently supported for multi-region write accounts).
  • Your Cosmos account doesn't use any of the following features:
  • Your Cosmos account uses bounded staleness, session, consistent prefix, or eventual consistency (merge isn't currently supported for strong consistency).
  • If you're using SQL API, your application must use the Azure Cosmos DB .NET V3 SDK, version 3.27.0 or higher. When merge preview enabled on your account, all requests sent from non .NET SDKs or older .NET SDK versions won't be accepted.
    • There are no SDK or driver requirements to use the feature with API for MongoDB.
  • Your Cosmos account doesn't use any currently unsupported connectors:
    • Azure Data Factory
    • Azure Stream Analytics
    • Logic Apps
    • Azure Functions
    • Azure Search

Account resources and configuration

  • Merge is only available for SQL API and API for MongoDB accounts. For API for MongoDB accounts, the MongoDB account version must be 3.6 or greater.
  • Merge is only available for single-region write accounts. Multi-region write account support isn't available.
  • Accounts using merge functionality can't also use these features (if these features are added to a merge enabled account, resources in the account will no longer be able to be merged):
  • Containers using merge functionality must have their throughput provisioned at the container level. Database-shared throughput support isn't available.
  • Merge is only available for accounts using bounded staleness, session, consistent prefix, or eventual consistency. It isn't currently supported for strong consistency.
  • After a container has been merged, it isn't possible to read the change feed with start time. Support for this feature is planned for the future.

SDK requirements (SQL API only)

Accounts with the merge feature enabled are supported only when you use the latest version of the .NET v3 SDK. When the feature is enabled on your account (regardless of whether you run the merge), you must only use the supported SDK using the account. Requests sent from other SDKs or earlier versions won't be accepted. As long as you're using the supported SDK, your application can continue to run while a merge is ongoing.

Find the latest version of the supported SDK:

SDK Supported versions Package manager link
.NET SDK v3 >= 3.27.0 https://www.nuget.org/packages/Microsoft.Azure.Cosmos/

Support for other SDKs is planned for the future.

Tip

You should ensure that your application has been updated to use a compatible SDK version prior to enrolling in the preview. If you're using the legacy .NET V2 SDK, follow the .NET SDK v3 migration guide.

Unsupported connectors

If you enroll in the preview, the following connectors will fail.

  • Azure Data Factory
  • Azure Stream Analytics
  • Logic Apps
  • Azure Functions
  • Azure Search

Support for these connectors is planned for the future.

Next steps