| title | titleSuffix | description | services | ms.service | ms.subservice | ms.author | author | ms.date | ms.topic | ms.custom |
|---|---|---|---|---|---|---|---|---|---|---|
Create workspaces with Azure CLI |
Azure Machine Learning |
Learn how to use the Azure CLI extension for machine learning to create a new Azure Machine Learning workspace. |
machine-learning |
machine-learning |
core |
larryfr |
Blackmist |
01/05/2022 |
how-to |
devx-track-azurecli, cliv2, cliv1, event-tier1-build-2022 |
In this article, you learn how to create and manage Azure Machine Learning workspaces using the Azure CLI. The Azure CLI provides commands for managing Azure resources and is designed to get you working quickly with Azure, with an emphasis on automation. The machine learning extension to the CLI provides commands for working with Azure Machine Learning resources.
Note
Examples in this article refer to both CLI v1 and CLI v2 versions. If no version is specified for a command, it will work with either the v1 or CLI v2. The machine learning CLI v2 is currently in public preview. This preview version is provided without a service-level agreement, and it's not recommended for production workloads.
-
An Azure subscription. If you do not have one, try the free or paid version of Azure Machine Learning.
-
To use the CLI commands in this document from your local environment, you need the Azure CLI.
If you use the Azure Cloud Shell, the CLI is accessed through the browser and lives in the cloud.
[!INCLUDE register-namespace]
[!INCLUDE application-insight]
Some of the Azure CLI commands communicate with Azure Resource Manager over the internet. This communication is secured using HTTPS/TLS 1.2.
With the Azure Machine Learning CLI extension v1 (azure-cli-ml), only some of the commands communicate with the Azure Resource Manager. Specifically, commands that create, update, delete, list, or show Azure resources. Operations such as submitting a training job communicate directly with the Azure Machine Learning workspace. If your workspace is secured with a private endpoint, that is enough to secure commands provided by the azure-cli-ml extension.
With the Azure Machine Learning CLI extension v2 ('ml'), all of the commands communicate with the Azure Resource Manager. This includes operational data such as YAML parameters and metadata. If your Azure Machine Learning workspace is public (that is, not behind a virtual network), then there is no additional configuration required. Communications are secured using HTTPS/TLS 1.2.
If your Azure Machine Learning workspace uses a private endpoint and virtual network and you are using CLI v2, choose one of the following configurations to use:
-
If you are OK with the CLI v2 communication over the public internet, use the following
--public-network-accessparameter for theaz ml workspace updatecommand to enable public network access. For example, the following command updates a workspace for public network access:az ml workspace update --name myworkspace --public-network-access enabled -
If you are not OK with the CLI v2 communication over the public internet, you can use an Azure Private Link to increase security of the communication. Use the following links to secure communications with Azure Resource Manager by using Azure Private Link.
- Secure your Azure Machine Learning workspace inside a virtual network using a private endpoint.
- Create a Private Link for managing Azure resources.
- Create a private endpoint for the Private Link created in the previous step.
[!IMPORTANT] To configure the private link for Azure Resource Manager, you must be the subscription owner for the Azure subscription, and an owner or contributor of the root management group. For more information, see Create a private link for managing Azure resources.
For more information on CLI v2 communication, see Install and set up the CLI.
Important
If you are using the Azure Cloud Shell, you can skip this section. The cloud shell automatically authenticates you using the account you log into your Azure subscription.
There are several ways that you can authenticate to your Azure subscription from the CLI. The most simple is to interactively authenticate using a browser. To authenticate interactively, open a command line or terminal and use the following command:
az login
If the CLI can open your default browser, it will do so and load a sign-in page. Otherwise, you need to open a browser and follow the instructions on the command line. The instructions involve browsing to https://aka.ms/devicelogin and entering an authorization code.
[!INCLUDE select-subscription]
For other methods of authenticating, see Sign in with Azure CLI.
The Azure Machine Learning workspace must be created inside a resource group. You can use an existing resource group or create a new one. To create a new resource group, use the following command. Replace <resource-group-name> with the name to use for this resource group. Replace <location> with the Azure region to use for this resource group:
Note
You should select a region where Azure Machine Learning is available. For information, see Products available by region.
az group create --name <resource-group-name> --location <location>
The response from this command is similar to the following JSON. You can use the output values to locate the created resources or parse them as input to subsequent CLI steps for automation.
{
"id": "/subscriptions/<subscription-GUID>/resourceGroups/<resourcegroupname>",
"location": "<location>",
"managedBy": null,
"name": "<resource-group-name>",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": null
}For more information on working with resource groups, see az group.
When you deploy an Azure Machine Learning workspace, various other services are required as dependent associated resources. When you use the CLI to create the workspace, the CLI can either create new associated resources on your behalf or you could attach existing resources.
Important
When attaching your own storage account, make sure that it meets the following criteria:
- The storage account is not a premium account (Premium_LRS and Premium_GRS)
- Both Azure Blob and Azure File capabilities enabled
- Hierarchical Namespace (ADLS Gen 2) is disabled These requirements are only for the default storage account used by the workspace.
When attaching Azure container registry, you must have the admin account enabled before it can be used with an Azure Machine Learning workspace.
To create a new workspace where the services are automatically created, use the following command:
az ml workspace create -w <workspace-name> -g <resource-group-name>
[!INCLUDE cli v1]
To create a workspace that uses existing resources, you must provide the resource ID for each resource. You can get this ID either via the 'properties' tab on each resource via the Azure portal, or by running the following commands using the Azure CLI.
- Azure Storage Account:
az storage account show --name <storage-account-name> --query "id" - Azure Application Insights:
az monitor app-insights component show --app <application-insight-name> -g <resource-group-name> --query "id" - Azure Key Vault:
az keyvault show --name <key-vault-name> --query "ID" - Azure Container Registry:
az acr show --name <acr-name> -g <resource-group-name> --query "id"
The returned resource ID has the following format: "/subscriptions/<service-GUID>/resourceGroups/<resource-group-name>/providers/<provider>/<subresource>/<resource-name>".
Once you have the IDs for the resource(s) that you want to use with the workspace, use the base az workspace create -w <workspace-name> -g <resource-group-name> command and add the parameter(s) and ID(s) for the existing resources. For example, the following command creates a workspace that uses an existing container registry:
az ml workspace create -w <workspace-name>
-g <resource-group-name>
--container-registry "/subscriptions/<service-GUID>/resourceGroups/<resource-group-name>/providers/Microsoft.ContainerRegistry/registries/<acr-name>"
[!INCLUDE cli v2]
To create a new workspace while bringing existing associated resources using the CLI, you will first have to define how your workspace should be configured in a configuration file.
:::code language="YAML" source="~/azureml-examples-main/cli/resources/workspace/with-existing-resources.yml":::
Then, you can reference this configuration file as part of the workspace creation CLI command.
az ml workspace create -g <resource-group-name> --file workspace.yml
If attaching existing resources, you must provide the ID for the resources. You can get this ID either via the 'properties' tab on each resource in the Azure portal, or by running the following commands using the Azure CLI.
- Azure Storage Account:
az storage account show --name <storage-account-name> --query "id" - Azure Application Insights:
az monitor app-insights component show --app <application-insight-name> -g <resource-group-name> --query "id" - Azure Key Vault:
az keyvault show --name <key-vault-name> --query "ID" - Azure Container Registry:
az acr show --name <acr-name> -g <resource-group-name> --query "id"
The Resource ID value looks similar to the following: "/subscriptions/<service-GUID>/resourceGroups/<resource-group-name>/providers/<provider>/<subresource>/<resource-name>".
Important
When you attaching existing resources, you don't have to specify all. You can specify one or more. For example, you can specify an existing storage account and the workspace will create the other resources.
The output of the workspace creation command is similar to the following JSON. You can use the output values to locate the created resources or parse them as input to subsequent CLI steps.
{
"applicationInsights": "/subscriptions/<service-GUID>/resourcegroups/<resource-group-name>/providers/microsoft.insights/components/<application-insight-name>",
"containerRegistry": "/subscriptions/<service-GUID>/resourcegroups/<resource-group-name>/providers/microsoft.containerregistry/registries/<acr-name>",
"creationTime": "2019-08-30T20:24:19.6984254+00:00",
"description": "",
"friendlyName": "<workspace-name>",
"id": "/subscriptions/<service-GUID>/resourceGroups/<resource-group-name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace-name>",
"identityPrincipalId": "<GUID>",
"identityTenantId": "<GUID>",
"identityType": "SystemAssigned",
"keyVault": "/subscriptions/<service-GUID>/resourcegroups/<resource-group-name>/providers/microsoft.keyvault/vaults/<key-vault-name>",
"location": "<location>",
"name": "<workspace-name>",
"resourceGroup": "<resource-group-name>",
"storageAccount": "/subscriptions/<service-GUID>/resourcegroups/<resource-group-name>/providers/microsoft.storage/storageaccounts/<storage-account-name>",
"type": "Microsoft.MachineLearningServices/workspaces",
"workspaceid": "<GUID>"
}
Dependent on your use case and organizational requirements, you can choose to configure Azure Machine Learning using private network connectivity. You can use the Azure CLI to deploy a workspace and a Private link endpoint for the workspace resource. For more information on using a private endpoint and virtual network (VNet) with your workspace, see Virtual network isolation and privacy overview. For complex resource configurations, also refer to template based deployment options including Azure Resource Manager.
[!INCLUDE cli v1]
If you want to restrict access to your workspace to a virtual network, you can use the following parameters as part of the az ml workspace create command or use the az ml workspace private-endpoint commands.
az ml workspace create -w <workspace-name>
-g <resource-group-name>
--pe-name "<pe name>"
--pe-auto-approval "<pe-autoapproval>"
--pe-resource-group "<pe name>"
--pe-vnet-name "<pe name>"
--pe-subnet-name "<pe name>"
--pe-name: The name of the private endpoint that is created.--pe-auto-approval: Whether private endpoint connections to the workspace should be automatically approved.--pe-resource-group: The resource group to create the private endpoint in. Must be the same group that contains the virtual network.--pe-vnet-name: The existing virtual network to create the private endpoint in.--pe-subnet-name: The name of the subnet to create the private endpoint in. The default value isdefault.
For more details on how to use these commands, see the CLI reference pages.
[!INCLUDE cli v2]
When using private link, your workspace cannot use Azure Container Registry to build docker images. Hence, you must set the image_build_compute property to a CPU compute cluster name to use for Docker image environment building. You can also specify whether the private link workspace should be accessible over the internet using the public_network_access property.
:::code language="YAML" source="~/azureml-examples-main/cli/resources/workspace/privatelink.yml":::
az ml workspace create -g <resource-group-name> --file privatelink.yml
After creating the workspace, use the Azure networking CLI commands to create a private link endpoint for the workspace.
az network private-endpoint create \
--name <private-endpoint-name> \
--vnet-name <vnet-name> \
--subnet <subnet-name> \
--private-connection-resource-id "/subscriptions/<subscription>/resourceGroups/<resource-group-name>/providers/Microsoft.MachineLearningServices/workspaces/<workspace-name>" \
--group-id amlworkspace \
--connection-name workspace -l <location>
To create the private DNS zone entries for the workspace, use the following commands:
# Add privatelink.api.azureml.ms
az network private-dns zone create \
-g <resource-group-name> \
--name 'privatelink.api.azureml.ms'
az network private-dns link vnet create \
-g <resource-group-name> \
--zone-name 'privatelink.api.azureml.ms' \
--name <link-name> \
--virtual-network <vnet-name> \
--registration-enabled false
az network private-endpoint dns-zone-group create \
-g <resource-group-name> \
--endpoint-name <private-endpoint-name> \
--name myzonegroup \
--private-dns-zone 'privatelink.api.azureml.ms' \
--zone-name 'privatelink.api.azureml.ms'
# Add privatelink.notebooks.azure.net
az network private-dns zone create \
-g <resource-group-name> \
--name 'privatelink.notebooks.azure.net'
az network private-dns link vnet create \
-g <resource-group-name> \
--zone-name 'privatelink.notebooks.azure.net' \
--name <link-name> \
--virtual-network <vnet-name> \
--registration-enabled false
az network private-endpoint dns-zone-group add \
-g <resource-group-name> \
--endpoint-name <private-endpoint-name> \
--name myzonegroup \
--private-dns-zone 'privatelink.notebooks.azure.net' \
--zone-name 'privatelink.notebooks.azure.net'
By default, metadata for the workspace is stored in an Azure Cosmos DB instance that Microsoft maintains. This data is encrypted using Microsoft-managed keys. Instead of using the Microsoft-managed key, you can also provide your own key. Doing so creates an additional set of resources in your Azure subscription to store your data.
To learn more about the resources that are created when you bring your own key for encryption, see Data encryption with Azure Machine Learning.
Below CLI commands provide examples for creating a workspace that uses customer-managed keys for encryption using the CLI v1 and CLI v2 versions.
[!INCLUDE cli v1]
Use the --cmk-keyvault parameter to specify the Azure Key Vault that contains the key, and --resource-cmk-uri to specify the resource ID and uri of the key within the vault.
To limit the data that Microsoft collects on your workspace, you can additionally specify the --hbi-workspace parameter.
az ml workspace create -w <workspace-name>
-g <resource-group-name>
--cmk-keyvault "<cmk keyvault name>"
--resource-cmk-uri "<resource cmk uri>"
--hbi-workspace
[!INCLUDE cli v2]
Use the customer_managed_key parameter and containing key_vault and key_uri parameters, to specify the resource ID and uri of the key within the vault.
To limit the data that Microsoft collects on your workspace, you can additionally specify the hbi_workspace property.
:::code language="YAML" source="~/azureml-examples-main/cli/resources/workspace/cmk.yml":::
Then, you can reference this configuration file as part of the workspace creation CLI command.
az ml workspace create -g <resource-group-name> --file cmk.yml
Note
Authorize the Machine Learning App (in Identity and Access Management) with contributor permissions on your subscription to manage the data encryption additional resources.
Note
Azure Cosmos DB is not used to store information such as model performance, information logged by experiments, or information logged from your model deployments. For more information on monitoring these items, see the Monitoring and logging section of the architecture and concepts article.
Important
Selecting high business impact can only be done when creating a workspace. You cannot change this setting after workspace creation.
For more information on customer-managed keys and high business impact workspace, see Enterprise security for Azure Machine Learning.
To get information about a workspace, use the following command:
[!INCLUDE cli v1]
az ml workspace show -w <workspace-name> -g <resource-group-name>
[!INCLUDE cli v2]
az ml workspace show -n <workspace-name> -g <resource-group-name>
For more information, see the az ml workspace show documentation.
To update a workspace, use the following command:
[!INCLUDE cli v1]
az ml workspace update -w <workspace-name> -g <resource-group-name>
[!INCLUDE cli v2]
az ml workspace update -n <workspace-name> -g <resource-group-name>
For more information, see the az ml workspace update documentation.
If you change access keys for one of the resources used by your workspace, it takes around an hour for the workspace to synchronize to the new key. To force the workspace to sync the new keys immediately, use the following command:
[!INCLUDE cli v1]
az ml workspace sync-keys -w <workspace-name> -g <resource-group-name>
[!INCLUDE cli v2]
az ml workspace sync-keys -n <workspace-name> -g <resource-group-name>
For more information on changing keys, see Regenerate storage access keys.
For more information on the sync-keys command, see az ml workspace sync-keys.
[!INCLUDE machine-learning-delete-workspace]
To delete a workspace after it is no longer needed, use the following command:
[!INCLUDE cli v1]
az ml workspace delete -w <workspace-name> -g <resource-group-name>
[!INCLUDE cli v2]
az ml workspace delete -n <workspace-name> -g <resource-group-name>
Important
Deleting a workspace does not delete the application insight, storage account, key vault, or container registry used by the workspace.
You can also delete the resource group, which deletes the workspace and all other Azure resources in the resource group. To delete the resource group, use the following command:
az group delete -g <resource-group-name>
For more information, see the az ml workspace delete documentation.
If you accidentally deleted your workspace, are still able to retrieve your notebooks. Please refer to this documentation.
[!INCLUDE machine-learning-resource-provider]
Warning
Moving your Azure Machine Learning workspace to a different subscription, or moving the owning subscription to a new tenant, is not supported. Doing so may cause errors.
The Azure Machine Learning workspace uses Azure Container Registry (ACR) for some operations. It will automatically create an ACR instance when it first needs one.
[!INCLUDE machine-learning-delete-acr]
For more information on the Azure CLI extension for machine learning, see the az ml documentation.
To check for problems with your workspace, see How to use workspace diagnostics.
To learn how to move a workspace to a new Azure subscription, see How to move a workspace.