Skip to content

Files

Latest commit

BlackmistBlackmist
moe-vnet-public-note
Jun 6, 2022
11e6d6e · Jun 6, 2022

History

History
265 lines (167 loc) · 16.1 KB

how-to-secure-online-endpoint.md

File metadata and controls

265 lines (167 loc) · 16.1 KB
title titleSuffix description services ms.service ms.subservice ms.topic ms.reviewer ms.author author ms.date ms.custom
Network isolation of managed online endpoints
Azure Machine Learning
Use private endpoints to provide network isolation for Azure Machine Learning managed online endpoints.
machine-learning
machine-learning
enterprise-readiness
how-to
larryfr
seramasu
rsethur
06/06/2022
event-tier1-build-2022

Use network isolation with managed online endpoints (preview)

When deploying a machine learning model to a managed online endpoint, you can secure communication with the online endpoint by using private endpoints. Using a private endpoint with online endpoints is currently a preview feature.

[!INCLUDE preview disclaimer]

You can secure the inbound scoring requests from clients to an online endpoint. You can also secure the outbound communications between a deployment and the Azure resources used by the deployment. Security for inbound and outbound communication is configured separately. For more information on endpoints and deployments, see What are endpoints and deployments.

The following diagram shows how communications flow through private endpoints to the managed online endpoint. Incoming scoring requests from clients are received through the workspace private endpoint from your virtual network. Outbound communication with services is handled through private endpoints to those service instances from the deployment:

:::image type="content" source="./media/how-to-secure-online-endpoint/endpoint-network-isolation-ingress-egress.png" alt-text="Diagram of overall ingress/egress communication.":::

Prerequisites

  • To use Azure machine learning, you must have an Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the free or paid version of Azure Machine Learning today.

  • You must install and configure the Azure CLI and ML extension. For more information, see Install, set up, and use the CLI (v2).

  • You must have an Azure Resource Group, in which you (or the service principal you use) need to have Contributor access. You'll have such a resource group if you configured your ML extension per the above article.

  • You must have an Azure Machine Learning workspace, and the workspace must use a private endpoint. If you don't have one, the steps in this article create an example workspace, VNet, and VM. For more information, see Configure a private endpoint for Azure Machine Learning workspace.

    The workspace can be configured to allow or disallow public network access. If you plan on using managed online endpoint deployments that use public outbound, then you must also configure the workspace to allow public access.

    Outbound communication from managed online endpoint deployment is to the workspace API. When the endpoint is configured to use public outbound, then the workspace must be able to accept that public communication (allow public access).

  • When the workspace is configured with a private endpoint, the Azure Container Registry for the workspace must be configured for Premium tier. For more information, see Azure Container Registry service tiers.

  • The Azure Container Registry and Azure Storage Account must be in the same Azure Resource Group as the workspace.

Important

The end-to-end example in this article comes from the files in the azureml-examples GitHub repository. To clone the samples repository and switch to the repository's cli/ directory, use the following commands:

git clone https://github.com/Azure/azureml-examples
cd azureml-examples/cli

Limitations

  • The v1_legacy_mode flag must be disabled (false) on your Azure Machine Learning workspace. If this flag is enabled, you won't be able to create a managed online endpoint. For more information, see Network isolation with v2 API.

  • If your Azure Machine Learning workspace has a private endpoint that was created before May 24, 2022, you must recreate the workspace's private endpoint before configuring your online endpoints to use a private endpoint. For more information on creating a private endpoint for your workspace, see How to configure a private endpoint for Azure Machine Learning workspace.

  • Secure outbound communication creates three private endpoints per deployment. One to Azure Blob storage, one to Azure Container Registry, and one to your workspace.

  • Azure Log Analytics and Application Insights aren't supported when using network isolation with a deployment. To see the logs for the deployment, use the az ml online-deployment get_logs command instead.

  • You can configure public access to a managed online endpoint (inbound and outbound). You can also configure public access to an Azure Machine Learning workspace.

    Outbound communication from managed online endpoint deployment is to the workspace API. When the endpoint is configured to use public outbound, then the workspace must be able to accept that public communication (allow public access).

Note

Requests to create, update, or retrieve the authentication keys are sent to the Azure Resource Manager over the public network.

Inbound (scoring)

To secure scoring requests to the online endpoint to your virtual network, set the public_network_access flag for the endpoint to disabled:

az ml online-endpoint create -f endpoint.yml --set public_network_access=disabled

When public_network_access is disabled, inbound scoring requests are received using the private endpoint of the Azure Machine Learning workspace and the endpoint can't be reached from public networks.

Outbound (resource access)

To restrict communication between a deployment and the Azure resources used to by the deployment, set the egress_public_network_access flag to disabled. Use this flag to ensure that the download of the model, code, and images needed by your deployment are secured with a private endpoint.

The following are the resources that the deployment communicates with over the private endpoint:

  • The Azure Machine Learning workspace.
  • The Azure Storage blob that is the default storage for the workspace.
  • The Azure Container Registry for the workspace.

When you configure the egress_public_network_access to disabled, a new private endpoint is created per deployment, per service. For example, if you set the flag to disabled for three deployments to an online endpoint, nine private endpoints are created. Each deployment would have three private endpoints that are used to communicate with the workspace, blob, and container registry.

az ml online-deployment create -f deployment.yml --set egress_public_network_access=disabled

Scenarios

The following table lists the supported configurations when configuring inbound and outbound communications for an online endpoint:

Configuration Inbound
(Endpoint property)		 Outbound
(Deployment property)		 Supported?
secure inbound with secure outbound public_network_access is disabled egress_public_network_access is disabled Yes
secure inbound with public outbound public_network_access is disabled
The workspace must also allow public access.		 egress_public_network_access is enabled Yes
public inbound with secure outbound public_network_access is enabled egress_public_network_access is disabled Yes
public inbound with public outbound public_network_access is enabled
The workspace must also allow public access.		 egress_public_network_access is enabled Yes

Important

Outbound communication from managed online endpoint deployment is to the workspace API. When the endpoint is configured to use public outbound, then the workspace must be able to accept that public communication (allow public access).

End-to-end example

Use the information in this section to create an example configuration that uses private endpoints to secure online endpoints.

Tip

In this example, and Azure Virtual Machine is created inside the VNet. You connect to the VM using SSH, and run the deployment from the VM. This configuration is used to simplify the steps in this example, and does not represent a typical secure configuration. For example, in a production environment you would most likely use a VPN client or Azure ExpressRoute to directly connect clients to the virtual network.

Create workspace and secured resources

The steps in this section use an Azure Resource Manager template to create the following Azure resources:

  • Azure Virtual Network
  • Azure Machine Learning workspace
  • Azure Container Registry
  • Azure Key Vault
  • Azure Storage account (blob & file storage)

Public access is disabled for all the services. While the Azure Machine Learning workspace is secured behind a vnet, it's configured to allow public network access. For more information, see CLI 2.0 secure communications. A scoring subnet is created, along with outbound rules that allow communication with the following Azure services:

  • Azure Active Directory
  • Azure Resource Manager
  • Azure Front Door
  • Microsoft Container Registries

The following diagram shows the different components created in this architecture:

The following diagram shows the overall architecture of this example:

:::image type="content" source="./media/how-to-secure-online-endpoint/endpoint-network-isolation-diagram.png" alt-text="Diagram of the services created.":::

To create the resources, use the following Azure CLI commands. Replace <UNIQUE_SUFFIX> with a unique suffix for the resources that are created.

:::code language="azurecli" source="~/azureml-examples-main/setup-repo/azure-github.sh" id="managed_vnet_workspace_suffix":::

:::code language="azurecli" source="~/azureml-examples-main/setup-repo/azure-github.sh" id="managed_vnet_workspace_create":::

Create the virtual machine jump box

To create an Azure Virtual Machine that can be used to connect to the VNet, use the following command. Replace <your-new-password> with the password you want to use when connecting to this VM:

# create vm
az vm create --name test-vm --vnet-name vnet-$SUFFIX --subnet snet-scoring --image UbuntuLTS --admin-username azureuser --admin-password <your-new-password>

Important

The VM created by these commands has a public endpoint that you can connect to over the public network.

The response from this command is similar to the following JSON document:

{
  "fqdns": "",
  "id": "/subscriptions/<GUID>/resourceGroups/<my-resource-group>/providers/Microsoft.Compute/virtualMachines/test-vm",
  "location": "westus",
  "macAddress": "00-0D-3A-ED-D8-E8",
  "powerState": "VM running",
  "privateIpAddress": "192.168.0.12",
  "publicIpAddress": "20.114.122.77",
  "resourceGroup": "<my-resource-group>",
  "zones": ""
}

Use the following command to connect to the VM using SSH. Replace publicIpAddress with the value of the public IP address in the response from the previous command:

ssh azureusere@publicIpAddress

When prompted, enter the password you used when creating the VM.

Configure the VM

  1. Use the following commands from the SSH session to install the CLI and Docker:

    :::code language="azurecli" source="~/azureml-examples-main/cli/endpoints/online/managed/vnet/setup_vm/scripts/vmsetup.sh" id="setup_docker_az_cli":::

  2. To create the environment variables used by this example, run the following commands. Replace <YOUR_SUBSCRIPTION_ID> with your Azure subscription ID. Replace <YOUR_RESOURCE_GROUP> with the resource group that contains your workspace. Replace <SUFFIX_USED_IN_SETUP> with the suffix you provided earlier. Replace <LOCATION> with the location of your Azure workspace. Replace <YOUR_ENDPOINT_NAME> with the name to use for the endpoint.

    [!TIP] Use the tabs to select whether you want to perform a deployment using an MLflow model or generic ML model.

    Generic model

    :::code language="azurecli" source="~/azureml-examples-main/cli/deploy-moe-vnet.sh" id="set_env_vars":::

    MLflow model

    :::code language="azurecli" source="~/azureml-examples-main/cli/deploy-moe-vnet-mlflow.sh" id="set_env_vars":::

  3. To sign in to the Azure CLI in the VM environment, use the following command:

    :::code language="azurecli" source="~/azureml-examples-main/cli/misc.sh" id="az_login":::

  4. To configure the defaults for the CLI, use the following commands:

    :::code language="azurecli" source="~/azureml-examples-main/cli/endpoints/online/managed/vnet/setup_vm/scripts/vmsetup.sh" id="configure_defaults":::

  5. To clone the example files for the deployment, use the following command:

    sudo mkdir -p /home/samples; sudo git clone -b main --depth 1 https://github.com/Azure/azureml-examples.git /home/samples/azureml-examples

  6. To build a custom docker image to use with the deployment, use the following commands:

    :::code language="azurecli" source="~/azureml-examples-main/cli/endpoints/online/managed/vnet/setup_vm/scripts/build_image.sh" id="build_image":::

    [!TIP] In this example, we build the Docker image before pushing it to Azure Container Registry. Alternatively, you can build the image in your vnet by using an Azure Machine Learning compute cluster and environments. For more information, see Secure Azure Machine Learning workspace.

Create a secured managed online endpoint

  1. To create a managed online endpoint that is secured using a private endpoint for inbound and outbound communication, use the following commands:

    [!TIP] You can test or debug the Docker image locally by using the --local flag when creating the deployment. For more information, see the Deploy and debug locally article.

    :::code language="azurecli" source="~/azureml-examples-main/cli/endpoints/online/managed/vnet/setup_vm/scripts/create_moe.sh" id="create_vnet_deployment":::

  2. To make a scoring request with the endpoint, use the following commands:

    :::code language="azurecli" source="~/azureml-examples-main/cli/endpoints/online/managed/vnet/setup_vm/scripts/score_endpoint.sh" id="check_deployment":::

Cleanup

To delete the endpoint, use the following command:

:::code language="azurecli" source="~/azureml-examples-main/cli/deploy-moe-vnet.sh" id="delete_endpoint":::

To delete the VM, use the following command:

:::code language="azurecli" source="~/azureml-examples-main/cli/deploy-moe-vnet.sh" id="delete_vm":::

To delete all the resources created in this article, use the following command. Replace <resource-group-name> with the name of the resource group used in this example:

az group delete --resource-group <resource-group-name>

Troubleshooting

[!INCLUDE network isolation issues]

Next steps