Skip to content

Files

Latest commit

PatAltimorePatAltimore
Change owner
Mar 7, 2022
aad32b1 · Mar 7, 2022

History

History
299 lines (193 loc) · 18.8 KB

how-to-store-data-blob.md

File metadata and controls

299 lines (193 loc) · 18.8 KB
title description author ms.author ms.reviewer ms.date ms.topic ms.service services
Store block blobs on devices - Azure IoT Edge | Microsoft Docs
Understand tiering and time-to-live features, see supported blob storage operations, and connect to your blob storage account.
PatAltimore
patricka
arduppal
12/13/2019
conceptual
iot-edge
iot-edge

Store data at the edge with Azure Blob Storage on IoT Edge

[!INCLUDE iot-edge-version-all-supported]

Azure Blob Storage on IoT Edge provides a block blob and append blob storage solution at the edge. A blob storage module on your IoT Edge device behaves like an Azure blob service, except the blobs are stored locally on your IoT Edge device. You can access your blobs using the same Azure storage SDK methods or blob API calls that you're already used to. This article explains the concepts related to Azure Blob Storage on IoT Edge container that runs a blob service on your IoT Edge device.

This module is useful in scenarios:

  • where data needs to be stored locally until it can be processed or transferred to the cloud. This data can be videos, images, finance data, hospital data, or any other unstructured data.
  • when devices are located in a place with limited connectivity.
  • when you want to efficiently process the data locally to get low latency access to the data, such that you can respond to emergencies as quickly as possible.
  • when you want to reduce bandwidth costs and avoid transferring terabytes of data to the cloud. You can process the data locally and send only the processed data to the cloud.

Watch the video for quick introduction

[!VIDEO https://www.youtube.com/embed/xbwgMNGB_3Y]

This module comes with deviceToCloudUpload and deviceAutoDelete features.

deviceToCloudUpload is a configurable functionality. This function automatically uploads the data from your local blob storage to Azure with intermittent internet connectivity support. It allows you to:

  • Turn ON/OFF the deviceToCloudUpload feature.
  • Choose the order in which the data is copied to Azure like NewestFirst or OldestFirst.
  • Specify the Azure Storage account to which you want your data uploaded.
  • Specify the containers you want to upload to Azure. This module allows you to specify both source and target container names.
  • Choose the ability to delete the blobs immediately, after upload to cloud storage is finished
  • Do full blob upload (using Put Blob operation) and block level upload (using Put Block, Put Block List and Append Block operations).

This module uses block level upload, when your blob consists of blocks. Here are some of the common scenarios:

  • Your application updates some blocks of a previously uploaded block blob or appends new blocks to an append blob, this module uploads only the updated blocks and not the whole blob.
  • The module is uploading blob and internet connection goes away, when the connectivity is back again it uploads only the remaining blocks and not the whole blob.

If an unexpected process termination (like power failure) happens during a blob upload, all blocks that were due for the upload will be uploaded again once the module comes back online.

deviceAutoDelete is a configurable functionality. This function automatically deletes your blobs from the local storage when the specified duration (measured in minutes) expires. It allows you to:

  • Turn ON/OFF the deviceAutoDelete feature.
  • Specify the time in minutes (deleteAfterMinutes) after which the blobs will be automatically deleted.
  • Choose the ability to retain the blob while it's uploading if the deleteAfterMinutes value expires.

Prerequisites

An Azure IoT Edge device:

  • You can use your development machine or a virtual machine as an IoT Edge device by following the steps in the quickstart for Linux or Windows devices.

  • Refer to Azure IoT Edge supported systems for a list of supported operating systems and architectures. The Azure Blob Storage on IoT Edge module supports following architectures:

    • Windows AMD64
    • Linux AMD64
    • Linux ARM32
    • Linux ARM64

Cloud resources:

A standard-tier IoT Hub in Azure.

deviceToCloudUpload and deviceAutoDelete properties

Use the module's desired properties to set deviceToCloudUploadProperties and deviceAutoDeleteProperties. Desired properties can be set during deployment or changed later by editing the module twin without the need to redeploy. We recommend checking the "Module Twin" for reported configuration and configurationValidation to make sure values are correctly propagated.

deviceToCloudUploadProperties

The name of this setting is deviceToCloudUploadProperties. If you are using the IoT Edge simulator, set the values to the related environment variables for these properties, which you can find in the explanation section.

Property Possible Values Explanation
uploadOn true, false Set to false by default. If you want to turn the feature on, set this field to true.

Environment variable: deviceToCloudUploadProperties__uploadOn={false,true}
uploadOrder NewestFirst, OldestFirst Allows you to choose the order in which the data is copied to Azure. Set to OldestFirst by default. The order is determined by last modified time of Blob.

Environment variable: deviceToCloudUploadProperties__uploadOrder={NewestFirst,OldestFirst}
cloudStorageConnectionString "DefaultEndpointsProtocol=https;AccountName=<your Azure Storage Account Name>;AccountKey=<your Azure Storage Account Key>;EndpointSuffix=<your end point suffix>" is a connection string that allows you to specify the storage account to which you want your data uploaded. Specify Azure Storage Account Name, Azure Storage Account Key, End point suffix. Add appropriate EndpointSuffix of Azure where data will be uploaded, it varies for Global Azure, Government Azure, and Microsoft Azure Stack.

You can choose to specify Azure Storage SAS connection string here. But you have to update this property when it expires. SAS permissions may include create access for containers and create, write, and add access for blobs.

Environment variable: deviceToCloudUploadProperties__cloudStorageConnectionString=<connection string>
storageContainersForUpload "<source container name1>": {"target": "<target container name>"},

"<source container name1>": {"target": "%h-%d-%m-%c"},

"<source container name1>": {"target": "%d-%c"}		 Allows you to specify the container names you want to upload to Azure. This module allows you to specify both source and target container names. If you don't specify the target container name, it will automatically assign the container name as <IoTHubName>-<IotEdgeDeviceID>-<ModuleName>-<SourceContainerName>. You can create template strings for target container name, check out the possible values column.
* %h -> IoT Hub Name (3-50 characters).
* %d -> IoT Edge Device ID (1 to 129 characters).
* %m -> Module Name (1 to 64 characters).
* %c -> Source Container Name (3 to 63 characters).

Maximum size of the container name is 63 characters, while automatically assigning the target container name if the size of container exceeds 63 characters it will trim each section (IoTHubName, IotEdgeDeviceID, ModuleName, SourceContainerName) to 15 characters.

Environment variable: deviceToCloudUploadProperties__storageContainersForUpload__<sourceName>__target=<targetName>
deleteAfterUpload true, false Set to false by default. When it is set to true, it will automatically delete the data when upload to cloud storage is finished.

CAUTION: If you are using append blobs, this setting will delete append blobs from local storage after a successful upload, and any future Append Block operations to those blobs will fail. Use this setting with caution, do not enable this if your application does infrequent append operations or does not support continuous append operations

Environment variable: deviceToCloudUploadProperties__deleteAfterUpload={false,true}.

deviceAutoDeleteProperties

The name of this setting is deviceAutoDeleteProperties. If you are using the IoT Edge simulator, set the values to the related environment variables for these properties, which you can find in the explanation section.

Property Possible Values Explanation
deleteOn true, false Set to false by default. If you want to turn the feature on, set this field to true.

Environment variable: deviceAutoDeleteProperties__deleteOn={false,true}
deleteAfterMinutes <minutes> Specify the time in minutes. The module will automatically delete your blobs from local storage when this value expires. Current maximum minutes allowed is 35791.

Environment variable: deviceAutoDeleteProperties__ deleteAfterMinutes=<minutes>
retainWhileUploading true, false By default it is set to true, and it will retain the blob while it is uploading to cloud storage if deleteAfterMinutes expires. You can set it to false and it will delete the data as soon as deleteAfterMinutes expires. Note: For this property to work uploadOn should be set to true.

CAUTION: If you are using append blobs, this setting will delete append blobs from local storage when the value expires, and any future Append Block operations to those blobs will fail. You may want to make sure the expiry value is large enough for the expected frequency of append operations performed by your application.

Environment variable: deviceAutoDeleteProperties__retainWhileUploading={false,true}

Using SMB share as your local storage

You can provide SMB share as your local storage path, when you deploy Windows container of this module on Windows host.

Make sure the SMB share and IoT device are in mutually trusted domains.

You can run New-SmbGlobalMapping PowerShell command to map the SMB share locally on the IoT device running Windows.

Below are the configuration steps:

$creds = Get-Credential
New-SmbGlobalMapping -RemotePath <remote SMB path> -Credential $creds -LocalPath <Any available drive letter>

For example:

$creds = Get-Credential
New-SmbGlobalMapping -RemotePath \\contosofileserver\share1 -Credential $creds -LocalPath G:

This command will use the credentials to authenticate with the remote SMB server. Then, map the remote share path to G: drive letter (can be any other available drive letter). The IoT device now have the data volume mapped to a path on the G: drive.

Make sure the user in IoT device can read/write to the remote SMB share.

For your deployment the value of <storage mount> can be G:/ContainerData:C:/BlobRoot.

Granting directory access to container user on Linux

If you have used volume mount for storage in your create options for Linux containers then you don't have to do any extra steps, but if you used bind mount then these steps are required to run the service correctly.

Following the principle of least privilege to limit the access rights for users to bare minimum permissions they need to perform their work, this module includes a user (name: absie, ID: 11000) and a user group (name: absie, ID: 11000). If the container is started as root (default user is root), our service will be started as the low-privilege absie user.

This behavior makes configuration of the permissions on host path binds crucial for the service to work correctly, otherwise the service will crash with access denied errors. The path that is used in directory binding needs to be accessible by the container user (example: absie 11000). You can grant the container user access to the directory by executing the commands below on the host:

sudo chown -R 11000:11000 <blob-dir>
sudo chmod -R 700 <blob-dir>

For example:

sudo chown -R 11000:11000 /srv/containerdata
sudo chmod -R 700 /srv/containerdata

If you need to run the service as a user other than absie, you can specify your custom user ID in createOptions under "User" property in your deployment manifest. In such case you need to use default or root group ID 0.

"createOptions": {
  "User": "<custom user ID>:0"
}

Now, grant the container user access to the directory

sudo chown -R <user ID>:<group ID> <blob-dir>
sudo chmod -R 700 <blob-dir>

Configure log files

For information on configuring log files for your module, see these production best practices.

Connect to your blob storage module

You can use the account name and account key that you configured for your module to access the blob storage on your IoT Edge device.

Specify your IoT Edge device as the blob endpoint for any storage requests that you make to it. You can Create a connection string for an explicit storage endpoint using the IoT Edge device information and the account name that you configured.

  • For modules that are deployed on the same device as where the Azure Blob Storage on IoT Edge module is running, the blob endpoint is: http://<module name>:11002/<account name>.
  • For modules or applications running on a different device, you have to choose the right endpoint for your network. Depending on your network setup, choose an endpoint format such that the data traffic from your external module or application can reach the device running the Azure Blob Storage on IoT Edge module. The blob endpoint for this scenario is one of:
    • http://<device IP >:11002/<account name>
    • http://<IoT Edge device hostname>:11002/<account name>
    • http://<fully qualified domain name>:11002/<account name>

Important

Azure IoT Edge is case-sensitive when you make calls to modules, and the Storage SDK also defaults to lowercase. Although the name of the module in the Azure Marketplace is AzureBlobStorageonIoTEdge, changing the name to lowercase helps to ensure that your connections to the Azure Blob Storage on IoT Edge module aren't interrupted.

Azure Blob Storage quickstart samples

The Azure Blob Storage documentation includes quickstart sample code in several languages. You can run these samples to test Azure Blob Storage on IoT Edge by changing the blob endpoint to connect to your local blob storage module.

The following quickstart samples use languages that are also supported by IoT Edge, so you could deploy them as IoT Edge modules alongside the blob storage module:

  • .NET
    • The Azure Blob Storage on Iot Edge module v1.4.0 and earlier are compatible with WindowsAzure.Storage 9.3.3 SDK and v1.4.1 also supports Azure.Storage.Blobs 12.8.0 SDK.
  • Python
    • Versions before V2.1 of the Python SDK have a known issue where the module does not return blob creation time. Because of that issue, some methods like list blobs does not work. As a workaround, explicitly set the API version on the blob client to '2017-04-17'. Example: block_blob_service._X_MS_VERSION = '2017-04-17'
    • Append Blob Sample
  • Node.js
  • JS/HTML
  • Ruby
  • Go
  • PHP

Connect to your local storage with Azure Storage Explorer

You can use Azure Storage Explorer to connect to your local storage account.

  1. Download and install Azure Storage Explorer

  2. Connect to Azure Storage using a connection string

  3. Provide connection string: DefaultEndpointsProtocol=http;BlobEndpoint=http://<host device name>:11002/<your local account name>;AccountName=<your local account name>;AccountKey=<your local account key>;

  4. Go through the steps to connect.

  5. Create container inside your local storage account

  6. Start uploading files as Block blobs or Append Blobs.

    [!NOTE] This module does not support Page blobs.

  7. You can choose to connect your Azure storage accounts in Storage Explorer, too. This configuration gives you a single view for both your local storage account and Azure storage account

Supported storage operations

Blob storage modules on IoT Edge use the Azure Storage SDKs, and are consistent with the 2017-04-17 version of the Azure Storage API for block blob endpoints.

Because not all Azure Blob Storage operations are supported by Azure Blob Storage on IoT Edge, this section lists the status of each.

Account

Supported:

  • List containers

Unsupported:

  • Get and set blob service properties
  • Preflight blob request
  • Get blob service stats
  • Get account information

Containers

Supported:

  • Create and delete container
  • Get container properties and metadata
  • List blobs
  • Get and set container ACL
  • Set container metadata

Unsupported:

  • Lease container

Blobs

Supported:

  • Put, get, and delete blob
  • Get and set blob properties
  • Get and set blob metadata

Unsupported:

  • Lease blob
  • Snapshot blob
  • Copy and abort copy blob
  • Undelete blob
  • Set blob tier

Block blobs

Supported:

  • Put block
  • Put and get block list

Unsupported:

  • Put block from URL

Append blobs

Supported:

  • Append block

Unsupported:

  • Append block from URL

Event Grid on IoT Edge Integration

Caution

The integration with Event Grid on IoT Edge is in preview

This Azure Blob Storage on IoT Edge module now provides integration with Event Grid on IoT Edge. For detailed information on this integration, see the tutorial to deploy the modules, publish events and verify event delivery.

Release Notes

Here are the release notes in docker hub for this module. You might be able to find more information related to bug fixes and remediation in the release notes of a specific version.

Next steps

Learn how to Deploy Azure Blob Storage on IoT Edge

Stay up-to-date with recent updates and announcement in the Azure Blob Storage on IoT Edge blog