You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hpc-cache/customer-keys.md
+17-6Lines changed: 17 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: How to use Azure Key Vault with Azure HPC Cache to control encrypti
4
4
author: ekpgh
5
5
ms.service: hpc-cache
6
6
ms.topic: how-to
7
-
ms.date: 07/20/2020
7
+
ms.date: 07/01/2021
8
8
ms.author: v-erkel
9
9
---
10
10
@@ -71,6 +71,8 @@ You must specify the encryption key source when you create your Azure HPC Cache.
71
71
> [!TIP]
72
72
> If the **Disk encryption keys** page does not appear, make sure that your cache is in one of the [supported regions](https://azure.microsoft.com/global-infrastructure/services/?regions=all&products=hpc-cache,key-vault).
73
73
74
+

75
+
74
76
The user who creates the cache must have privileges equal to the [Key Vault contributor role](../role-based-access-control/built-in-roles.md#key-vault-contributor) or higher.
75
77
76
78
1. Click the button to enable privately managed keys. After you change this setting, the key vault settings appear.
@@ -87,6 +89,15 @@ The user who creates the cache must have privileges equal to the [Key Vault cont
87
89
88
90
1. Specify the version for the selected key. Learn more about versioning in the [Azure Key Vault documentation](../key-vault/general/about-keys-secrets-certificates.md#objects-identifiers-and-versioning).
89
91
92
+
These settings are optional:
93
+
94
+
* Check the **Always use current key version** box if you want to use [automatic key rotation](../virtual-machines/disk-encryption.md#automatic-key-rotation-of-customer-managed-keys-preview).
95
+
96
+
* If you need to use a specific managed identity for this cache, select **User assigned** in the **Managed identities** section and select the identity to use. Read the [managed identities documentation](../active-directory/managed-identities-azure-resources/overview.md#managed-identity-types) for help.
97
+
98
+
> [!NOTE]
99
+
> You cannot change the assigned identity after you create the cache.
100
+
90
101
Continue with the rest of the specifications and create the cache as described in [Create an Azure HPC Cache](hpc-cache-create.md).
91
102
92
103
## 3. Authorize Azure Key Vault encryption from the cache
@@ -104,11 +115,11 @@ This two-step process is necessary because the Azure HPC Cache instance needs an
104
115
105
116
The cache shows the status **Waiting for key**. Click the **Enable encryption** button at the top of the page to authorize the cache to access the specified key vault.
106
117
107
-

118
+

108
119
109
120
Click **Enable encryption** and then click the **Yes** button to authorize the cache to use the encryption key. This action also enables soft-delete and purge protection (if not already enabled) on the key vault.
110
121
111
-

122
+

112
123
113
124
After the cache requests access to the key vault, it can create and encrypt the disks that store cached data.
114
125
@@ -120,17 +131,17 @@ You can change the key vault, key, or key version for your cache from the Azure
120
131
121
132
You cannot change a cache between customer-managed keys and system-managed keys.
122
133
123
-

134
+

124
135
125
136
Click the **Change key** link, then click **Change the key vault, key, or version** to open the key selector.
126
137
127
-

138
+

128
139
129
140
Key vaults in the same subscription and same region as this cache are shown in the list.
130
141
131
142
After you choose the new encryption key values, click **Select**. A confirmation page appears with the new values. Click **Save** to finalize the selection.
132
143
133
-

144
+

Copy file name to clipboardExpand all lines: articles/hpc-cache/hpc-cache-create.md
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -55,7 +55,7 @@ When you choose a throughput value, keep in mind that the actual data transfer r
55
55
56
56
The values you choose set the maximum throughput for the entire cache system, but some of that is used for overhead tasks. For example, if a client requests a file that isn't already stored in the cache, or if the file is marked as stale, your cache uses some of its throughput to fetch it from back-end storage.
57
57
58
-
Azure HPC Cache manages which files are cached and preloaded to maximize cache hit rates. Cache contents are continuously assessed, and files are moved to long-term storage when they're less frequently accessed.
58
+
Azure HPC Cache manages which files are cached and pre-loaded to maximize cache hit rates. Cache contents are continuously assessed, and files are moved to long-term storage when they're less frequently accessed.
59
59
60
60
Choose a cache storage size that can comfortably hold the active set of working files, plus additional space for metadata and other overhead.
Copy file name to clipboardExpand all lines: articles/hpc-cache/hpc-cache-edit-storage.md
+4-4Lines changed: 4 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -4,15 +4,16 @@ description: How to edit Azure HPC Cache storage targets
4
4
author: ekpgh
5
5
ms.service: hpc-cache
6
6
ms.topic: how-to
7
-
ms.date: 06/28/2021
7
+
ms.date: 06/30/2021
8
8
ms.author: v-erkel
9
9
---
10
10
11
11
# Edit storage targets
12
12
13
13
You can modify storage targets with the Azure portal or by using the Azure CLI. For example, you can change access policies, usage models, and namespace paths for an existing storage target.
14
14
15
-
Storage target *management* tasks, like suspending or deleting a storage target, are described in [Manage storage targets](manage-storage-targets.md).
15
+
> [!TIP]
16
+
> Read [Manage storage targets](manage-storage-targets.md) to learn how to delete or suspend storage targets, or make them write cached data to back-end storage.
16
17
17
18
Depending on the type of storage, you can modify these storage target values:
18
19
@@ -29,8 +30,7 @@ Depending on the type of storage, you can modify these storage target values:
29
30
30
31
You can't edit a storage target's name, type, or back-end storage system. If you need to change these properties, delete the storage target and create a replacement with the new value.
31
32
32
-
> [!TIP]
33
-
> The [Managing Azure HPC Cache video](https://azure.microsoft.com/resources/videos/managing-hpc-cache/) shows how to edit a storage target in the Azure portal.
33
+
The [Managing Azure HPC Cache video](https://azure.microsoft.com/resources/videos/managing-hpc-cache/) shows how to edit a storage target in the Azure portal.
34
34
35
35
## Change a blob storage target's namespace path or access policy
Copy file name to clipboardExpand all lines: articles/hpc-cache/hpc-cache-ingest.md
+16-12Lines changed: 16 additions & 12 deletions
Original file line number
Diff line number
Diff line change
@@ -4,29 +4,33 @@ description: How to populate Azure Blob storage for use with Azure HPC Cache
4
4
author: ekpgh
5
5
ms.service: hpc-cache
6
6
ms.topic: how-to
7
-
ms.date: 10/30/2019
7
+
ms.date: 06/30/2021
8
8
ms.author: v-erkel
9
9
---
10
10
11
11
# Move data to Azure Blob storage
12
12
13
-
If your workflow includes moving data to Azure Blob storage, make sure you are using an efficient strategy. You can either preload data in a new Blob container before defining it as a storage target, or add the container and then copy your data using Azure HPC Cache.
13
+
If your workflow includes moving data to Azure Blob storage, make sure you are using an efficient strategy. You can either pre-load data in a new blob container before defining it as a storage target, or add the container and then copy your data using Azure HPC Cache.
14
14
15
-
This article explains the best ways to move data to Blob storage for use with Azure HPC Cache.
15
+
This article explains the best ways to move data to blob storage for use with Azure HPC Cache.
16
+
17
+
> [!TIP]
18
+
>
19
+
> This article does not apply to NFS-mounted blob storage (ADLS-NFS storage targets). You can use any NFS-based method to populate an ADLS-NFS blob container before adding it to the HPC Cache. Read [Pre-load data with NFS protocol](nfs-blob-considerations.md#Pre-load-data-with-nfs-protocol) to learn more.
16
20
17
21
Keep these facts in mind:
18
22
19
-
* Azure HPC Cache uses a specialized storage format to organize data in Blob storage. This is why a Blob storage target must either be a new, empty container, or a Blob container that was previously used for Azure HPC Cache data.
23
+
* Azure HPC Cache uses a specialized storage format to organize data in blob storage. This is why a blob storage target must either be a new, empty container, or a blob container that was previously used for Azure HPC Cache data.
20
24
21
25
* Copying data through the Azure HPC Cache to a back-end storage target is more efficient when you use multiple clients and parallel operations. A simple copy command from one client will move data slowly.
22
26
23
-
A Python-based utility is available to load content into a Blob storage container. Read [Pre-load data in Blob storage](#pre-load-data-in-blob-storage-with-clfsload) to learn more.
27
+
A Python-based utility is available to load content into a blob storage container. Read [Pre-load data in blob storage](#pre-load-data-in-blob-storage-with-clfsload) to learn more.
24
28
25
29
If you don't want to use the loading utility, or if you want to add content to an existing storage target, follow the parallel data ingest tips in [Copy data through the Azure HPC Cache](#copy-data-through-the-azure-hpc-cache).
26
30
27
-
## Pre-load data in Blob storage with CLFSLoad
31
+
## Pre-load data in blob storage with CLFSLoad
28
32
29
-
You can use the Avere CLFSLoad utility to copy data to a new Blob storage container before you add it as a storage target. This utility runs on a single Linux system and writes data in the proprietary format needed for Azure HPC Cache. CLFSLoad is the most efficient way to populate a Blob storage container for use with the cache.
33
+
You can use the Avere CLFSLoad utility to copy data to a new blob storage container before you add it as a storage target. This utility runs on a single Linux system and writes data in the proprietary format needed for Azure HPC Cache. CLFSLoad is the most efficient way to populate a blob storage container for use with the cache.
30
34
31
35
The Avere CLFSLoad utility is available by request from your Azure HPC Cache team. Ask your team contact for it, or open a [support ticket](hpc-cache-support-ticket.md) to request assistance.
32
36
@@ -42,20 +46,20 @@ A general overview of the process:
42
46
43
47
The Avere CLFSLoad utility needs the following information:
44
48
45
-
* The storage account ID that contains your Blob storage container
46
-
* The name of the empty Blob storage container
49
+
* The storage account ID that contains your blob storage container
50
+
* The name of the empty blob storage container
47
51
* A shared access signature (SAS) token that allows the utility to write to the container
48
52
* A local path to the data source - either a local directory that contains the data to copy, or a local path to a mounted remote system with the data
49
53
50
54
## Copy data through the Azure HPC Cache
51
55
52
-
If you don't want to use the Avere CLFSLoad utility, or if you want to add a large amount of data to an existing Blob storage target, you can copy it through the cache. Azure HPC Cache is designed to serve multiple clients simultaneously, so to copy data through the cache, you should use parallel writes from multiple clients.
56
+
If you don't want to use the Avere CLFSLoad utility, or if you want to add a large amount of data to an existing blob storage target, you can copy it through the cache. Azure HPC Cache is designed to serve multiple clients simultaneously, so to copy data through the cache, you should use parallel writes from multiple clients.
53
57
54
-

58
+

55
59
56
60
The ``cp`` or ``copy`` commands that you typically use to transfer data from one storage system to another are single-threaded processes that copy only one file at a time. This means that the file server is ingesting only one file at a time - which is a waste of the cache's resources.
57
61
58
-
This section explains strategies for creating a multi-client, multi-threaded file copying system to move data to Blob storage with Azure HPC Cache. It explains file transfer concepts and decision points that can be used for efficient data copying using multiple clients and simple copy commands.
62
+
This section explains strategies for creating a multi-client, multi-threaded file copying system to move data to blob storage with Azure HPC Cache. It explains file transfer concepts and decision points that can be used for efficient data copying using multiple clients and simple copy commands.
59
63
60
64
It also explains some utilities that can help. The ``msrsync`` utility can be used to partially automate the process of dividing a dataset into buckets and using rsync commands. The ``parallelcp`` script is another utility that reads the source directory and issues copy commands automatically.
Copy file name to clipboardExpand all lines: articles/hpc-cache/nfs-blob-considerations.md
+4-3Lines changed: 4 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -30,7 +30,8 @@ To work around this difference, Azure HPC Cache automatically disables NFS attri
30
30
31
31
This setting persists for the lifetime of the container, even if you remove it from the cache.
32
32
33
-
## Preload data with NFS protocol
33
+
## Pre-load data with NFS protocol
34
+
<!-- cross-referenced from hpc-cache-ingest.md and here -->
34
35
35
36
On an NFS-enabled blob container, *a file can only be edited by the same protocol used when it was created*. That is, if you use the Azure REST API to populate a container, you cannot use NFS to update those files. Because Azure HPC Cache only uses NFS, it can't edit any files that were created with the Azure REST API.
36
37
@@ -44,7 +45,7 @@ If the files in your container were created with Azure Blob's REST API instead o
44
45
* Empty the file (truncate it to 0).
45
46
* Save a copy of the file. The copy is marked as an NFS-created file, and it can be edited using NFS.
46
47
47
-
Azure HPC Cache **can't** edit the contents of a file that was created using REST. This means that it can't save a changed file from a client back to the storage target.
48
+
**Azure HPC Cache can't edit the contents of a file that was created using REST.** This means that the cache can't save a changed file from a client back to the storage target.
48
49
49
50
It's important to understand this limitation, because it can cause data integrity problems if you use read/write caching usage models on files that were not created with NFS.
50
51
@@ -103,7 +104,7 @@ Azure HPC Cache can help improve performance in a workload that includes writing
103
104
104
105
One of the limitations outlined in the NFS-enabled blob [Performance considerations article](../storage/blobs/network-file-system-protocol-support-performance.md) is that ADLS-NFS storage is not very efficient at overwriting existing files. If you use Azure HPC Cache with NFS-mounted blob storage, the cache handles intermittent rewrites as clients modify an active file. The latency of writing a file to the back end container is hidden from the clients.
105
106
106
-
Keep in mind the limitations explained above in [Preload data with NFS protocol](#preload-data-with-nfs-protocol).
107
+
Keep in mind the limitations explained above in [Pre-load data with NFS protocol](#pre-load-data-with-nfs-protocol).
0 commit comments