Skip to content

Files

Latest commit

JasonWHowellJasonWHowell
Fixing unmoved includes
Jun 17, 2021
778bb53 · Jun 17, 2021

History

History
146 lines (94 loc) · 5.41 KB

hdinsight-administer-use-powershell.md

File metadata and controls

146 lines (94 loc) · 5.41 KB
title description ms.service ms.topic ms.custom ms.date
Manage Apache Hadoop clusters with PowerShell - Azure HDInsight
Learn how to perform administrative tasks for the Apache Hadoop clusters in HDInsight using Azure PowerShell.
hdinsight
conceptual
hdinsightactive, devx-track-azurepowershell
02/13/2020

Manage Apache Hadoop clusters in HDInsight by using Azure PowerShell

[!INCLUDE selector]

Azure PowerShell can be used to control and automate the deployment and management of your workloads in Azure. In this article, you learn how to manage Apache Hadoop clusters in Azure HDInsight by using the Azure PowerShell Az module. For the list of the HDInsight PowerShell cmdlets, see the Az.HDInsight reference.

If you don't have an Azure subscription, create a free account before you begin.

Prerequisites

[!INCLUDE updated-for-az]

The PowerShell Az Module installed.

Create clusters

See Create Linux-based clusters in HDInsight using Azure PowerShell

List clusters

Use the following command to list all clusters in the current subscription:

Get-AzHDInsightCluster

Show cluster

Use the following command to show details of a specific cluster in the current subscription:

Get-AzHDInsightCluster -ClusterName <Cluster Name>

Delete clusters

Use the following command to delete a cluster:

Remove-AzHDInsightCluster -ClusterName <Cluster Name>

You can also delete a cluster by removing the resource group that contains the cluster. Deleting a resource group deletes all the resources in the group including the default storage account.

Remove-AzResourceGroup -Name <Resource Group Name>

Scale clusters

The cluster scaling feature allows you to change the number of worker nodes used by a cluster that is running in Azure HDInsight without having to re-create the cluster. To change the Hadoop cluster size by using Azure PowerShell, run the following command from a client machine:

Set-AzHDInsightClusterSize -ClusterName <Cluster Name> -TargetInstanceCount <NewSize>

For more information about scaling clusters, see Scale HDInsight clusters.

Update HTTP user credentials

Set-AzHDInsightGatewayCredential sets the gateway HTTP credentials of an Azure HDInsight cluster.

$clusterName = "CLUSTERNAME"
$credential = Get-Credential -Message "Enter the HTTP username and password:" -UserName "admin"

Set-AzHDInsightGatewayCredential -ClusterName $clusterName -HttpCredential $credential

Find the default storage account

The following PowerShell script demonstrates how to get the default storage account name and the related information:

#Connect-AzAccount
$clusterName = "<HDInsight Cluster Name>"

$clusterInfo = Get-AzHDInsightCluster -ClusterName $clusterName
$storageInfo = $clusterInfo.DefaultStorageAccount.split('.')
$defaultStoreageType = $storageInfo[1]
$defaultStorageName = $storageInfo[0]

echo "Default Storage account name: $defaultStorageName"
echo "Default Storage account type: $defaultStoreageType"

if ($defaultStoreageType -eq "blob")
{
    $defaultBlobContainerName = $cluster.DefaultStorageContainer
    $defaultStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $defaultStorageAccountName)[0].Value
    $defaultStorageAccountContext = New-AzStorageContext -StorageAccountName $defaultStorageAccountName -StorageAccountKey $defaultStorageAccountKey

    echo "Default Blob container name: $defaultBlobContainerName"
    echo "Default Storage account key: $defaultStorageAccountKey"
}

Find the resource group

In the Resource Manager mode, each HDInsight cluster belongs to an Azure resource group. To find the resource group:

$clusterName = "<HDInsight Cluster Name>"

$cluster = Get-AzHDInsightCluster -ClusterName $clusterName
$resourceGroupName = $cluster.ResourceGroup

Submit jobs

To submit MapReduce jobs

See Run the MapReduce examples included in HDInsight.

To submit Apache Hive jobs

See Run Apache Hive queries using PowerShell.

To submit Apache Sqoop jobs

See Use Apache Sqoop with HDInsight.

To submit Apache Oozie jobs

See Use Apache Oozie with Apache Hadoop to define and run a workflow in HDInsight.

Upload data to Azure Blob storage

See Upload data to HDInsight.

See Also