| title | description | ms.service | ms.topic | ms.date |
|---|---|---|---|---|
Apache Hadoop components and versions - Azure HDInsight |
Learn about the Apache Hadoop components and versions in Azure HDInsight. |
hdinsight |
conceptual |
02/15/2022 |
HDInsight bundles Apache Hadoop environment components and HDInsight platform into a package that is deployed on a cluster. For more details see how HDInsight versioning works.
This table lists the versions of HDInsight that are available in the Azure portal and other deployment methods like PowerShell, CLI and the .NET SDK.
| HDInsight version | VM OS | Release date | Support type | Support expiration date | Retirement date | High availability |
|---|---|---|---|---|---|---|
| HDInsight 4.0 | Ubuntu 18.0.4 LTS | September 24, 2018 | Standard | Yes | ||
| HDInsight 3.6 | Ubuntu 16.0.4 LTS | April 4, 2017 | Basic | Standard support expired on June 30, 2021 for all cluster types. Basic support expires on September 30, 2022. See HDInsight 3.6 component versions for cluster type details. |
October 1, 2022 | Yes |
Support expiration means that Microsoft no longer provides support for the specific HDInsight version. And it may no longer available through the Azure portal for cluster creation.
Retirement means that existing clusters of an HDInsight version continue to run as is. New clusters of this version can't be created through any means, which includes the CLI and SDKs. Other control plane features, such as manual scaling and autoscaling, are not guaranteed to work after retirement date. Support isn't available for retired versions.
Support is defined as a time period that an HDInsight version is supported by Microsoft Customer Service and Support. HDInsight offers two types of support:
- Standard support
- Basic support
Standard support provides updates and support on HDInsight clusters. Microsoft recommends building solutions using the most recent fully supported version.
Standard support includes the following:
- Ability to create support requests on HDInsight 4.0 clusters.
- Support for troubleshooting solutions built on 4.0 clusters.
- Requests to restart services or nodes.
- Root cause analysis investigations on support requests.
- Root cause analysis or fixes to improve job or query performance.
- Root cause analysis or fixes to improve customer-initiated changes, e.g., changing service configurations or issues due to custom script actions.
- Product updates for critical security fixes until version retirement.
- Scoped product updates to the HDInsight Resource provider.
- Selective fixes or changes to HDInsight 4.0 images or open-source software (OSS) component versions.
Basic support provides limited servicing to the HDInsight Resource provider. HDInsight images and open-source software (OSS) components will not be serviced. Only critical security fixes will be patched on HDInsight clusters.
Basic support includes the following:
- Continued use of existing HDInsight 3.6 clusters.
- Ability for existing HDInsight 3.6 customers to create new 3.6 clusters.
- Ability to scale HDInsight 3.6 clusters up and down via autoscale or manual scale.
- Scoped product updates to the HDInsight Resource provider.
- Product updates for critical security fixes until version retirement.
- Ability to create support requests on HDInsight 3.6 clusters.
- Requests to restart services or nodes.
Basic support does not include the following:
- Fixes or changes to HDInsight 3.6 images or open-source software (OSS) component versions.
- Support for troubleshooting solutions built on 3.6 clusters.
- Adding new features or functionality.
- Support for advice or ad-hoc queries.
- Root cause analysis investigations on support requests.
- Root cause analysis or fixes to improve job or query performance.
- Root cause analysis or fixes to improve customer-initiated changes, e.g., changing service configurations or issues due to custom script actions.
Microsoft does not encourage creating analytics pipelines or solutions on clusters in basic support. We recommend migrating existing clusters to the most recent fully supported version.
- Migrate Apache Spark 2.1 and 2.2 workloads to 2.3 and 2.4.
- Migrate Azure HDInsight 3.6 Hive workloads to HDInsight 4.0.
- Migrate Apache Kafka workloads to Azure HDInsight 4.0.
- Migrate an Apache HBase cluster to a new version.
- Migrate Azure HDInsight 3.6 Apache Storm to HDInsight 4.0 Apache Spark.
For additional release notes on the latest versions of HDInsight, see HDInsight release notes.
- Once a cluster is deployed with an image, that cluster is not automatically upgraded to newer image version. When creating new clusters, most recent image version will be deployed.
- Customers should test and validate that applications run properly when using new HDInsight version.
- HDInsight reserves the right to change the default version without prior notice. If you have a version dependency, specify the HDInsight version when you create your clusters.
- HDInsight may retire an OSS component version before retiring the HDInsight version.