set-up-pyspark-interactive-environment.md

title	description	keywords	ms.service	ms.topic	ms.custom	ms.date
PySpark interactive environment with Azure HDInsight Tools	Learn how to use the Azure HDInsight Tools for Visual Studio Code to create and submit queries and scripts.	VScode,Azure HDInsight Tools,Hive,Python,PySpark,Spark,HDInsight,Hadoop,LLAP,Interactive Hive,Interactive Query	hdinsight	how-to	seoapr2020, devx-track-python	03/30/2022

Set up the PySpark interactive environment for Visual Studio Code

The following steps show how to set up the PySpark interactive environment in VSCode. This step is only for non-Windows users.

We use python/pip command to build virtual environment in your Home path. If you want to use another version, you need to change default version of python/pip command manually. More details see update-alternatives.

1. Install Python and pip.

   • Install Python from https://www.python.org/downloads/.

   • Install pip from https://pip.pypa.io/en/stable/installing (if it's not installed from the Python installation).

   • Optionally validate that Python and pip are installed successfully by using the commands python --version, and pip --version, respectively.

     [!NOTE] It is recommended to manually install Python instead of using the macOS default version.

2. Install virtualenv by running command below.

   pip install virtualenv

Other packages

On Linux, if you come across the error message below, then install the required packages by running the following two commands.

:::image type="content" source="./media/set-up-pyspark-interactive-environment/install-libkrb5-package.png" alt-text="Install libkrb5 package for python" border="true":::

sudo apt-get install libkrb5-dev

sudo apt-get install python-dev

Restart VSCode, and then go back to the VSCode editor and run Spark: PySPark Interactive command.

Next steps

Demo

• HDInsight for VS Code: Video

Tools and extensions

• Use Azure HDInsight Tool for Visual Studio Code
• Use Azure Toolkit for IntelliJ to create and submit Apache Spark Scala applications
• Install Jupyter on your computer and connect to an HDInsight Spark cluster