| title | description | services | author | ms.author | ms.service | ms.topic | ms.date |
|---|---|---|---|---|---|---|---|
Automate builds, tests, and deployments of an Azure Stream Analytics job using CI/CD tools |
This article describes how to use Azure Stream Analytics CI/CD tools to auto build, test, and deploy an Azure Stream Analytics project. |
stream-analytics |
su-jie |
sujie |
stream-analytics |
how-to |
06/29/2021 |
You can use the Azure Stream Analytics CI/CD npm package to automatically build, test, and deploy your Azure Stream Analytics Visual Studio Code or Visual Studio projects. The projects can be created using development tools or they can be exported from existing Stream Analytics jobs. This article describes how to use the npm package with any CI/CD system. For deployment with Azure Pipelines, see Use Azure DevOps to create a CI/CD pipeline for a Stream Analytics job.
You can download the package directly, or install it globally using the npm install -g azure-streamanalytics-cicd command. We recommend using the command, which can also be used in a PowerShell or Azure CLI script task of a build pipeline in Azure Pipelines.
The asa-streamanalytics-cicd npm package provides the tools to generate Azure Resource Manager templates of Stream Analytics Visual Studio Code projects or Visual Studio projects. You can also use the npm package on Windows, macOS, and Linux without installing Visual Studio Code or Visual Studio.
Once you have installed the package, use the following command to build your Stream Analytics projects.
azure-streamanalytics-cicd build -project <projectFullPath> [-outputPath <outputPath>]The build command does a keyword syntax check and outputs the Azure Resource Manager template.
| Parameter | Description |
|---|---|
-project |
The absolute path of the asaproj.json file for your Visual Studio Code project or [Your project name].asaproj for Visual Studio project. |
-outputPath |
The path of the output folder for Azure Resource Manager Templates. If it is not specified, the templates will be placed in the current directory. |
azure-streamanalytics-cicd build -project "/Users/username/projects/samplejob/asaproj.json"azure-streamanalytics-cicd build -project "/Users/username/projects/samplejob/samplejob.asaproj"When a Stream Analytics project builds successfully, it generates the following two files under the output folder:
-
Azure Resource Manager template file
[ProjectName].JobTemplate.json -
Azure Resource Manager parameter file
[ProjectName].JobTemplate.parameters.json
The default parameters in the parameters.json file are from the settings in your Visual Studio Code or Visual Studio project. If you want to deploy to another environment, replace the parameters accordingly.
The default values for all credentials are null. You are required to set the values before you deploy to Azure.
"Input_EntryStream_sharedAccessPolicyKey": {
"value": null
},To use Managed Identity for Azure Data Lake Store Gen1 as an output sink, you need to provide access to the service principal using PowerShell before you deploy to Azure. Learn more about how to deploy ADLS Gen1 with Managed Identity with Resource Manager template.
If your project has specified local input files, you can run a Stream Analytics script locally by using the localrun command.
azure-streamanalytics-cicd localrun -project <projectFullPath> [-outputPath <outputPath>] [-customCodeZipFilePath <zipFilePath>]| Parameter | Description |
|---|---|
-project |
The path of the asaproj.json file for your Visual Studio Code project or [Your project name].asaproj for Visual Studio project. |
-outputPath |
The path of the output folder. If it is not specified, the output result files will be placed in the current directory. |
-customCodeZipFilePath |
The path of the zip file for C# custom code, such as a UDF or deserializer, if they are used. Package the DLLs into a zip file and specify this path. |
azure-streamanalytics-cicd localrun -project "/Users/roger/projects/samplejob/asaproj.json"azure-streamanalytics-cicd localrun -project "/Users/roger/projects/samplejob/samplejob.asaproj"Note
JavaScript UDF only works on Windows.
You can use the CI/CD npm package to configure and run automated tests for your Stream Analytics script.
The test cases are described in a test configuration file. To get started, use the addtestcase command to add a test case template to the test configuration file. If the test configuration file doesn't exist, one is created by default.
azure-streamanalytics-cicd addtestcase -project <projectFullPath> [-testConfigPath <testConfigFileFullPath>]| Parameter | Description |
|---|---|
-project |
The path of the asaproj.json file for your Visual Studio Code project or [Your project name].asaproj for Visual Studio project. |
-testConfigPath |
The path of the test configuration file. If it is not specified, the file will be searched in \test under the current directory of the asaproj.json file, with default file name testConfig.json. A new file will be created if not existed. |
Note
The Script value in the generated testConfig.json file is only for providing the context; It's not used in the testing logic.
azure-streamanalytics-cicd addtestcase -project "/Users/roger/projects/samplejob/asaproj.json"azure-streamanalytics-cicd addtestcase -project "/Users/roger/projects/samplejob/samplejob.asaproj"If the test configuration file is empty, the following content is written into the file. Otherwise, a test case is added into the array of TestCases. Necessary input configurations are automatically filled according to the input configuration files, if they exist. Otherwise, default values are configured. FilePath of each input and expected output must be specified before running the test. You can modify the configuration manually.
{
"Script": "",
"TestCases": [
{
"Name": "Case 1",
"Inputs": [
{
"InputAlias": [Input alias string],
"Type": "Data Stream",
"Format": "JSON",
"FilePath": [Required],
"ScriptType": "InputMock"
}
],
"ExpectedOutputs": [
{
"OutputAlias": [Output alias string],
"FilePath": [Required],
"Required": true
}
]
}
]
}The following example shows two test cases, on a query using two inputs. If you want the test validation to ignore a certain output, set the Required field of that expected output to false. In this case, the FilePath property must not be empty, but doesn't need to be valid.
{
"Script": "C:\\...\\...\\samplejob.asaql",
"TestCases": [
{
"Name": "Case 1 - Testing all outputs at once",
"Inputs": [
{
"InputAlias": "entry",
"Type": "Data Stream",
"Format": "Json",
"FilePath": "C:\\...\\...\\Case1_Data_entry.json",
"ScriptType": "InputMock"
},
{
"InputAlias": "exit",
"Type": "Data Stream",
"Format": "Json",
"FilePath": "C:\\...\\...\\Case1_Data_exit.json",
"ScriptType": "InputMock"
}
],
"ExpectedOutputs": [
{
"OutputAlias": "output1",
"FilePath": "C:\\...\\...\\Case1_output1.json",
"Required": true
},
{
"OutputAlias": "output2",
"FilePath": "C:\\...\\...\\Case1_output2.json",
"Required": true
}
]
},
{
"Name": "Case 2 - Testing only one output at a time (recommended)",
"Inputs": [
{
"InputAlias": "entry",
"Type": "Data Stream",
"Format": "Json",
"FilePath": "C:\\...\\...\\Case2_Data_entry.json",
"ScriptType": "InputMock"
},
{
"InputAlias": "exit",
"Type": "Data Stream",
"Format": "Json",
"FilePath": "C:\\...\\...\Case2_Data_exit.json",
"ScriptType": "InputMock"
}
],
"ExpectedOutputs": [
{
"OutputAlias": "output1",
"FilePath": "C:\\...\\...\\Case2_output1.json",
"Required": true
},
{
"OutputAlias": "output2",
"FilePath": "[N/A]",
"Required": false
}
]
}
]
}Note
Currently, the only allowed value for the ScriptType element is InputMock, which is also the default value. If you set it to any other value, it's ignored and the default value (InputMock) is used.
You can use the following command to run multiple test cases for your project. A summary of test results is generated in the output folder. The process exits with code 0 for all tests passed; -1 for exception occurred; -2 for tests failed.
azure-streamanalytics-cicd test -project <projectFullPath> [-testConfigPath <testConfigFileFullPath>] [-outputPath <outputPath>] [-customCodeZipFilePath <zipFilePath>]| Parameter | Description |
|---|---|
-project |
The path of the asaproj.json file for your Visual Studio Code project or [Your project name].asaproj for Visual Studio project. |
-testConfigPath |
The path to the test configuration file. If it is not specified, the file will be searched in \test under the current directory of the asaproj.json file, with default file name testConfig.json. |
-outputPath |
The path of the test result output folder. If it is not specified, the output result files will be placed in the current directory. |
-customCodeZipFilePath |
The path of the zip file for custom code such as a UDF or deserializer, if they are used. |
As an example, in a PowerShell enabled terminal, if all test assets are located in a test subfolder of the project folder. With each test run output stored in a new timestamped subfolder of a testResults subfolder:
$projectPath = "C:\...\...\samplejob\"
azure-streamanalytics-cicd test `
-project ($projectPath + "asaproj.json") `
-testConfigPath ($projectPath + "test\testConfig.json") `
-outputPath ($projectPath + "test\testResults\$(Get-Date -Format "yyyyMMddHHmmss")\")When all tests are finished, a summary of the test results in JSON format is generated in the output folder. The summary file is named testResultSummary.json.
{
"Total": (integer) total_number_of_test_cases,
"Passed": (integer) number_of_passed_test_cases,
"Failed": (integer) number_of_failed_test_cases,
"Script": (string) absolute_path_to_asaql_file,
"Results": [ (array) detailed_results_of_test_cases
{
"Name": (string) name_of_test_case,
"Status": (integer) 0(passed)_or_1(failed),
"Time": (string) time_span_of_running_test_case,
"OutputMatched": [ (array) records_of_actual_outputs_equal_to_expected_outputs
{
"OutputAlias": (string) output_alias,
"ExpectedOutput": (string) path_to_the_expected_output_file,
"Output": (string) path_to_the_actual_output_file
}
],
"OutputNotEqual": [ (array) records_of_actual_outputs_not_equal_to_expected_outputs
{
"OutputAlias": (string) output_alias,
"ExpectedOutput": (string) path_to_the_expected_output_file,
"Output": (string) path_to_the_actual_output_file
}
],
"OutputMissing": [ (array) records_of_actual_outputs_missing
{
"OutputAlias": (string) output_alias,
"ExpectedOutput": (string) path_to_the_expected_output_file,
"Output": ""
}
],
"OutputUnexpected": [ (array) records_of_actual_outputs_unexpected
{
"OutputAlias": (string) output_alias,
"ExpectedOutput": "",
"Output": (string) path_to_the_actual_output_file
}
],
"OutputUnrequired": [ (array) records_of_actual_outputs_unrequired_to_be_checked
{
"OutputAlias": (string) output_alias,
"ExpectedOutput": (string) path_to_the_expected_output_file,
"Output": (string) path_to_the_actual_output_file
}
]
}
],
"Time": (string) time_span_of_running_all_test_cases,
}You can use the Azure Resource Manager template and parameter files generated from Build to deploy your job to Azure.