Skip to content

Files

Latest commit

lauradolanlauradolan
Fix title
Jan 12, 2022
c84bb05 · Jan 12, 2022

History

History
168 lines (118 loc) · 11.3 KB

batch-job-prep-release.md

File metadata and controls

168 lines (118 loc) · 11.3 KB
title description ms.topic ms.date ms.devlang ms.custom
Create tasks to prepare and complete jobs on compute nodes
Make job-level preparation tasks to minimize data transfer to Azure Batch compute nodes, and release tasks for node cleanup at job completion.
how-to
01/12/2021
csharp
seodec18, devx-track-csharp

Create tasks to prepare and complete jobs on Batch compute nodes

An Azure Batch job often requires some form of setup before its tasks are executed. It also may require post-job maintenance when its tasks are completed. For example, you might need to download common task input data to your compute nodes, or upload task output data to Azure Storage after the job completes. You can use job preparation and job release tasks to perform these operations.

What are job preparation and release tasks?

Before a job's tasks run, the job preparation task runs on all compute nodes scheduled to run at least one task. Once the job is completed, the job release task runs on each node in the pool that executed at least one task. As with normal Batch tasks, you can specify a command line to be invoked when a job preparation or release task is run.

Job preparation and release tasks offer familiar Batch task features such as file download (resource files), elevated execution, custom environment variables, maximum execution duration, retry count, and file retention time.

In the following sections, you'll learn how to use the JobPreparationTask and JobReleaseTask classes found in the Batch .NET library.

Tip

Job preparation and release tasks are especially helpful in "shared pool" environments, in which a pool of compute nodes persists between job runs and is used by many jobs.

When to use job preparation and release tasks

Job preparation and job release tasks are a good fit for the following situations:

  • Downloading common task data: Batch jobs often require a common set of data as input for the job's tasks. For example, in daily risk analysis calculations, market data is job-specific, yet common to all tasks in the job. This market data, often several gigabytes in size, should be downloaded to each compute node only once so that any task that runs on the node can use it. Use a job preparation task to download this data to each node before the execution of the job's other tasks.

  • Job and task output deletion: In a "shared pool" environment, where a pool's compute nodes are not decommissioned between jobs, you may need to delete job data between runs. You might need to conserve disk space on the nodes, or satisfy your organization's security policies. Use a job release task to delete data that was downloaded by a job preparation task, or that was generated during task execution.

  • Log retention: You might want to keep a copy of log files that your tasks generate, or perhaps crash dump files that can be generated by failed applications. Use a job release task in such cases to compress and upload this data to an Azure Storage account.

Job preparation task

Before executing tasks in a job, Batch runs the job preparation task on each compute node scheduled to run a task. By default, Batch waits for the job preparation task to complete before running the tasks scheduled to execute on the node. However, you can configure the service not to wait. If the node restarts, the job preparation task runs again. You can also disable this behavior. If you have a job with a job preparation task and a job manager task configured, the job preparation task runs before the job manager task, just as it does for all other tasks. The job preparation task always runs first.

The job preparation task is executed only on nodes that are scheduled to run a task. This prevents the unnecessary execution of a preparation task in case a node is not assigned any tasks. This can occur when the number of tasks for a job is less than the number of nodes in a pool. It also applies when concurrent task execution is enabled, which leaves some nodes idle if the task count is lower than the total possible concurrent tasks.

Note

JobPreparationTask differs from CloudPool.StartTask in that JobPreparationTask executes at the start of each job, whereas StartTask executes only when a compute node first joins a pool or restarts.

Job release task

Once a job is marked as completed, the job release task runs on each node in the pool that executed at least one task. You mark a job as completed by issuing a terminate request. This request sets the job state to terminating, terminates any active or running tasks associated with the job, and runs the job release task. The job then moves to the completed state.

Note

Deleting a job also executes the job release task. However, if a job has already been terminated, the release task is not run a second time if the job is later deleted.

Jobs release tasks can run for a maximum of 15 minutes before being terminated by the Batch service. For more information, see the REST API reference documentation.

Job prep and release tasks with Batch .NET

To use a job preparation task, assign a JobPreparationTask object to your job's CloudJob.JobPreparationTask property. Similarly, to use a job release task, initialize a JobReleaseTask and assign it to your job's CloudJob.JobReleaseTask.

In this code snippet, myBatchClient is an instance of BatchClient, and myPool is an existing pool within the Batch account.

// Create the CloudJob for CloudPool "myPool"
CloudJob myJob =
    myBatchClient.JobOperations.CreateJob(
        "JobPrepReleaseSampleJob",
        new PoolInformation() { PoolId = "myPool" });

// Specify the command lines for the job preparation and release tasks
string jobPrepCmdLine =
    "cmd /c echo %AZ_BATCH_NODE_ID% > %AZ_BATCH_NODE_SHARED_DIR%\\shared_file.txt";
string jobReleaseCmdLine =
    "cmd /c del %AZ_BATCH_NODE_SHARED_DIR%\\shared_file.txt";

// Assign the job preparation task to the job
myJob.JobPreparationTask =
    new JobPreparationTask { CommandLine = jobPrepCmdLine };

// Assign the job release task to the job
myJob.JobReleaseTask =
    new JobReleaseTask { CommandLine = jobReleaseCmdLine };

await myJob.CommitAsync();

As mentioned earlier, the release task is executed when a job is terminated or deleted. Terminate a job with JobOperations.TerminateJobAsync. Delete a job with JobOperations.DeleteJobAsync. You typically terminate or delete a job when its tasks are completed, or when a timeout that you've defined has been reached.

// Terminate the job to mark it as completed; this will initiate the
// job release task on any node that executed job tasks. Note that the
// job release task is also executed when a job is deleted, so you don't
// have to call Terminate if you delete jobs after task completion.

await myBatchClient.JobOperations.TerminateJobAsync("JobPrepReleaseSampleJob");

Code sample on GitHub

To see job preparation and release tasks in action, check out the JobPrepRelease sample project on GitHub. This console application does the following:

  1. Creates a pool with two nodes.
  2. Creates a job with job preparation, release, and standard tasks.
  3. Runs the job preparation task, which first writes the node ID to a text file in a node's "shared" directory.
  4. Runs a task on each node that writes its task ID to the same text file.
  5. Once all tasks are completed (or the timeout is reached), prints the contents of each node's text file to the console.
  6. When the job is completed, runs the job release task to delete the file from the node.
  7. Prints the exit codes of the job preparation and release tasks for each node on which they executed.
  8. Pauses execution to allow confirmation of job and/or pool deletion.

Output from the sample application is similar to the following:

Attempting to create pool: JobPrepReleaseSamplePool
Created pool JobPrepReleaseSamplePool with 2 nodes
Checking for existing job JobPrepReleaseSampleJob...
Job JobPrepReleaseSampleJob not found, creating...
Submitting tasks and awaiting completion...
All tasks completed.

Contents of shared\job_prep_and_release.txt on tvm-2434664350_1-20160623t173951z:
-------------------------------------------
tvm-2434664350_1-20160623t173951z tasks:
  task001
  task004
  task005
  task006

Contents of shared\job_prep_and_release.txt on tvm-2434664350_2-20160623t173951z:
-------------------------------------------
tvm-2434664350_2-20160623t173951z tasks:
  task008
  task002
  task003
  task007

Waiting for job JobPrepReleaseSampleJob to reach state Completed
...

tvm-2434664350_1-20160623t173951z:
  Prep task exit code:    0
  Release task exit code: 0

tvm-2434664350_2-20160623t173951z:
  Prep task exit code:    0
  Release task exit code: 0

Delete job? [yes] no
yes
Delete pool? [yes] no
yes

Sample complete, hit ENTER to exit...

Note

Due to the variable creation and start time of nodes in a new pool (some nodes are ready for tasks before others), you may see different output. Specifically, because the tasks complete quickly, one of the pool's nodes may execute all of the job's tasks. If this occurs, you will notice that the job prep and release tasks do not exist for the node that executed no tasks.

Inspect job preparation and release tasks in the Azure portal

You can use the Azure portal to view the properties of the job and its tasks. After you run the sample application, you can also download the shared text file that is modified by the job's tasks.

The screenshot below shows the Preparation tasks blade in the Azure portal. Navigate to the JobPrepReleaseSampleJob properties after your tasks have completed (but before deleting your job and pool) and click Preparation tasks or Release tasks to view their properties.

:::image type="content" source="media/batch-job-prep-release/portal-jobprep-01.png" alt-text="Screenshot showing job preparation task properties in the Azure portal.":::

Next steps