Welcome to the Nightingale HQ overview about Azure Batch services.
Here we aim to introduce people to what they need to know.
Azure Batch is a Microsoft Azure product. Read more about Microsoft Azure here.
Azure Batch is a robust service that provides parallel batch processing to execute intensive workloads of varying size. It creates a pool of compute nodes (virtual machines) to tackle heavy loads. With Azure Batch, batch processing has become more streamlined and viable to complete data-intensive workloads at any scale.
Azure Batch is a service that manages the workload of applications. A workload is the work assigned to an application over a given time period. Sometimes the workload of an application is greater than it can handle in that time period, for example if it needs to process vast volumes of data, and this can lead to slow processing time, crashes, and expensive server costs if you use a cloud-based server. Azure Batch is designed to take the workload that is greater than the capability of your application, and divide it between a number of nodes - virtual machines (VMs) - that can each run your application and perform different parts of the workload in parallel. It is like asking a group of people to each make a car component and bring them together, instead of asking one person to build a car on their own. Each node performs a task that is a subset of the overall workload. Azure Batch can create the nodes required to complete the workload, assign tasks to them, schedule the tasks, get the data it needs from your storage solution and pass it to the nodes, and scale the number of nodes to suit your budget and timescale.
You may need Azure Batch if:
Azure Batch lists the following example use-cases:
Benefits of Azure Batch include:
To get started with Azure Batch, you need a Microsoft Azure account. Read more about Microsoft Azure here. You also need to create a Batch Account. The Batch Account is used to authenticate your application when you run tasks. You can create a Batch Account here.
Most Batch solutions use Azure Storage for storing resource files and output files. You can associate a storage account with your Batch account when you create the Batch account, or later. You can create an Azure Storage Account here.
One of the benefits of Azure Batch is that you can choose the operating system and developer tools that you use to run workloads. Nodes running Windows will accept Windows code, including Microsoft .NET, and with Linux there is a choice of distrbutions including CentOS, Ubuntu, and SUSE Linux Enterprise Server. When creating a pool of nodes, you can set it up to run tasks in Docker containers.
Nodes can run any executable or script that is supported by the operating system environment of the node. Executables or scripts include *.exe, *.cmd, *.bat, PowerShell scripts for Windows, and binaries, shell, and Python scripts for Linux.
One Batch is up and running you can monitor your applications via the Azure portal, the Batch Explorer tool, or from the command line.
Azure Batch is built on Microsoft Azure security infrastructure and uses all Microsoft Azure security measures. You can read more about Microsoft Azure security here.
All compute nodes in Azure Batch have configurable Firewall settings. When you create a pool of nodes they operate in isolation from other pools, so data is not processed or transported uneccesarily.
Microsoft Azure carries an impressive list of compliance certifications which you can view here.
Azure Batch is a pay-as-you-go service so you only pay for what you use, and there are no up-front set-up fees. Read more about Microsoft Azure and its pricing structure here.
There’s no charge for Batch itself, only the underlying compute and other resources consumed to run your batch jobs, including applicable software licence costs. You are billed by the second of compute resources, and can choose the compute power and storage of the nodes you run to suit your budget. You can also choose between low-priority and high-priority virtual machines to further manage the per-second cost of computing. Reserved pricing is available for Azure Batch.
You can read more about Azure Batch pricing here.
If you already use Azure storage and compute services, Azure Batch is an obvious choice to start job scheduling and parallel process management. However, if your storage is elsewhere or you are not familiar with Azure services you may find an alternative solution suits your organisation better. Examples of alternative job scheduling and compute management services are: