Azure ML environment with custom docker image for training and deployment

Hello everyone,

today i would like to talk about Azure ML Environments with custom Docker images and the pitfalls which may arise.

Introduction

In any machine learning project, you encounter the challenge to ensure an consistent environment for your experiments, training and deployments. Differences in the environments are likely to produce unexpected results. Furthermore you loose your predictability and reproducibility.

As a solution, Azure ML proposes Environments. These work quite well if you have a standard setup like scikit-learn.
In Addition, if you can use an predefined environment provisioned by Azure you should be fine.

However, your environment may differ due to special dependencies or exotic machine learning frameworks.

Obstacles

So, you are now ready to create your own environment for training and deployment, aren’t you?

• Write an simple docker file
• Look up the documentation for Azure ML Environments
• Add something like this to your machine learning pipeline

from azureml.core import Workspace, Environment

ws = Workspace.from_config()
myenv= Environment.get(workspace=ws, name="my-environment")

myenv.docker.base_image="base-image"
myenv.docker.base_image_registry="url-to-base-image"
myenv.python.user_managed_dependencies=TrueCode language: JavaScript (javascript)

At least I thought that…

For my deployment, this worked quite well… because i didn’t use the Azure ML Environment at all.
The Deployment was purely based on another Docker file on top of the environment base image.
The problem arised as i tried to execute some training code on the Azure ML Compute Targets with my custom Azure ML Environment.

For me, all my runs crashed with an exception, far away from my code and the following error:

object x has no attribute 'time'Code language: JavaScript (javascript)

MPI to the resuce

So I digged a bit deeper: Your custom docker image should have MPI installed to work on the Azure ML Compute Targets.
This knowledge was, at least for me, hidden in this repository:
https://github.com/Azure/AzureML-Containers

After adding

FROM mcr.microsoft.com/azureml/base:openmpi3.1.2-ubuntu18.04

to my docker file for my Azure ML Environment, everything worked as expected!

Mind the gap between miniconda and pip

The base image provided by Microsoft uses miniconda as its python distribution.
If you use pip for your dependency management:
Keep in mind to prefix your calls to pip with “python -m” to ensure that the right version of pip is used.

RUN python -m pip install -r requirements.txtCode language: CSS (css)

The End

That’s all for today, folks. I hope, i could give you little guidance for using a custom docker file for your Azure ML Environment.

Appendix

As requested, the docker file, used for the custom environment:

FROM mcr.microsoft.com/azureml/base:openmpi3.1.2-ubuntu18.04

RUN apt-get update

# https://github.com/Azure/MachineLearningNotebooks/issues/1110
RUN python -m pip install --upgrade pip==20.1.1

COPY requirements.txt .

RUN python -m pip install --no-cache-dir -r requirements.txtCode language: JavaScript (javascript)