↓Skip to main content

Exporting an AMI to multiple formats

·427 words·3 mins

This is a short follow-up on my previous post.

In my previous post about creating an AMI with Image Builder, I exported that AMI to a different image format (vmdk).

There’s two things I ran into with that solution:

  • You can only export to a single other image format
  • There is no apparent correlation between the exported image and the original AMI

To work around those two items, I created an ECS task definition, which uses a container with QEMU installed, as well as Python and Boto3. The entrypoint for the container is a Python script to handle the image conversion.

I set up the Image Builder pipeline without the export to S3, and have it send a notification to SNS.

The notification triggers a Lambda function (for which I used Python), which checks the status (event['Records'][0]['Sns']['Message']['state']['status']) of the pipeline run. This will be either FAILED or AVAILABLE.

The SNS message also contains the AMI ID (event['Records'][0]['Sns']['Message']['outputResources']['amis'][0]['image']), pipeline version (event['Records'][0]['Sns']['Message']['version']), build version (event['Records'][0]['Sns']['Message']['buildVersion']) and the name of the Image Builder pipeline (event['Records'][0]['Sns']['Message']['name']).

These are used by the Lambda function, to run an ECS task, using the pre-defined task definition with these values as environment variables for the ECS task.

The ECS task uses the AMI ID to start an export, which takes time. How much, depends on the AMI you’re exporting. Since the export is initiated using Boto3, we get the export image task ID (ExportImageTaskId) in the response; this gives us the correlation we miss when we do the export using the distribution configuration of the pipeline. Once the export is done, you can pick up the exported image, and do additional conversions using the qemu-img command.

You can use the pipeline name, version and build version to name the images, so you can correlate them to the actual pipeline run that created the AMI you’ve created your export(s) from.

For exporting the images, the filesystem that’s assigned to the Lambda function is used. In the task definition you can override the storage size using the EphemeralStorage setting.

When converting to multiple image formats, you might want to consider using multiprocessing. That way you can upload an image as soon as it’s conversion is done, while also being able to start conversion for the next image format.

Once all the conversions are done, you could, once again, send out an SNS message to inform you of the status of the conversion process.

No code this time, but I did want to at least describe how you can run your custom process once the AMI is created.