Getting Started with Idem for Microsoft Azure

May 7

Author: Nicholas M. Hughes

I recently announced on LinkedIn that version 2.0 of idem-azurerm has been released. This article will hopefully clear up any vagaries from that post and give you a good understanding of how to start using this plugin to interact with your Azure environment.

A Brief History of Everything

Our story begins with the Salt open source project. Salt is a great solution to many of the problems IT organizations face today. Notably, it is very good at event-based orchestration, remote execution, and configuration management. One component of Salt is Salt Cloud. Salt Cloud was initially added to the code base fairly early in the history of public cloud computing, back when compute instances were really all you’d get out of a cloud provider… and that’s pretty much all Salt Cloud can still do to this day. It can create and destroy compute instances in public and private cloud providers such as AWS, Azure, VMware, OpenStack, and more.

That’s a nice feature, but it’s definitely lagging far behind other tools used for Infrastructure as Code deployments as providers have added countless objects and services to interact with. You’d still have to use some other mechanism to stand up the infrastructure in which the instances reside. For instance, standing up a virtual machine in Microsoft Azure requires a resource group, virtual network, and subnet at minimum. The current paradigm in Salt is to use state modules to interact with the infrastructure and then hand off to Salt Cloud for instance creation. All of that can live inside a single orchestration file, but it’s still not the most straightforward approach.

To further complicate matters, Salt Cloud breaks with the pattern followed by Salt state modules. Namely, Salt Cloud instance configurations are not idempotent. When you run a Salt Cloud deployment, it checks to see if the name of the instance is already present in the cloud. If it isn’t, then the deployment continues and instances are created. If it is, it just stops right there. It doesn’t check to see if you’ve added changed anything about the deployment. This means that you yet again have to use another mechanism to manage your instances after they’re deployed if you want to do anything more than destroy them.

Fast forward to Fall of 2019, when Thomas Hatch, creator of Salt and CTO of SaltStack announces Idem, a POP application which is a pluggable iteration of Salt’s state system. That’s probably an oversimplification, but it makes it easier to understand if you’re already familiar with Salt. So, Tom creates Idem and then asks if I’d like to port all of the Azure code I’ve written for Salt over to a plugin for Idem. Back in 2017, I overhauled the Azure provider for Salt Cloud and added all of the Azure execution and state modules that are in Salt today. Based upon that and the fact that I had recently taken over as the Cloud Working Group captain of the Salt open source community, I was in a pretty good position to be able to help out.

So, I hope this puts things in context. This is all still evolving, but the 2.0 version of the Azure provider for Idem represents feature parity with Salt Cloud in a format that’s much closer to “regular” Salt states plus much, much more!

A Simple Example

The Salt fans out there should recognize a lot the syntax here. Idem’s default input format is YAML + Jinja, just like Salt.

Installing Idem

Idem is a Python 3 application as are the plugins. In order to install the application we only have to look so far as pip. Following best practices, let’s use a virtual environment so we don’t clutter our system Python packages:

$ python3 -m venv idemenv

$ source idemenv/bin/activate

(idemenv) $ pip3 install idem-azurerm==2.0.0

This creates a virtual Python 3 environment, enters that environment, and then installs idem-azurerm… which automatically installs the required pop, idem, and Azure SDK libraries from pip. That’s it!

Authentication

An authentication plugin is currently in the works for Idem, so the current mechanism for passing in authentication credentials is via the YAML definition for each object. In the future, this will all be more secure and transparent to the user.

Here’s an example of using service principal credentials for Idem:

#!yaml

Ensure resource group exists:
  azurerm.resource.group.present:
    - name: idem
    - location: eastus
    - tags:
        Owner: Elmer Fudd Gantry
        Organization: Everest
    - connection_auth:
        client_id: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
        secret: "X2KRwdcdsQn9mwjdt0EbxsQR3w5TuBOR"
        subscription_id: "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"
        tenant: "cccccccc-cccc-cccc-cccc-cccccccccccc"

That works fine for a single resource, but duplicating those parameters will quickly get tedious. Luckily, we can use Jinja! The Jinja variable can be passed to all of the resources, so your credentials only need to be defined in one place. You can even use different credentials to create/modify different resources in your stack.

#!jinja|yaml

{% set profile = {
        "client_id": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
        "secret": "X2KRwdcdsQn9mwjdt0EbxsQR3w5TuBOR",
        "subscription_id": "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",
        "tenant": "cccccccc-cccc-cccc-cccc-cccccccccccc"
    }
%}

Ensure resource group exists:
  azurerm.resource.group.present:
    - name: idem
    - location: eastus
    - tags:
        Owner: Elmer Fudd Gantry
        Organization: Everest
    - connection_auth:  {{ profile }}

Bare Minimum for a Virtual Machine

So, assuming we now have authentication worked out… how can we create a virtual machine and some required resources?

The YAML below creates a resource group, a virtual network, a subnet inside of the virtual network, a network interface (automagically), a public IP (because we told allocate_public_ip to do the work for us), and finally a virtual machine with associated operating system and data disks.

We can totally create the network interface and public IP as separate objects (and probably should), but we’re taking the easy road for the purposes of this demo. The network interface and public IP creation was a nice feature of Salt Cloud that made its way over to the Idem provider for lazy people like me. Note that we can’t enforce any parameters on those objects when created this way, but it’s an “easy button” feature for quick and dirty resource creation.

Put this in a file called mytest.sls:

Ensure resource group exists:
  azurerm.resource.group.present:
    - name: "idem"
    - location: "eastus"
    - tags:
        Owner: "Elmer Fudd Gantry"
        Organization: "Everest"
    - connection_auth:  {{ profile }}

Ensure virtual network exists:
  azurerm.network.virtual_network.present:
    - name: "vnet-dev-eastus-001"
    - resource_group: "idem"
    - address_prefixes:
        - "10.0.0.0/8"
    - subnets:
        - name: "default"
          address_prefix: "10.0.1.0/24"
    - tags:
        Owner: "Elmer Fudd Gantry"
        Organization: "Everest"
    - connection_auth: {{ profile }}

Ensure virtual machine exists:
  azurerm.compute.virtual_machine.present:
    - name: "vmidem001"
    - resource_group: "idem"
    - vm_size: "Standard_B2S"
    - image: "OpenLogic|CentOS|7.8|latest"
    - virtual_network: "vnet-dev-eastus-001"
    - subnet: "default"
    - allocate_public_ip: True
    - ssh_public_keys:
        - "/home/nmhughes/.ssh/id_rsa.pub"
    - data_disks:
        - disk_size_gb: 100
    - tags:
        Owner: "Elmer Fudd Gantry"
        Organization: "Everest"
    - connection_auth: {{ profile }}

Then you just need to run idem against our SLS file (note the dropped file extension):

(idemenv) $ idem --sls mytest

Once the command completes, you should have a brand new virtual machine in Azure along with all of the infrastructure required to run it! In our code above, the resources are created either in parallel or serially from top to bottom on the page depending on the idem run mode. If in parallel, a pretty cool feature called “transparent requisites” ensures that resources are created in the proper order. For instance, resource groups are required for all objects in Azure. If we listed the virtual network in the file before the resource group, the virtual network creation would fail since the resource group isn’t there to put it in. Luckily, there is a transparent requisite in the idem-azurerm code which dictates that all resource groups will be created before any other objects. Likewise, a virtual machine would have requisites for virtual networks, resource groups, etc. No more explicitly requiring states like in Salt!

Changing tags or any other modifiable attributes will result in changes in to the objects in the cloud on subsequent runs. Idem is idempotent!

A Promise of More

More what? More EVERYTHING! We’re constantly adding new features to idem-azurerm, so stay tuned. Also, it’s open source… so we’d love to have your feedback and code donations in order to make it even better in the future! Tom and other folks are also constantly enhancing POP and Idem, so definitely check out those projects as well. POP is a great new way to write pluggable Python applications. I definitely suggest checking out SaltStack’s Twitch stream and the POP book that Tom put out on Read the Docs for more information.

Finally, stay tuned for more content on this Azure provider from me in the coming weeks. I’m actively working on more uses to showcase and different ways to consume the content so it’s understandable. Happy “clouding” and I hope to see you back here again soon!

Nicholas Hughes helps businesses integrate cloud and cybersecurity automation into their IT lifecycle processes in order to amplify the efforts of the existing workforce, prevent employee burnout, and free them to do the more important tasks that might be currently neglected. As part of his daily duties as a founding partner and CEO of EITR Technologies LLC., he’s responsible for all of those super awesome elements of the CEO job that you read about as a kid, like setting the strategic direction of the company and modeling corporate values. Additionally, Nick still performs technical consulting work with specializations in Automation & Orchestration, Cloud Infrastructure, Cloud Security, and Systems Architecture. He has over 15 years of experience in a wide breadth of roles within Information Technology, which is invaluable to clients seeking comprehensive technical solutions to business problems. Nick highly values pragmatism, logical thinking, and integrity in both his business and personal life… which is a decidedly boring set of core values that reap great results when applied to the task at hand. He also has a wonderful wife and two boys who keep him on his toes.

Guest User