Airgapped Testing? Yes, please!

Recently, I got a bug report for the kitchen-vcenter driver, which allows lifecycle management of testing VMs on VMware vCenter environments. Apparently, a customer tried to create a VM without any network interface.

The problem was that this crashed in a very unintuitive way. But it made me wonder: Would it be possible to use non-networked machines for tests? Turns out: That’s absolutely possible!

First off, this is probably not what you want to do. At least some degree of software installation is likely to take place - and without network you are a bit limited in that regard. But maybe there are some valid usecases in your project?

The kitchen-vcenter bugfix version 2.11.4 replaces the noisy crash with a nice warning, that this VM will not have any network access.

Create a Non-Network Machine

For finishing creation of a VM, the kitchen-vcenter driver expects some IP address to be returned.

To set a static IP address for detection only, you need to set a dummy device. Be careful not to assign a “link-local” address1, because this usually is an indicator for a non-functioning DHCP server and the driver will simply warn you about this issue and not continue.

The following /etc/netplan/01-netcfg.yaml will create a dummy bridge including a private IP address2:

network:
  version: 2
  renderer: networkd
  bridges:
    dummy0:
      dhcp4: no
      dhcp6: no
      accept-ra: no
      interfaces: []
      addresses:
        - 192.0.2.1/24

Now we set the minimum configuration for the driver in kitchen.yml and can run kitchen create successfully:

driver:
  name: vcenter
  vcenter_host: vcsa.lab.local
  vcenter_username: administrator@vsphere.local
  vcenter_password: "..."
  vcenter_disable_ssl_verify: true
  datacenter: "Datacenter"

Transfer Files and Execute Commands

I can imagine people wondering about the purpose of this. We can provision (and deprovision) machines now which can only be reached via the vCenter web console. But for testing with Test Kitchen, files need to be transferred and commands need to be executed.

Well, as I described in a post back then about Instant Clones and VMware Guest Operations that is not a problem. VMware comes with a handy sidechannel for all that, as it needs to be able to inspect machines even without using their network interfaces - how else would you be able to use the Web console anyway, right?

This is done via a combination of RPC calls and mapped memory regions, handily providing API functionality to execute commands and transfer files (with proper OS authentication, though!).

The kitchen-vcenter driver actually uses functionality like that for it’s Active IP Discovery feature and it got ported into the train-vsphere-gom transport as well.

Even though Test Kitchen lent inspiration to the Train framework, there is official glue to allow connecting both pieces of software. But, there is the inofficial kitchen-transport-train which provides an adapter to use any Train transport with Test Kitchen. Including exotic ones like Telnet, Serial/USB and the AWS Systems Manager.

Install both via chef gem install train-vsphere-gom kitchen-transport-train, then configure your kitchen.yml section for file transfer/execution:

transport:
  name: train
  backend: vsphere-gom
  username: root
  password: "..."
  vcenter_host: vcsa.lab.local
  vcenter_username: administrator@vsphere.local
  vcenter_password: "..."

Notice that this does not provide the kitchen login command. But it also means that you can use Test Kitchen VMs even without routing/firewalling!

Converging

Now that we can provision non-networked machines and connect to them, you might start thinking on the provisioner. As we don’t want to transfer the whole Chef Infra packages over the transport, I recommend preinstalling it inside the VMware templates.

Alternatively, you can toy around with something like the shell provisioner which is included in core Test Kitchen:

provisioner:
  name: shell
  command: "uname -a"

Interactive access

Debugging might be tricky without any access to the machine or merely via the vCenter console. But there are alternatives:

govc is a tool mananged by VMware themselves, which allows most VMware-related tasks to be done on the command line. It also includes support for running commands via guest.run3.

GOVC_URL=https://...:...@vcsa.lab.local/sdk GOVC_INSECURE=1 govc guest.run -vm my-vm "uname -a"

But what if you want interactive access? Well, I was in that situation a while ago when a customer had their access to development machines closed shut. After jumping through hoops for a while, I decided to write an interactive shell which uses Train to connect to remote systems: TrainSH.

TrainSH can

  • use all installed Train transports (including train-vsphere-gom)
  • open multiple sessions in parallel, even with different backends
  • interactively execute commands
  • upload and download files
  • locally view and edit remote files, for example with vim
  • copy files between active sessions

VI_USERNAME="..." VI_PASSWORD="..." VI_SERVER=vcsa.lab.local trainsh connect vsphere-gom://root:...@my-vm

Have fun exploring machines without network connectivity!


  1. Link-local/APIPA addresses are specified in RFC 3927 ↩︎

  2. This address is from the non-routed documentation space in RFC 5737 ↩︎

  3. There are a lot more commands in the guest.* space of govc, see its USAGE ↩︎

Similar Posts You Might Enjoy

Update your Style in Test Kitchen (Part 2)

It is time for a follow-up to my blog post from last year - especially as Test Kitchen 3.0 changed some defaults. Let’s check some cargo-culted settings out in this blog post. - by Thomas Heinen

Third Party Platform Support for Chef

Currently, users of Chef are limited to working with the platforms supported by the Chef core. But with the advancements of Target Mode, the story could change quickly. Learn about the idea of a “Platform Support Pack” and what it could mean for the future. - by Thomas Heinen

Chef Technology Partner of the Year

To our pleasant surprise, Chef awarded tecRacer the “2021 Technology Partner of the Year” title at the opening keynote of ChefConf 2021. - by Thomas Heinen