Airgapped Testing - VMware Edition

Recently, I got a bug report for the kitchen-vcenter driver, which allows lifecycle management of testing VMs on VMware vCenter environments. Apparently, a customer tried to create a VM without any network interface.

The problem was that this crashed in a very unintuitive way. But it made me wonder: Would it be possible to use non-networked machines for tests? It turns out: That’s absolutely possible!

First off, this is probably not what you want to do. At least some degree of software installation is likely to take place - and without network access, you are a bit limited in that regard. But maybe there are some valid use cases in your project?

The kitchen-vcenter bugfix version 2.11.4 replaces the noisy crash with a friendly warning that this VM will not have network access.

Create a Non-Network Machine

To finish the creation of a VM, the kitchen-vcenter driver expects some IP addresses to be present.

To set a static IP address for detection only, you need to create a dummy device. Be careful not to assign a “link-local” address¹ because this is usually an indicator for a DHCP-related problem. In this case, the driver will warn you about this issue and not continue.

The following /etc/netplan/01-netcfg.yaml will create a dummy bridge including a private IP address²:

network:
  version: 2
  renderer: networkd
  bridges:
    dummy0:
      dhcp4: no
      dhcp6: no
      accept-ra: no
      interfaces: []
      addresses:
        - 192.0.2.1/24

Now we set the minimum configuration for the driver in kitchen.yml and can run kitchen create successfully:

driver:
  name: vcenter
  vcenter_host: vcsa.lab.local
  vcenter_username: administrator@vsphere.local
  vcenter_password: "..."
  vcenter_disable_ssl_verify: true
  datacenter: "Datacenter"

Transfer Files and Execute Commands

I can imagine people wondering about the purpose of this. We can provision (and de-provision) machines now, which are only accessible via the vCenter web console. But for testing with Test Kitchen, files need to be transferred, and commands need some way of execution.

Well, as I described in a post back then about Instant Clones and VMware Guest Operations that is not a problem. VMware comes with a handy side-channel for all that, as it needs to inspect machines even without using their network interfaces - how else would you be able to use the Web console in the end?

This type of access is done via a combination of RPC calls and mapped memory regions, handily providing API functionality to execute commands and transfer files (with proper OS authentication, though!).

The kitchen-vcenter driver also uses this functionality for its Active IP Discovery feature, and it got ported into the train-vsphere-gom transport.

Even though Test Kitchen lent inspiration to the Train framework, there is no official way to connect both software pieces. But, the inofficial kitchen-transport-train provides an adapter to use any Train transport with Test Kitchen. This includes exotic ones like Telnet, Serial/USB and AWS Systems Manager.

Install both via chef gem install train-vsphere-gom kitchen-transport-train, then configure your kitchen.yml section for file transfer/execution:

transport:
  name: train
  backend: vsphere-gom
  username: root
  password: "..."
  vcenter_host: vcsa.lab.local
  vcenter_username: administrator@vsphere.local
  vcenter_password: "..."

Notice that this does not provide the kitchen login command. But it also means that you can use Test Kitchen VMs even without routing/firewalling!

Converging

You might start thinking about the provisioner now that we can provision non-networked machines and connect to them. We don’t want to transfer the whole Chef Infra packages over the transport, so I recommend preinstalling it inside the VMware templates.

Alternatively, you can toy around with something like the shell provisioner, which is part of Test Kitchen itself:

provisioner:
  name: shell
  command: "uname -a"

Interactive access

Debugging might be tricky without any access to the machine or merely via the vCenter console. But there are alternatives:

govc is a tool managed by VMware themselves, which enables performing most VMware-related tasks via command line. It also includes support for running commands via guest.run³.

GOVC_URL=https://...:...@vcsa.lab.local/sdk GOVC_INSECURE=1 govc guest.run -vm my-vm "uname -a"

But what if you want interactive access? Well, I was in this specific situation a while ago when a customer had access to development machines limited too much. After trying different things for some time, I decided to write an interactive shell based on Train to connect to remote systems: TrainSH.

TrainSH can

• use all installed Train transports (including train-vsphere-gom)
• open multiple sessions in parallel, even with different backends
• interactively execute commands
• upload and download files
• locally view and edit remote files, for example, with vim
• copy files between active sessions

VI_USERNAME="..." VI_PASSWORD="..." VI_SERVER=vcsa.lab.local trainsh connect vsphere-gom://root:...@my-vm

Have fun exploring machines without network connectivity!