How do you manage VMs on your workstations?

Preamble:

I am yet to find a structured, comfortable and sustainable way to manage local VMs. I'm usually falling back to just Virtualbox with the disk file being a reoccurring annoyance and liability. I must have 100s GB of files scattered around in old backups where I'm not sure what's inside. Reproducibility and efficient use of resources is an issue. I'd also like to be quickly set up and tear down VMs a - much like Qubes OS.

I imagine many of you have similar scenarios, which is prompting this post.

Some use-cases could include:

  * Build environment for large projects with relatively complex pipelines where I have multiple on current configurations and checkouts
    * Examples: Various Linux dists, AOSP forks, embedded projects
  * Persisted isolated "persona" environments for e.g. web browsing and development, routing all its traffic through some particular gateway or proxy
  * Work environments for recording screencasts/demos
  * Pentesting
  * (Semi-) permanent services with exposed ports
  * Working on a VM remotely from a laptop (where the laptop is on Wayland)

I already host a lot of stuff on actual servers (tools include: Linux,Ansible,Terraform,Kubernetes/Nomad,Consul). While I'd love to use Terraform here, I'm not really intending to bring the machine into a cluster or anything like that.

Some requirements and wishes:

  * It should be possible to somehow manage the entire configuration in a git repo
  * Declarative configuration
  * I can interact with GUI applications (e.g. disposable browser window)
  * Selectively share Wayland clipboard
  * Reasonably straightforward routing config (e.g. "route all connections on this VM through whonix")
  * Some form of remote desktop
  * VM has its own kernel
  * Portable: If I bring the configuration and artefacts it should be straightforward to migrate to a new host OS.
  * Composability / inheritance for environments.

A lot of this is close to QubesOS, except:

  * I want Wayland. Qubes will take its time to get there.
  * More in general, I suspect qubes dom0 is not really configurable to behave like I want without significant patches to the point of becoming a new distro.
  * Still find maintaining Qubes and keeping updated has not been an ideal experience.
  * Not portable - at this point I'm not looking for a new host OS.
  * Not sure how to set up the remoting

As for Vagrant, it checks a lot of the boxes (heh). I don't have a ton of experience with it but been using it in projects every now and then and wrote my own Vagrantfiles maybe a handful times. It does seem perfect for dev or build workspaces - in practice I usually use Dockerfiles here.

libvirt has a variety of management interfaces and built-in remote support.

Did do some basic testing with existing Terraform providers for libvirt, virtualbox, and vagrant - the libvirt one seems mature enough.

So for now my candidate pieces here:

  libvirtd
  qemu/kvm
  virtualbox
  vagrant
  terraform
  kvm
  whonix
  packer

Now, while I have at least some rudimentary experience with a lot pof these, I am a bit at a loss for the best way to tie them together and which I even want in the mix. For example, one possible configuration would be:

Terraform -> terraform-provider-vagrant -> vagrant -> vagrant-libvirt -> libvirt -> qemu -> kvm (image built with packer-qemu)

This seems kind of silly but gets to my point: How do you do it?

On Arch Linux BTW.`