I would love to be able to plug'n'plug my spare (and distributed) linux machines into a company cloud and have them be available for ready provisioning/unprovisioning automatically (including e.g. auto detection of RAM, SSD, GPUs). Lets say I have about $8k worth of used but decent devices with enough specs for many of our companies current usage of VMs, and might be able to reduce our cloud bill by $5k to $electricity+extra work cost a month. I listened to the DHH "Leaving the Cloud" podcast and it's inspired me. Realistically we probably won't do databases since the value of the cloud is a bit higher there than for webservers. We don't have spikes in demand that cloud would help with either.
I had a look at OpenStack but they don't seem to support this - surely big companies find a benefit in this for some data privacy focused applications - are there any good tools?
In a dream there is a command line to join the network like
selfhostedcloudprovisioner --key='mypem.key'
and I could just start using the machine to launch a virtual machine (not just SSH tunneling) from another machine and network with it, to run a webserver, database, join kuberenetes pod, whatever.
Not that I am aware of. The closest would be VMWare which is rather expensive and using their lab modules. You've tried OpenStack and that is about the closest it gets in terms of free open source and bare metal provisioning of VM platforms. There are other paid tools to handle power-on provisioning on new fleets of servers but most of them are still in early development and still require a good deal of in-house development to tie everything together.