What Are You Working On? (September 2022)

Still working on growing my newsletter[0] which funnily enough is about ideas.

I’m also working on a proxy for the Mailgun API (as in take Mailgun requests and convert them to something else) —- finally getting the chance to do some more Rust which I haven’t in a while!

[0]: https://unvalidatedideas.com

Not so much "work" per se, but something that I've been thinking about quite a lot recently is how ML practitioners test their systems.

For example, before launching a model to prod, I've commonly seen that teams have a suite of datasets/evals, and a model has to be at least as good as the existing model in production before replacing it.

But what about the training pipelines? You can have unit tests for all the individual components, but the analogy of "integration tests" would involve just training a full model, which might be very expensive for large neural networks.

One option would be to have a special tiny dataset and tiny model that can be trained fast on CPU. But this would be an imperfect test of the real system. I've also never seen this in practice, so maybe it wouldn't actually be useful.

Curious if anyone can chime in with their experiences?

I've always found the OpenSSL tools painful for managing internal self-signed certificates. At work we make fairly heavy use of them, and are starting to make even heavier use. Our use is more than EasyRSA can provide. So I've been working on a new CA tool:

https://github.com/linsomniac/rgca

In a nod to OpenSSL config files, it can take almost all values: from the command line, from the environment, or from one or more config files. It also allows "pre" and "post" commands so you can run a script after generating the cert, for example for server certs I have a "post" script that will copy it into the appropriate location in the Ansible repo, encrypt the key file, and commit it all.

I still need to implement a "renew" which will take an existing cert, update the expiration date, but also allow adding/removing SANs, possibly other features. But I've been using it to generate all our certs recently and it's working great.

Hardware: I've got a cheap ($200) CNC mill I'm getting ready to use for various projects.

Software: MSTOICAL[1] is a port of a port of STOIC, a Forth like system that does types in the stack, and is weirder, but more logical, than Forth. It's written in C, a language I'm learning as I go.

Software: TreeHouse[2] is an exploration of the ideas show off by DION in this 2020 demo.[3] The direct manipulation of a program as an AST, with the source code being an OUTPUT, a view, of the program. I'm doing it in Pascal, because that's what I'm used to. It's amazing how cool it looks already, even though it's just a toy so far.

[1] https://github.com/mikewarot/mstoical

[2] https://github.com/mikewarot/TreeHouse

[3] https://media.handmade-seattle.com/dion-systems/ (watch the 1st video!)

I've been working on an event orchestration platform[1] with the initial use case being "surface your deadletter queue, and resolve problems" mostly to scratch an itch. In other words, what happens when you can't process an event and you want a human to investigate the why and perform some kind of action to resolve the issue.

[1] anketta.com

Finishing a book, trying to arrange a few workshops, and trying to put in some time to complete an API mocking service.