(I think) I cannot run any training because the test might be interrupted any second. And even checkpoints won't help because I won't always have the option to resume the training on another machine -- another machine might have different model of GPUs or different amount of them.
I tried generating thousands of images of cats in hats with sdxl but it quickly becomes boring.
If only there was a simple way to e.g. generate images and have a step that automatically judges them so that I could have a result of a several uniquely beautiful images. Or if I could generate songs lyrics and have a step which checks how well they rhyme.
How would you stress test a GPU so that it produces some interesting artifacts as a side effect?
I would suggest Aphrodite, which is faster and loads it down better, but you can also run Kobold.cpp which is much easier to set up.