We are seeing jobs that hang, lingering seemingly forever. I canceled one today that was 7 days old. They have un-openable logs for some buld step that should take <10 minutes but which it says has been running for days. Apparently, the official advice is to press the stop button and the a re-run button. But often the re-run jobs button never becomes available. This has been preventing an important PR of mine from ever completing the automatic checks, so it can't be merged.
Some bits of the system assume a maximum of 100 jobs. I think we have about 140.
Sometimes the status gets confused, especially if you can find and use the re-run button. For instance, here's one where it says "100 completed jobs", I know there are about 140. All of them have green circles, but there are also errors. Here's one. Some job failed with "unable to access 'https://github.com/[redacted]/': Failed to connect to github.com port 443: Operation timed out". Though githubstatus says things are fine right now. And also inexplicably, this particular job I checked twice has a green check next to it at the left.
Sometimes asset transfer from S3 takes a long time, but of course I don't know that's github/azure/microsoft's fault.
In the past, we saw lots of builds fail due to HTTP errors retrieving packages from azure.archive.ubuntu.com. This may be fixed, so that's something.