After mainframe OSes, I'd have to say VAX/VMS is the worst I've used. This is partly because of the aggressive "need to know" culture that typically kept the official documentation locked away, but also because the OS just didn't make sense. It was very idiosyncratic, from file naming, to command line flag style, to its process model.