Is it worth switching to an uncommon keyboard layout?

It depends. If you don't have any problems with RSI or other similar conditions I would recommend against switching layouts, as there has never been conclusive studies to demonstrate that any other layout is faster for typing than QWERTY and transitioning takes a significant amount of time. If you do have issues with QWERTY I would first recommend an ergonomic keyboard (particularly a split board with as many adjustments as necessary), and if that fails then another layout may be necessary. Anecdotally a lot of people have benefited from switching layouts, but usually it's more hassle than it's worth.

I experimented with the Dvorak layout for a year or so. It felt much more natural, like I'd been doing typing wrong my entire life.

The problem was this-- it made it difficult to use a normal keyboard layout. Such that anytime I had to use a colleagues computer for a moment, I was muddled. I couldn't keep two layouts in my head at the same time and switch between them easily.

No, I feel like the time it takes to learn a new layout can be used to improve skills on your current layout. Are we really chasing WPM these days?

No.