Writing a layout engine for multiple operating systems

I want to write a minimal browser layout engine. I know it's an extremely daunting task, I don't intend to converge with HTML5 compatibility. I just find browsers fascinating and would like to learn more about them.

I want to have an actual Window appear that can emit text, color, and shapes. I don't want to use a graphic UI framework like WPF. I also believe a graphics library like DirectX or OpenGL would be overkill as it's adding features I don't need, but I really don't know. Chrome and Firefox use ANGLE, maybe that's a start?

TLDR If I wanted to write a layout engine for multiple OS's, should this be built on top of a graphics library, a windowing system, both, or something entirely different?