Is building a realtime web application hard (for a startup)?

I've been struggling during the past months building a realtime mobile application that scales.

We started with Firebase which is great to kickstart a project, but after a couple of months we needed to create our own socket-based realtime infrastructure.

It was really tough, especially for our small team.

I wonder if it's a problem that you've ever faced?