Sometimes the page will be detected immediately. Other times, no amount of finagling will make it appear. There doesn’t seem to be any simple command line way to force the discovery and authentication process either.
I normally get around this by just using my phone as a hotspot so it’s not a huge issue, but other times when I travel that’s not a good solution (especially internationally).
Is there somebody out there who has figured this out? Starbucks never works for me. Chick-fil-A is always reliable. Most hotels give me issues. My local university, totally fine.
There has to be some solution, doesn’t there?
This is because captive portals typically intercept any HTTP requests made by users and redirect them to a login page. However, HTTPS requests are not intercepted in the same way, as the SSL handshake prevents the captive portal from intercepting the traffic and presenting the login page.
By visiting a non-SSL (HTTP) page initially, the captive portal can intercept the request and redirect to the login page, allowing you to authenticate and gain access to the network.
Often captive portals will restrict certain kinds of traffic (e.g. by blocking ports) just "because". However, bear in mind there are financial incentives and ramifications to "free internet". So, every provider tries to lever things in one way or another; this includes cell-phone companies and ISPs themselves. DNS hi-jacking is a good example.
All that said, the combination of ways networks can be configured (both yours and that of the captive portal) are enormous. And, most likely this is what's causing the pain... I empathize.
In this specific case, if you're not already doing so, I'd consider trying a tool which could help at certain locations/providers. I've heard good things about Gnome's network-manager-applet. And, like you say, keep your phone's hotspot nearby just in case.
I travel with my Apple TV so I don't have to enter my Netflix or whatever credentials into someone else's TV. Also, Apple TV has a much nicer UI than most smart TVs. However, no Apple TV model has a browser, so there's no way to directly log into a captive portal such as those used in a hotel.
My workaround is to set the MAC on my travel router to be the same as the Apple TV, then log in to the hotel's captive portal through the travel router. Once that's done, the hotel will remember that MAC address (the Apple TV's wifi address) and the Apple TV will not need any further authentication.
The travel router I use is a TP-Link TL-WR902AC. It has a horrible UI, but if you persist, it can be made to do a lot of unusual things you might need for travel or conferences.
And to address the problem you're having: At least once or twice when I couldn't connect to a captive portal using my Debian laptop, I was able to connect through the travel router (though I never figured out what the specific problem was).
My problems have been more with Microsoft owned websites and Firefox on Debian. Poor Microsoft, they have so little money they can't afford developers to make sure image uploads work on LinkedIn or Github w/ Firefox. At least on Github you can upload images by checking in and doing a push. Until they break Github and make it a windows-only thing.
I routinely see them fail on mobile devices (phones and tablets) and laptops
My personal failure rate with captive portals is at least 50% - and that's on macOS, iOS, Android, and Windows