I'll give a related example using HSTS (headers that tell the browser to always use HTTPS). If I can MITM a connection that a person has never visited and assuming the domain name is not preloaded in the browser with HSTS, then I can simply strip out the HSTS header. This only affects people visiting the site for the first time, or the first time with that browser.
Or another example closer to your use case may be GPG keys for Linux YUM software repositories. If a person installs their OS directly from a repo mirror and I have compromised that mirror, I can modify any of the packages, re-sign them and put my GPG key in place of the OS maintainers GPG key.
Does your implementation fall into those examples, or would it be impossible to interact with your site if they crypto libraries were stripped out?
Beyond that, HTTPS is still useful for privacy if there would be any reason a person would not want someone to know what they are doing on your site. HTTPS is also useful for not getting an alert or red icon in the URL bar depending on the browser being used of course.