I think there is certain use cases/envs where Spark makes a lot of sense, but I don't think is viable for most cases/teams, specially if you don't plan to use Databricks. The vanilla developer experience is pretty rough: automation is lacking, UI is pretty bad, local dev environments (beyond "hello world" level) are hard to setup, etc; and that's not even accounting for the infrastructure deployment/management side of things.
Mix all this with the fact that (at least in my experience) DE and DS are not know for writing robust/defensive code, and you get systems that break very often.