Generically an e2e test requires you replicate all aspects of the system. Even if everything goes right that is a lot of work to set up and a lot of time to wait. Many parts means many opportunities for something to break, and you'll be really resentful of how long the happy path takes when you have to redo it because of the one part that isn't happy.
Also specifying a good outcome in a robust way is hard. You might build into your specification a certain kind of UI element is blue and then you decide to make it red. On one level the corporate people think the branding is important and might take forever to decide on the right color, on an another level it is superficial to what the application does. Robust tests capture what is generic/important and aren't bothered by what is specific/unimportant.
Contrast that to unit tests where you can be sketchy.
I was writing a unit test the other day that involved an object had 20 fields. Only 2 of those fields were used by the system under test so I left the other 18 null. If one of those fields being null is a problem later I can give it a value. If I had to make up values for the other 18 fields it is a lot of effort and probably one of those values will be problematic in the future and have to be changed. An E2E test though would require that all the fields are specified because that object could exercise the whole system.
Specifically E2E testing of some systems is less fragile than other systems.
Unit tests are that way too. In some cases writing unit tests can be a joy, in other cases it is a constant fight and you end up with "test induced design dysfunction".
If you make peace with the process and address all the sorts of fragility you might think your automated E2E tests are less bad than what they cure; if you don't find peace you went from the frying pan to the fire.