How is predictive suggestion possible on encrypted services like Gmail?

Even if it's done locally on device, where would they get the training data set for it to work so well?

Same goes for reply response, follow-up reminders and other new features that if offers.