Cryptsetup Luks Security

Luks dmcrypt is unlocked at boot from initrd image and kernel that are not encrypted which means that they are vulnerable to manipulation for anyone that has physical access to the server, is this correct? How to mitigate?