If you have a build system capable of deterministically reproducible builds (at the level of individual bits), then the vendor can publish the commit ids for every binary, and users can build their own version from that commit and verify the signatures matched.
In practice, very few build systems offer this level of reproducibility. I've only ever seen this supported by the in-house build system at Amazon.