Do 'selfish genes' explain Galapagos birds etc.?

I understood that extravagant mating displays, which are detrimental to the male in particular (requiring calories, making vulnerable to predators etc) are selected for because they signal to the female that the male is strong enough to survive that handicap.

But isn't it enough that the gene for expressing such elaborate displays is 'selfish'? Consider that a gene that 'wants to be copied' (all genes), if it could signal it's presence and simultaneously cause that signal to trigger mating, would succeed in continuing in the species? In that case mating rituals are not there for any fitness proof, but simply a sort of parasite. A gene that's using an animal for it's own purposes.

How could we even distinguish this sort of parasitic gene, from any other kind of fitness selection? Is there a meaningful difference?