My questions to HN -- (1) Is one more strong PhD letter of recommendation (I already have one) and experience working with researchers that much more valuable to graduate schools to justify the downsides of moving? (2) What else do people take into account when they jump ship to a new team?
If you only have one letter, then I'd say yeah, another one helps. However, not all letters are equal. If you can, look into the backgrounds of the people you'd be working with. If they got their PhDs with good profs at schools you want to apply to, good letters from them will be worth much more than a letter from a random research scientist without a good network.
As far as research experience, the main thing that will give you a leg up for grad school is papers, preferably in good conferences or journals. Will you be able to get authorship? If so, could be worth it.
(2) What else do people take into account when they jump ship to a new team?
Does the work sound interesting? Will I be motivated to work on it without a lot of external pressure? Are the team members detectable assholes?