Would this make your hand holding the iced coffee cup warmer or colder?
When you are holding something, you are transferring heat via conduction, which is generally much faster than radiation or convection. You also transfer heat to the nearby air, heating it up. As the air gets closer to your body temperature, you loose less heat, as most heat transference is a function of temperature difference. If something like a jacket keeps the warm air close to your body, then you feel warm. If something like wind moves the warm air away, then you will will feel cold, this is the source of wind chill. But either way, you don't transfer as much heat to air as you would a solid or liquid.
To answer your question, it will probably make you colder because you are going to transfer heat faster to a solid than the air in most cases. Even if its slightly warmer than the ambient air temperature, its much colder than you. There might be an edge case where windchill is extreme enough that shelter from the wind by touching the drink may be warmer. Also the drink itself will stay colder longer than compared to a hot day, as it will also be transferring heat to the air.
This is also related to why being wet and cold is much more dangerous than just cold. Cold water acts as a heat wick that transfers heat out of your body, and then to the air. Unlike your skin, it doesn't have a barrier to the air and will radiate heat to the air faster, also your body will waste energy heating up the water without any benefit.