Elemental Affinity improves one damage roll of a spell, not multiple rolls? So with scorching ray, I don’t add my Charisma modifier to each ray that hits?

That’s correct. Elemental Affinity benefits one damage roll per casting of a spell, even if the spell allows more than one roll. So, for example, the feature improves one of the rays of a scorching ray spell or one of the beams of an eldritch blast spell.