Draconic Sorcerer - adding Charisma modifier?

There is a difference between hitting and damage

The sorcerer feature Elemental Affinity says (p. 102 PHB):

Starting at 6th level, when you cast a spell that deals damage of the type associated with your draconic ancestry, add your Charisma modifier to that damage.

You normally do not add your Charisma modifier to damage, you add it to the roll you make to see if you hit (or to the DC someone has to beat to save against your spells). Only once you hit, you then resolve the damage your hit causes.

For example, if you cast a fire bolt spell on an orc, you first make a Spell Attack roll by adding your Spell Attack modifier (your Charisma modifier + your Proficiency Bonus) to your attack roll (usually a d20, unless there are special circumstances like Advantage). If that number is equal or better than the orc's AC, you hit and deal 1d10 plus —due to this feature— your Charisma modifier fire damage to the orc.

Or, if you cast a fireball that includes the orc in its area of effect, you deal 8d6 fire damage to the orc, plus your Charisma modifier, again due to the feature. The orc must make a saving throw against your spellcasting DC (8 + your Charisma modifier + your Proficiency Bonus), if it fails it takes full damage (otherwise, in the case of fireball, half damage).

You don't normally add spellcasting ability modifier to damage roll

When you attack using weapon, you add your ability modifier to your attack roll and damage roll. However, a spellcaster don't do that: they only add ability modifier on their attack roll, but not on damage roll.

Some spells require the caster to make an attack roll to determine whether the spell effect hits the intended target. Your attack bonus with a spell attack equals your spellcasting ability modifier + your proficiency bonus.

This is an important difference between martial class vs spellcaster: spellcaster don't get to add ability modifier to their damage roll. This Elemental Affinity feature allows you to do that.

So this feature will work for you, for your fire damage spells.