How do I determine the point at which getting -res is worth losing skill points/+res damage? I assume there is no easy way. I am ditching the tal set for flickering flame (-29 res) and other random junk that gives my meteor like a 6k boost, maybe 8k, but the tal set overall gives me -40 res. Only a mere -10 difference, but how do I compare? It is hard to know since there is nothing on the character screen with -res to compare to...

keep the flickering flame unless the other set mods are that much better (defensively).

in general, i would think you would like at least -60 end game for powerful dps characters. some situations call for more -res, others call for more damage. since monster res fluctuates quickly (between different areas, monsters, acts), sometimes -res works better and sometimes more damage works better (a 99 res monster needs a lot more -res; on the other hand, a 0 res monster needs a lot more damage). you want to keep a general rule of thumb of "balance"...if you see your gear severely lacking one or the other, balance it out.

as a sorc, since you have a mastery for damage already, keep to +skills and -res. %skill damage will have the smallest effect since your mastery should already be multiple 100s %. +skills not only increase the skill damage itself, but also mastery, which gives you more %skill damage. my cold sorc was waiting for level 95 to wear a fathom with 3 uber sapphires for a total of -61 res. Templars was another source of -res, and then lastly i got a couple of facets for the templars. Besides that, gear wise i was focusing on skills and surviving. Conviction, lower resist, or holy freeze from either a party member or a merc sufficed for the rest of the pierce.

I have like -79 res, and I definitely need more resistance for myself. Cold is -10, so I'll work on that. Thanks.

It should be pretty easy to compare I think. There's only 3 things to consider.

Your damage formula is just:
Damage = BaseSkillDmg * SynergyBonus * Mastery * (100% - EnemyRes)

BaseSkillDmg only benefits from +Skills
SynergyBonus is static and predetermined for all skills but Cleansing and Meditation, in general this isn't considered since it doesn't change.
Mastery = SorcMastery + (+%Mastery from all gear, such as facets)
EnemyRes = normal resistance of monster lowered by -%Res gear

When you consider a new piece of equipment, just consider it's change from your previous equipment.

Example:
BaseSkillDmg = 1000
Synergybonus = 500%
Mastery = 200%
EnemyRes = 0% (also 0% -%Res)
Damage = 1000 * (100% + 500%) * (100% + 200%) * (100%-0%) = 18000 dmg

New gear will drop BaseSkillDmg to 800, and lowers Mastery by 30%, but increases -Res% by 20%

BaseSkillDmg = 800
SynergyBonus = 500%
Mastery = 170%
EnemyRes = -20%
Damage = 800 * (100% + 500%) * (100% + 170%) * (100% + 20%) = 15500

In this example, where the enemy didn't resist your damage to begin with, then dropping your base skill damage by 20%, while lowering enemy res by 20%, there was a decrease in your damage mostly due to loss of MasteryBonus. This is a good example for Norm difficulty, where monster res are low (+Skills and +%Matery gear are great).

In NM and Hell, just assume that monster res starts off higher, maybe 30% and 60% respectively. In those cases, the example changes to:

NM before = 12600
NM after = 11700 (-20% res still doesn't make up for a drop in 20% base dmg)

Hell before = 7200
Hell after = 7800 (-20% Res finally becomes worthwhile despite 20% drop in base dmg and lower mastery)

These examples illustrate why Convictors are alright even in Hell diffiulty. Their damage doesn't change much, but everyone else's got lowered drastically (50% for phys-dmg chars (lowered only 15% to 30% if they've got a high level Amp)). HU was balanced around the "Everyone Else" category.

Oh thanks, that's good to know.

Does this mean lowering an enemy's resistance to below zero is worth it? Or should I only have the -60 res needed to counter an enemy's resistance in hell and then focus the rest of my gear towards skills?

im pretty sure (not 100% though, confirm with a second source) that monster res is capped at -100 res (like players are). Will definitely be worth it to get over 100 pierce if you can make that without a large sacrifice in skills or survivability. -100 res will double your damage on a monster with 0 res, be 4 times as effective as a monster with 50 res.

Oh thank you, very good to know.