[snip]
40K doesn't do that. In 40K, the side with the most killy stuff wins. It doesn't matter that your Chapter Master is surrounded by 108 Gaunts, he still wins because he killed more of them. If 40K was done right, the Tyranid player would just sit there and announce "you need to kill more Gaunts than you have attacks in order to win this fight!" and allow combat resolution to kill the guy for him.
That's a 5th edition thing. In 4th the combat modifier was based on how many of each side were involved in the combat, not how many had previously died. Some of the tyranid upgrades (like for the carnifex) said that it counted as 10 models rather than 1 for purposes of outnumbering in melee. But that system had weaknesses too. 8 terminators who just killed 14 of 31 termagants without losing any of their own could still be scared by the fact that they were "outnumbered 2 to 1", which didn't make much sense.
Armour Modifiers would go some way to rebalancing this clusterfuck of a system. Ditching Invuns would be even better, since that means we'd see an end to half the klkn Space Marine players insist on pulling.
I don't think invulnerables are the problem. They are reasonably rare and tend to be weaker than the regular saves. Dark Eldar Wyches have a 4+ invuln but only in melee. A couple of their characters have a 2+, but only until he fails it once. Daemons all have invulnerables, but most of them are 5+; Tzeentch units have 4+ and the Lord of Change has a 3+, but they are either expensive or vulnerable to melee. The only thing I can think of that has a constant, good invulnerable save is the assault terminator. Yes, a 2+/3+ is a pain to deal with, but the models are expensive and just as vulnerable to a torrent of fire as a terminator with a 2+/5+. First-Rank-Fire-Second-Rank-Fire, markerlight-assisted Fish Of Fury, Bladestorm, or several other things will usually get rid of them. They're only twice as hard to kill with massed fire as a regular space marine, but cost significantly more than twice as much, don't they?
Now a strong enough attack modifying armor save might make sense, but only if it starts fairly high. A strength 5 or 6 hit should not reduce an armor save for a toughness 4 character since you're getting more wounds. Now if you get to a strength of +3, maybe that should drop the save by one, but the situations where it would matter are small enough that I don't think it's worth the extra arithmetic. I think a better system would be if the AP stat was a modifier, although that would require turning everything upside-down. Maybe if AP is turned into an armor save modifier of "5 minus AP." So an AP3 weapon would worsen the armor save by 2, giving space marines a 5+, terminators a 4+, and anything with a 5+ or worse would get nothing at all. An AP1 weapon would worsen the save by 4, giving terminators a 6+, but nobody else would get anything.
The math would be kind of a pain I admit, but it still wouldn't be as bad as the AD&D 2nd edition's concept of THAC0 and negative armor classes that a few of us might be able to remember, and everybody just dealt with that!