You know, I think you are in an unfortunate position, aren’t you? He won’t test, he won’t talk to you, except to drop this bomb on you.
If I were in your situation, I would say the hell with him. You used a condom, you’ve had HSV 2 for a very long time, you didn’t have an outbreak at the time, and the lesions, as you mentioned, were in a place covered by a condom and his wife had HSV 1. Try to not buy into this. I find his behavior reprehensible.