It is true that HSV 2 can be transmitted to the face, I have seen oral HSV 2 perhaps 10 times in my entire career. And these people all had genital infection as well. So they gave oral sex to and had intercourse with the same person and got it in both locations.
We can go round and round about this but it would be best if both of you had confirmatory testing done, the best is western blot, to know what you both may or may not have.
You had a positive swab test for something herpes but we don’t know the type. he has something that may or may not be a cold sore. Don’t you think instead of wondering or guessing, it would best just to know for sure?
Terri