So do you get cold sores or canker sores inside the mouth? Is he the giver of oral sex to you with the cold sores?
I am concerned about the presentation that you describe; it could be genital herpes and you think so too since you posted here. Have you considered having a type specific IgG antibody test done to see what that shows? I think that would be good and also certainly have this area swab tested if you get another one. Has you boyfriend had a herpes antibody test?