So have you ever in your life had a cold sore on your lip? How about your partner? Is he the giver of oral sex to you?
I think your reaction is very minor and healed very quickly for this to be herpes. It would be good if he could get an IgG test to help resolve and clarify some of your concerns. Would he do that or at least check on his test results? Have you been previously tested for HSV?