I actually do think you should trust your OB and drop it, yes. If you test positive for HSV 1, in your current state of mind, you are likely to believe that you have this genitally and start really worrying about that. I could be wrong about this. In my mind, I think it is unlikely that both of you would be asymptomatic at the same time. When an adult acquires HSV 1 genitally, they often have symptoms, especially if they are looking. Some are quite remarkable. Honestly, I think you would notice something if you had gotten infected given your state of mind. Most people who have herpes really do have symptoms, they just don’t attribute them to genital herpes.
So you have several things suggesting that you are not infected:
1) you have had NO symptoms
2) he had NO symptoms
3) you have no idea if he has herpes or not but you could ask him about it. If it is genital, no worries
4) it was a one day thing, not repeated exposure which would put you more at risk.
If you test and it is positive, you cannot know where you infection is, oral or genital. I think it would be best to skip it and I don’t often have that thought.