While it is possible that he only has it only on his hand, 95% of people who test positive for HSV 2 have it genitally (when people who test positive for HSV 2 antibody do daily home swabbing of the genital area for 16 weeks using a very sensitive test).
I don’t actually agree with your doctor but you need to do what is most comfortable for you. To be clear: About 47% of people ages 14-49 have HSV 1, 16% of HSV 2. 80% of people who are HSV 2 positive don’t know it but can transmit even without ever having symptoms that they recognize. The way that this would impact your current relationship if you are positive yourself is that you wouldn’t need to worry about transmission. But it sounds like you’ve come to an excellent place with this and that’s what is important.