My guess is that you have only oral HSV 1. The swabbing not extensive enough to pick up genital HSV 1 infection, however, since it is shed infrequently and would require swabbing for several months perhaps to get a positive.
I think you should believe that you have HSV 1 orally and live that way until and if you get a lesion genitally that could actually be swab tested. It is very possible that you have had HSV 1 since childhood and you know now for certain that you have it orally. Having it in both places happens but is not all that common.
Terri