The IgM is a completely unreliable test. The Mayo Clinic discontinued the test in 2019 and both Quest and LabCorp will be discontinuing its use at the end of 2022. It has so many false positives.
I think it is unlikely that you have HSV 2, given what you’ve said in the post. It isn’t impossible though. I would suggest that you obtain an IgG test for HSV 2 if you are concerned about that possibility.