If someone with a herpes outbreak used a towel and someone immediately came along and used that towel to wipe their mouth of genital, transmission is possible but very very unlikely.
And no, you would not get infected in that way with your brother and mom.
And no, viruses don’t walk on skin.
Terri