It's a little bit too pirates of the caribean for me, but overall I'd you're really wanting to keep it go for it, just trim a little :-) you can totally pull off facial hair, not all men can
Well, you could definitely get a part as one of the Musketeers with the long hair and facial hair. Does the fact that women make fun of you tell you anything about it? You're a good looking guy, so why go odd on the beard/stache?