If you're feeling this hesitancy, I'd say that's a good sign that you probably shouldn't do it. Some relationships can take this, others can't. Even with permission.
It doesn't mean that you've got a weaker relationship or you're less of a man. It means that something in the back of your mind is screaming out a warning.
It might be a good idea to listen.