It happens to us all, at one time or another. Your sibling, your best friend, maybe even your own parent or kid, brings home a new partner — and you simply cannot stand them. Maybe they keep bringing up the fact that they have money, or they’re rude to the waiter, or the arrogance is just leaking out of their pores. Whatever the case, you’re all good, thanks. But what should you do when you genuinely dislike a loved one’s partner? We asked a relationship expert when you should speak up and tell your friend how you feel, and when it’s better to keep your opinion to yourself.

If you’re concerned for your friend’s safety

For starters, carefully consider why you don’t like this person. Are you just not vibing with the new boyfriend or girlfriend, or are you truly worried about your friend’s safety and well-being? Your answer will determine your next steps, but if you suspect abuse, you should definitely talk to your loved one immediately.

“In the case of worrying about their safety, I would speak up and say something to your friend,” says Vanessa Marin, a licensed psychotherapist and expert on sex and relationships. “I would really focus on how much you care about them and how much you’re trying to protect them.”

Be prepared for pushback, Marin warns — most poeple in relationships can’t see the dynamics at play as well as outsiders because their feelings are involved. “These are always really difficult conversations to have. You have to be prepared for the possibility that your friend is going to be really upset with you or potentially even cut off the friendship.” If this is the direction things go, do everything you can to ensure your friend knows you are there to help and support them in any way possible.

If your first impression of them sucks

That said, Marin feels like most people dislike a friend’s partner for other reasons — they’re annoying, or rude to you, or just obnoxious in some way that grates. If you just met and you got off on the wrong foot, Marin’s expert advice is to do exactly the opposite of what you probably wish she’d say. You need spend more time with this person before forming an opinion.

“We’re so naturally protective of our friends. We want the best for our friends and the partners often know that and they can be nervous and a little bit on edge. I think everybody, when you’re meeting the friends, partners can come off kind of obnoxious or awkward or weird because they’re actually trying really hard to be liked by you. I’d give them the benefit of the doubt. Try to spend a little bit more time with them and see, as they get comfortable with you, if some of those behaviors lessen.”

If they suck

Look, not everyone on Earth is meant to get along. Even though it’s hard to imagine your close friend or family member picking a partner so different from everyone else they know, what they’re looking for in a romantic relationship might be nothing like what they need in a friendship. As Marin puts it, “two perfectly nice, wonderful people can just not hit it off sometimes.”

In short, if you think your friend’s boyfriend is super freakin’ irritating, keep it to yourself. “If you just don’t get along with your friend’s partner, you find them a little bit annoying or whatever, I honestly would not say anything. There’s really nothing good that can come of it,” Marin says. “If your friend is really into this person, you saying you don’t really like them is not going to have that big of an impact on that. You’re really just setting your friend up to be defensive and to potentially even cut off contact with you. It’s just not worth saying anything, and it’s not really your place to say anything.”

Instead of broaching the subject with your friend, step back and reset your own expectations, Marin says. “Recognize that it’s OK if you don’t really love your friend’s partner as long as they’re treating your friend well. like You’re not owed having a friend’s partner that you absolutely love.”

If the reason you dislike the friend has a little more substance — say they made a weird comment to you that makes you uncomfortable — you can certainly draw some boundaries. Maybe you share them with your friend if you have serious concerns, or maybe they’re just something you keep to yourself. “Make an agreement with yourself like, ‘I don’t want to make myself spend time with this person that makes me really uncomfortable, so I’m only going to hang out with my friend when we’re one-on-one or when we’re hanging out with other friends.”