The Logic :
You're like a brother to me. I would never sleep with my brother; Therefore, I would never sleep with you, as I've just likened you to the status of a sibling. No, seriously.

If you were to write a simplistic perl-esque pseudocode for this, it would be something like :

If (!(brother()) {
   fork(); # always use protection
}

Else {
   die;
   die; # just in case the last one wasn't clear
   die; # if the last statement wasn't obvious...
}

This statement is especially damaging when uttered by someone who does not in fact have any other brothers, because then they are simply presupposing the idea that if they had a brother, you would be like that person, as they know they would never want to sleep with that person -- and they also know that they never want to sleep with you.

Don't pretend that this isn't the context in which this is being used, because it is.

Similar Cases:
The same can be true for "You are like a sister to me" if uttered by a boy to a girl.

Exceptions:
If uttered from girl to girl, this is usually a complement, unless both of said girls are lesbians or bisexuals -- in which case the meaning may get confusing.

If you live in Alabama or New Hampshire, this phrase may actually be an invitiation towards intimacy -- proceed with caution.

This phrase has been uttered to me on two occasions.