Quote:
I like a nonchalant "Sure" but if you don't establish a boundary then people will walk all over you its in their nature to do so. Perhaps if I combined this with a dismissive pat on the chest followed by turning my shoulder and attention elsewhere
I'm not so sure that people will walk over you. You seem physically impressive.
But YES, physical contact with guys is a great way to remain dominant over any conversation. Next time, don't give him dismissive pats, just put your arm over his shoulders and lean on him a little. This emphasizes the size difference, makes you appear physically over him, and will intimidate him.
If he meant you no harm by his comment (many times, "AMOG" comments are accidental and if you overreact to something harmless, you come off very badly) then it will become clear that he meant you no harm. If you put your arm around me, for example, I'd clap your back, and emphasize to the group that you're cool, and how we're tight now. The arm then becomes AMOGs acknowledging each other and becoming allies. You should always prefer to make an ally rather than an enemy.
If he did mean harm, and wants to control the situation, then his reaction would be different. He would realize, "Man, I hate this guy, get him off me" and react that way. Then he would be the one overreacting and lose status in the group, and also may get physical back at you, which would be a one-sided fight.
When a guy says something that seems to lower you, and you don't have a good response, ignore it and be physically friendly. That's a good general rule, especially for someone of your size.