Animals living in social groups will almost inevitably experience competition for limited resources. One consequence of competition can be agonism, an activity that is not only costly to participate in at the individual level but potentially also at the group level due the detrimental effects that agonism can have on group stability and cohesion. Agonism rates across primate species have previously been associated with group size and terrestriality; therefore primates, particularly those in large groups, should develop strategies to mitigate or counter-act agonism. Here, we use phylogenetically controlled analyses to evaluate whether the known relationship between brain size and group size may partially reflect an association between agonism and brain size in large groups. We find strong positive associations between group level agonism and 2 measures of brain size (endocranial volume and neocortex ratio) in 45 separate populations across 23 different primate species. In contrast, dyadic (pair-wise) rates of agonism are inversely associated with group size and not with brain size. Moreover, we find a distinct absence of relationships between agonism and the prevalence of prosocial, cooperative behaviors. That overall rates of agonism increase but dyadic rates decrease with group size suggests that individuals in larger groups either can buffer aggression better or only species with low levels of dyadic conflict can maintain large groups.