Christopher Allen (2004) has written a long, fascinating post on the "Dunbar Number" - the optimal (or possibly maximum) size of human groups that can be deduced from primate research. The bottom line (not just from the Dunbar research but from the many anecdotes included in Allen's post) - groups work well if they consist of either 5 to 9 or 45 to 55 people. Also read the insightful comments at the end of the post.
Clay Shirky's take on large group dynamics
In "A group is its own worst enemy", Clay Shirky reviews many examples where the introduction of social software at first resulted in much creative collaboration, but then became overwhelmed by the volume of discordant talk because "dense, interconnected pattern that drives group conversation and collaboration isn't supportable at any large scale". Shirky's solution to this problem is to have some kind of explicit government with a set of enforceable rules. Particularly what is important is that different types of user or member will arise - i.e., that there will be different roles (also see the section in this book about collaboration roles). Where the group develops more or less organically, one must expect that an "inner group" (those who are most committed to and work hardest for) the group will emerge. If there is no direct means for this group to communicate among one another and to enforce some kind of government, they will find indirect ways of doing so. This core group could, of couse, become something sinister, but often is more like a "volunteer fire department".
Another way Shirky suggests for dealing with issues of scale is to engage in "soft forking". Hard forking (which doesn't always work) is where a large group splits into two or more subgroups (e.g. "group A is for discussing technical issues and group B for discussing strategic issues"). Soft forking is where group members from the start belong to smaller, somewhat overlapping groups, so that one is part of a smaller group, but simultaneously also only a few links away from anybody else in the larger system.
"You have to find some way to protect your own users from scale. This doesn't mean the scale of the whole system can't grow. But you can't try to make the system large by taking individual conversations and blowing them up like a balloon; human interaction, many to many interaction, doesn't blow up like a balloon. It either dissipates, or turns into broadcast, or collapses. So plan for dealing with scale in advance, because it's going to happen anyway."