Decisions about political, economic, legal, and health issues are often made by simple majority voting in groups that rarely exceed 30–40 members and are typically much smaller. Given that wisdom is usually attributed to large crowds, shouldn't committees be larger? In many real-life situations, expert groups encounter a number of different tasks. Most are easy, with average individual accuracy being above chance, but some are surprisingly difficult, with most group members being wrong. Examples are elections with surprising outcomes, sudden turns in financial trends, or tricky knowledge questions. Most of the time, groups cannot predict in advance whether the next task will be easy or difficult. We show that under these circumstances moderately sized groups, whose members are selected randomly from a larger crowd, can achieve higher average accuracy across all tasks than either larger groups or individuals. This happens because an increase in group size can lead to a decrease in group accuracy for difficult tasks that is larger than the corresponding increase in accuracy for easy tasks. We derive this nonmonotonic relationship between group size and accuracy from the Condorcet jury theorem and use simulations and further analyses to show that it holds under a variety of assumptions. We further show that situations favoring moderately sized groups occur in a variety of real-life situations including political, medical, and financial decisions and general knowledge tests. These results have implications for the design of decision-making bodies at all levels of policy.