You get this impression not only in Harry Potter, but in Game of Thrones and other fandoms. We see, or at least hear of, armies and other groups made up of thousands or millions of people, and we know we're dealing with a world fairly vast and large, yet all the widely-strewn characters keep bumping into each other within it. And when you do need an army of millions, they aren't there and you end up with seven or eight familiar faces doing the heroic defending. (GoT does have people hiring entire armies, I know. But at the same time, they also frequently have people traveling hundreds of miles and randomly encountering someone they know. And you occasionally get the weird impression that some entire kingdoms have, like, fifty or sixty people living there.)
TV Tropes does have the "It's a Small World After All" trope and the "Contrived Coincidence" trope, which both overlap what I'm describing, but are not quite the same thing. Thoughts? Anyone else have the Not Enough People For This World impression in other material?