Even in the first century, about half the Jews lived outside Judea and it went down from there. And it’s not like they were displaced by moslems. They just left.