What is the "tipping point" at which a population is safe from becoming too inter-related?

I may not be phrasing my question correctly, but here is what I am getting at. Suppose you have ethnic group / relgious group "X", of which there are 100,000 members worldwide. For the sake of argument, suppose that members of X have children only with other members of X. Would there a be a point in time when members of X would become so inter-related that they would have to start having children with non-X people in order to prevent too many bad recessive genes from matching up? Would the answer change if there were only 1,000 or only 10,000 members of X?|||yes to both, just with the 100,000 population it would take much longer to reach that point|||no to the 1st and yes to the 2nd