First, rich countries often outsource to other rich countries. This suggests that the decision to outsource is not only a cost-related one; other variables are also driving this decision. Rich countries produce a lot of sophisticated goods that often require sophisticated inputs. Thus, firms must outsource to places that have the knowhow to provide such inputs.
Second, connections to and from poor countries are weak.