Looking at builder.py, force_xcode=True is set if any modules are detected. Is there a reason for this? If I comment out that line then simulator builds return to normal speed. A cursory look at the latest git indicates this problem still exists in the dev code.
I'm wondering myself about exactly this one thing too. It's taking so much time to rebuild the whole thing.