On the one hoof, you will find these 'worlds' are not terribly large. At the time, Twilight Town was the only world available besides Traverse Town. A mysterious gap in a wall leading to an equally mysterious mansion in the middle of an out-of-place forest stands out a great deal. When you go to see it for yourself, I think you will agree.

On the other hoof, the worlds that we go through are all connected to people here in dreams. In this case, some of the people here knew the mansion to have had major significance in the waking world, and so knew to investigate it here.