come down to (some place)
To travel south of the state or area that one is currently in. If you ever want a break from those brutal Chicago winters, come down to Arizona and spend some time with us in the desert.
come down to some place
to come to some place in the south or in a lower altitude for a visit. (Assumes a perspective of a visitor from a northern state of the U.S.) Come down to our place in Florida this winter if you want.