Nehemiah 13:21

‘Then I testified against them, and said to them, “Why do you lodge about the wall? If you do so again, I will lay hands on (arrest) you.” From that time forth they came no more on the sabbath.'

And the reason that they only did it once or twice was because Nehemiah warned them that if they appeared again and encamped outside the city on the Sabbath they would be arrested. His concern may have been that they were still profaning the Sabbath, even though not in Jerusalem, or it may have been because he considered that their proximity to the holy city on the Sabbath day marred the holiness of the city on that day, in the same way as Tobiah's continued presence had marred the holiness of the Temple.