Mertens, ranked 23rd in the world, entered the tournament in Istanbul as a favorite.
But the first-seeded Belgian had a difficult journey against Peterson.
The Swede, ranked 76th in the world, stood for a strong turn in the first set.
Mertens had gone up to 5-3 but Peterson reduced and then broke the Belgian's serve and was finally able to take home the first set with 7-5.
In the second set, Peterson connected the grip early and won 4-1 before Mertens threw in the towel due to injury.
In the round of 16, Peterson will face the winner of this afternoon's match between Poland's Magda Linette and Yulia Putintseva, Kazakhstan.