Ligue 1 games in Nice and Metz were paused in the space of a week last month after discriminatory banners were seen in the crowd.
A second-tier encounter between Nancy and Le Mans was first to be stopped earlier in August as match officials acted on new instructions to intervene in the event of homophobic chanting or displays.
French sports minister Roxana Maracineanu backed the Ligue de Football Professionnel's attempts to stamp out discrimination but Le Graet, a FIFA Council member, thinks interrupting play is misguided.
"I would not stop the matches," Le Graet said.
"It's a mistake, totally, because I don't want to be held hostage to homophobia. I would stop a match for racist abuse, that's clear, or for security reasons.
"We're going to make sure the banners disappear, there is security to make sure that happens, but stopping matches is something else.
"It's not football but society in general that has to think about the problem, at school and in companies, everywhere."
Addressing the fans responsible, Le Graet said: "Do not come to the stadium with banners, it's as simple as that.
"You can support your team, but [homophobia is] absolutely wrong morally."