Bach's trip had been scheduled for May 17-18 but organisers "decided to postpone it based on various situations including the extension of a virus state of emergency" by the Japanese government.
The Japanese government and Olympic officials insist the virus-postponed Games can go ahead safely this summer, although polls show most Japanese people support cancellation or another delay.
Japanese media reports said the organisers plan to reschedule Bach's visit for June.
Japan's state of emergency, which is less strict than blanket lockdowns in other countries, has been extended to the end of May in the hardest hit regions, including Tokyo.
Seiko Hashimoto, president of the Tokyo 2020 Games, said Friday it would be "very difficult" to arrange the visit given the state of emergency.
Japan's Covid-19 outbreak remains much smaller than in many countries, with around 10,800 deaths.
But its vaccine rollout is moving slowly and some areas have seen record cases as more infectious variants drive fresh waves of contagion.
Prime Minister Yoshihide Suga was forced on the defensive on Monday, insisting in parliament he has "never put the Olympics first" and that his priority remains "the lives and health of the Japanese people".