Just over 80 percent of those asked by Kyodo news agency said the huge global event should be cancelled or postponed again -- a jump from around 60 percent in a December 6 survey by the same outlet.
Kyodo said the survey asked 1,041 participants nationwide, who were selected by random dialling.
Tokyo 2020 organisers have said another delay is out of the question and are insisting the Games will go ahead despite a state of emergency declared in the greater Tokyo area this week over a surge in Covid-19 cases.
In Sunday's national telephone poll, around 35 percent of people told Kyodo they favoured outright cancellation, while some 45 percent said the event should be postponed a second time.
The month-long emergency in the capital and surrounding regions is less strict than harsh lockdowns seen elsewhere in the world and primarily targets restaurants and bars, which have been asked to close early.
Prime Minister Yoshihide Suga said this week that Japan is committed to holding a "safe and secure" Olympics.
He said he believed the public mood will change when the country begins vaccinations, currently scheduled for late February.
But senior International Olympic Committee official Dick Pound told the BBC he could not "be certain" the Games will go ahead, because "the ongoing elephant in the room would be the surges in the virus".
Public sentiment towards the Olympics in Japan has been less than optimistic for months.
Two polls in July showed the majority thought the event should be postponed again or cancelled, while a survey released in December by national broadcaster NHK found that only 27 percent of respondents supported holding the Games in 2021.