Athletes from all over the world are due to head to the Japanese capital for a showpiece that is scheduled to be opened a year later than originally planned on July 23.
The coronavirus pandemic prevented the Olympics from being staged last year and there have been growing concerns that it will not be possible for the Games to go ahead in 2021.
An unnamed senior Japanese government source told The Times: "No one wants to be the first to say so but the consensus is that it's too difficult. Personally, I don't think it's going to happen."
Yet Thomas Bach, the IOC president, this week told Kyodo News there is "no reason whatsoever" to believe the Games will be called off for a second time due to the COVID-19 crisis and said there is "no plan B" for the Games.
The Japanese government also firmly stated on Friday there is no truth in claims that it has concluded the Olympics will have to be cancelled.
A statement from the IOC said: "We refer you to the strong and clear statement that the Japanese Government made today, saying that the report is "categorically untrue".
The statement from the Japanese government said: "Some news reports circulating today are claiming that the Government of Japan has privately concluded that the Tokyo Olympics will have to be cancelled because of the coronavirus. This is categorically untrue.
"At an IOC Executive Board meeting in July last year, it was agreed that the Opening Ceremony of the Olympic Games Tokyo 2020 would be held on July 23 this year, and the programme and venues for the Games were rescheduled accordingly.
"All parties involved are working together to prepare for a successful Games this summer.
"We will be implementing all possible countermeasures against COVID-19 and will continue to work closely with the IOC, the Tokyo 2020 Organising Committee and the Tokyo Metropolitan Government in our preparations for holding a safe and secure Games this summer."