Kyodo News said the government had concluded it was "not possible" to allow fans from abroad to attend this summer's Games, already delayed by a year because of the pandemic.
This was due to "concerns among the Japanese public over the coronavirus and the fact that more contagious variants have been detected in many countries", Kyodo said, citing unnamed officials.
The Asahi Shimbun daily said the International Olympic Committee (IOC) had asked Japan to make exceptions for overseas guests linked to sponsors, and that the government was considering the request.
Despite a spike in infections this winter, Japan has had a comparatively small outbreak overall, with about 8,300 deaths. It has avoided imposing blanket lockdowns, but the Tokyo area is currently under a state of emergency that requests bars and restaurants close early.
Tokyo 2020 organisers have stressed that public safety will be "top priority" at the Olympics, which are due to start on July 23.
Around 900,000 tickets have reportedly been sold outside Japan.
Games organisers have said they plan to officially rule this month on the matter of foreign fans -- likely before the nationwide torch relay begins on March 25.
Earlier Tuesday, there were reports that the torch-relay starting ceremony would probably be held without spectators, but fans would still be able to line the route.
And a poll by the Yomiuri Shimbun published this week found that more than 75 percent of Japanese opposed overseas fans attending the Games.
Tokyo 2020 president Seiko Hashimoto said Friday that organisers "really want to hold the event in full stadiums with fans from around the world".
But this would be difficult "if we're not in a position where we can accept them and the situation with medical facilities isn't perfect", she said.
Organisers have already laid out strict rules for the nationwide torch relay, which was called off in 2020 after the historic decision to postpone the Games by a year.
Reports said organisers feared crowding at the March 25 starting event in Fukushima, and would probably bar the 3,000 spectators initially planned.
A spokesman for Tokyo 2020 organisers would say only that "details of the (relay) Grand Start are still under discussion".
Organisers are also set to decide on limits on overall spectator numbers in April.