Olympic bosses acknowledged on Wednesday there was no "ideal" solution as a growing number of athletes expressed concern.
The COVID-19 pandemic is playing havoc with the global sporting calendar, forcing the postponement of Euro 2020 and a suspension of the tennis season.
International Olympic Committee chairman Thomas Bach said earlier this week that starting on schedule on July 24 remained the organisation's goal.
But Coe, who is a member of the Tokyo Olympics Games Coordination Commission, conceded in an interview with the BBC that a delay was possible.
"That is possible, anything is possible at the moment," said Coe when asked whether the Games could be postponed to September or October.
"But I think the position that sport has certainly taken, and it was certainly the temperature of the room in the conversation I had the other day with the IOC and our other federations, is that nobody is saying we will be going to the Games come what may.
"But it isn't a decision that has to be made at this moment."
Coe, who played a pivotal role in securing the Olympics for London in 2012, said postponing the Games until 2021 could present problems.
"That seems on the surface of it an easy proposition, but member federations actually avoid Olympic years often to have their world championships," he said.
Britain's retired four-time rowing Olympic gold medallist Matthew Pinsent called for decisive action.
"On a global front we have other priorities and I think the Olympics should at the very least be saying we should postpone or indeed just cancel at this stage and we'll talk about postponement later on," he told the BBC.
"I just don't think there's much of a choice at this stage."