Targeting a first medal in the event, the host nation almost suffered last-gasp heartbreak after recovering from 70-61 behind in the final quarter.
Saki Hayashi’s three-pointer put them in front by one with 16 seconds remaining, but there was still time for Belgium’s Kim Mestdagh to take aim right at the death.
However, her last-second jump shot bounced off the rim, meaning Japan go through to a last-four clash with France.
"There were so many peaks and valleys. We were hanging on to the cliff by a fingernail in the fourth quarter," coach Tom Hovasse said.
"We just came up with plays and towards the middle of the fourth quarter, we ramped up our defence and that took them out of their comfort zone.
"We believe in ourselves, and I am hoping more people outside our locker room believe in us.
"I think it is safe to say it is the biggest win in Japan basketball history."
FRANCE FEND OFF SPAIN FIGHTBACK
Japan's next opponents are France, who beat Spain 67-64 after another epic encounter.
Despite dominating most of the contest – Marine Johannes leading the way with 18 points – France appeared in danger of throwing it all away as they fell 61-60 behind.
However, they recovered to snatch victory and secure a third consecutive appearance in the last four.
Astou Ndour had 16 points in a losing cause for Spain, runners up from the Rio Games who will not be taking home a medal this time around.
SERBIA STUN CHINA
Serbia were another team to produce an inspired turnaround as they defeated China 77-70.
Bronze medallists on their debut in Rio, the European champions recovered from 58-50 down to reach their second successive semi-final at the Games.
"How many times have we done this, 20, 30 times?" said shooting guard Ana Dabovic, who claimed six assists during the game to go alongside her 13 points.
"We never quit, and we play the hardest when we are down. We showed today we can find energy.
"This is a great success for a small country. Second time at the Olympics for Serbia; second time in the semi-finals. This is great."
Jelena Brooks top-scored with 18 points for Serbia, while Sonja Vasic had 16.
SEVEN IN A ROW STILL ON
The United States remain on course for a seventh straight Olympic gold after easing to a 79-55 win over three-time silver medallists Australia.
Breanna Stewart led the way with 20 first-half points – she would finish the contest with 23 overall - as USA ran out 79-55 winners.
"I thought we came out and played inspired basketball on both sides of the ball," said coach Dawn Staley.
"We played with an incredible desire to advance and it was just contagious.
"I thought our team was focused on keeping the heat on Australia and not let them back in the game."
Team USA have not failed to win the women’s tournament at an Olympics since Barcelona in 1992.