Two goals from Ellen White had turned the game on its head and looked to be enough to send Hege Riise's side through, but Matildas talisman Sam Kerr levelled in the 89th minute with a crisp low strike.
The contest then swung Australia's way in a dramatic minute in extra-time, Mary Fowler's deflected shot finding the top corner after Caroline Weir had a penalty saved at the other end.
Kerr scored another before White completed her hat-trick with a deft header to set up a frantic finale, with Australia holding on for a famous win.
Sweden, the next opponent for the Matildas, ended Japan's quest for a medal on home soil with a 3-1 victory.
Mina Tanaka cancelled out Magdalena Eriksson's sixth-minute opener, but Sweden took control in the second half through further goals from Stina Blackstenius and Kosovare Asllani.
United States and Canada will do battle in the other semi-final after edging into the final four.
USA, which has won the Olympic tournament four times previously, managed to beat Netherlands 4-2 on penalties in their quarter-final after the game finished 2-2 at the end of extra-time.
Having finished with fewer than six points in the group stage of a major event for the first time in its history, the USA knew a much-improved display was needed against a Dutch side that scored 21 times in its opening three games, the most of any women's side at a single Olympics.
It duly delivered a strong display as it twice took the lead only for Vivianne Miedema to continue her sensational form, scoring twice to cancel out goals from Samantha Mewis and Lynn Williams in an engrossing contest in Yokohama.
Megan Rapinoe swept the decisive penalty kick high into the net after Alyssa Naeher had made two stops to her right.
Canada awaits USA after it inflicted further heartbreak on Brazil, whose wait for gold in this event goes on.
A beaten semi-finalist in Rio five years ago, Brazil was held to a 0-0 draw before suffering another defeat on penalties to the 2016 bronze medallist.
Goalkeeper Stephanie Labbe, who was injured in the closing stages of extra-time, recovered to make two critical saves in the shootout to secure a 4-3 win.
"In the moment, it was about trusting myself, trusting my instincts, and trusting my ability to make a save," she said after the game.
"Pain is temporary. We have our eyes on the prize."