Defending champion Nadal, who came into the tournament having won both Monte Carlo and Barcelona for an 11th time, also saw his record 50-set winning streak on clay ended by the fifth-seeded Thiem.
Thiem, the runner-up to the great Spaniard in the 2017 Madrid final, had been the last player to defeat Nadal on clay at the 2017 Italian Open in Rome -- 357 days ago.
The shock result also means that Roger Federer will reclaim the world number one ranking on Monday.
"A player like Dominic has a lot of potential. Whenever he plays really well, it's very difficult to stop him," admitted Nadal.
"He played well and I didn't play well. We're not playing a game where the differences are big or massive. The differences are very small."
Five-time Madrid winner Nadal, who will be bidding for an 11th French Open title in Paris later this month, had won his previous 21 matches on clay.
That run included him surpassing John McEnroe's 1984 record of 49 sets won in succession on the same surface during his last-16 duel with Diego Schwartzman on Thursday.
However, in their ninth career clash, Thiem put Nadal under pressure almost from the start on Friday and broke for a 4-3 lead in the first set.
A double fault and a long forehand allowed Nadal to retrieve the break at 5-5.
'He killed me in Monaco'
It was a brief reprieve as a lazy forehand volley gave Thiem a 6-5 advantage and the set came the Austrian's way in the 12th game off an ace.
Thiem, watched by his girlfriend -- and WTA star -- Kristina Mladenovic, turned the screw for a 3-1 lead in the second set but Nadal battled back to 3-3.
That was as good as it got for the Spaniard as Thiem broke again for 4-3, held to love for 5-3 and secured victory with another blistering forehand.
Thiem, who next faces South Africa's Kevin Anderson for a place in Sunday's final, joins Novak Djokovic and Gaston Gaudio as just the third man to have beaten Nadal three times on clay.
"It takes a really good match to beat Rafa on clay. But I think a very important thing also was today that I went in with the attitude that I can beat him," said Thiem.
"Two weeks ago in Monte Carlo, he killed me love and two. It was very important I went in with a positive attitude, with an attitude to win. I should go into every match like this against him."
World number eight Anderson battled into his first Masters semi-final with a 7-6 (7/3), 3-6, 6-3 win over Serbian qualifier Dusan Lajovic.
Anderson, the 31-year-old US Open runner-up last year, hit 15 aces and 35 winners.
Canada's Denis Shapovalov became the youngest player to reach the Madrid semi-finals with a 7-5, 6-7 (6/8) 6-4 win over Britain's Kyle Edmund.
The 19-year-old Shapovalov, ranked at 43 in the world, fired 29 winners past Edmund, a surprise semi-finalist at the Australian Open in January.
Shapovalov, who had never won a clay-court match on the main tour before Madrid, will be playing in his second Masters semi-final where he will on Saturday face German second seed Alexander Zverev.
Zverev defeated John Isner 6-4, 7-5 to gain revenge for his defeat to the American in the Miami Masters final in April.
Zverev has now reached five Masters semi-finals in the last year.
Dutchwoman Kiki Bertens continued her remarkable run at the Madrid Open by thrashing French seventh seed Caroline Garcia to reach the womens' final.
The world number 20 will face Petra Kvitova after brushing aside Garcia 6-2, 6-2.
Bertens had already stunned world number two Caroline Wozniacki and five-time Grand Slam champion Maria Sharapova earlier in the week.
Two-time Madrid champion Kvitova reached the final with a 7-6 (7/4), 6-3 win over Czech compatriot Karolina Pliskova.