Karim Benzema, who has now scored more LaLiga goals than any other player since the resumption in June, opened the scoring with a thunderous strike – his fifth goal in his last four top-flight meetings with Valencia.
But Benzema's brilliance was not enough to inspire Zinedine Zidane's defensively suspect side, whose misery was compounded when Sergio Ramos handled to concede a third spot-kick, which Soler duly converted to condemn Madrid to a second LaLiga defeat of 2020-21.
Madrid struck first – Benzema catching out Jaume Domenech with a thumping 23rd-minute strike – but Lucas Vazquez's handball handed Valencia an opportunity to equalize.
Yet the hosts did so the hard way. After Soler saw his penalty saved and hit the post on the rebound, Musah slammed home, only for it to be disallowed after VAR spotted the 17-year-old had encroached into the area.
Referee Jesus Gil Manzano ordered the penalty to be re-taken and, although Thibaut Courtois again guessed the right way, Soler beat Madrid's goalkeeper at the second time of asking.
More drama followed as Valencia took the lead before half-time – Varane inadvertently looping Maxi Gomez's cross over Courtois, who failed to stop the ball from crossing the line.
However, it was only given after a long VAR check, which also deemed Denis Cheryshev not to have fouled Marco Asensio in the build-up.
Courtois made a fantastic save straight after the restart, tipping Lee Kang-In's strike onto the upright, but he was soon picking the ball out of his net again.
Marcelo scythed down Gomez, and Soler stepped up to convert into the bottom-right corner.
A remarkable hat-trick, and a brilliant Valencia victory, was rounded off in the 63rd minute – Soler picking out the same corner after had Ramos hit the ball away from Musah.
What does it mean? Gracia ends Madrid hoodoo
At the seventh attempt, Valencia coach Javi Gracia has managed his first LaLiga win over Madrid, and the triumph takes Valencia up to ninth, level on 11 points with Barcelona.
They are five behind Madrid, who sit fourth, four points off pacesetters Real Sociedad heading into the international break. Zidane also lost Benzema to what appeared to be a groin problem late on.
Soler into LaLiga's record books
Soler's first attempt from 12 yards was poor, but the following three were all dispatched with supreme confidence. In scoring a hat-trick of spot-kicks, the 23-year-old became the first player to score three penalties in a single LaLiga match in the 21st century, and he has been directly involved in five goals in his last three meetings with Madrid at Mestalla.
Varane's bad luck strikes again
While Madrid's entire defence must shoulder the blame, one man in particular stands out. Varane's attempted clearance from Gomez's cross which resulted in Valencia's second goal was a dreadful one, and the France international has now scored two own goals in his last five games for Madrid in all competitions, as many as he had in his previous 324 matches.
Madrid travel to Villarreal in their first fixture back after the international break, while Valencia take on Deportivo Alaves a day later.