Rayo took an early lead in a see-saw game before Real went 2-1 ahead. Rajo equalised before half-time before winning the match with a twice-taken penalty from Oscar Trejo in the 67th minute.
Playing without Ballon d'Or winner Karim Benzema, Real Madrid faced a tough opponent in a sold-out stadium in the working class suburb of Vallecas.
Santiago Comesana put the home side ahead to raucous scenes after just five minutes before two goals in the space of four minutes turned the match Madrid's way.
First Luca Modric levelled from the penalty spot in the 37th minute before Eder Militao fired Carlo Ancelotti's previously unbeaten side into the lead.
It was not to last, however, as Rayo were level at the break with Alvaro Garcia getting the better of a goalmouth scramble and snapping a shot through a sea of legs.
In a bad tempered and hotly-disputed game Real's Belgian goalkeeper Thibaut Courtois saved a penalty from Trejo on 67 minutes but was adjudged to have moved early.
Trejo showed no nerves converting the retaken spot-kick.
Barcelona beat Almeria 2-0 with goals from Ousmane Dembele and Frenkie de Jong to move top of La Liga on Saturday on an emotional night as Gerard Pique retired.
Earlier Monday, Real were draw against Liverpool in the Champions League for a double header in February and March.