Carlo Ancelotti's side trails leader Barcelona by two points after losing at Vallecas, where referee Juan Martinez Munuera and the VAR played a significant part on Tuesday.
Luka Modric's penalty and an Eder Militao header had put Madrid into the lead after Santiago Comesana's opener, before Alvaro Garcia equalised during a breathless first half.
Yet there was to be more drama when Dani Carvajal was deemed to have encroached as Thibaut Courtois saved a 67th-minute penalty from Trejo, who converted the retake to secure victory.
Comesana fired Rayo into a fifth-minute lead after angling a first-time left-footed shot past Courtois following Fran Garcia's whipped left-wing cross.
Martínez Munuera gave a 37th-minute penalty to Madrid after consulting the touchline monitor following a tangle of legs that saw Marco Asensio felled by Fran Garcia, with Modric coolly converting down the middle from the spot.
Militao directed a header into the bottom-left corner from Asensio's in-swinging corner to give Madrid the lead, only for Alvaro Garcia to power a left-footed strike past Courtois to equalise three minutes later.
Courtois saved Trejo's penalty after Carvajal was adjudged to have handled in the second half – again following a review of the touchline monitor – but the Madrid right-back was penalised for encroachment, allowing the Rayo midfielder to find the bottom-right corner at the second attempt.
That was to prove the winner, although Madrid almost salvaged a draw when Rodrygo spurned a glorious 89th-minute chance, turning over Asensio's centre.
