Ex-Perth midfielder Brimmer scored twice in the space of four minutes to give the Victory their first win of the campaign – his second coming from the penalty spot in the final minute of regulation time.
Robbie Kruse made his first start for the Victory in almost a year and he was denied a goal to mark the occasion by a brilliant save from Tando Velaphi, who started in place of Liam Reddy for Perth after they conceded eight goals in their first two games.
Velaphi kept out Brimmer either side of half-time and the Glory took advantage by moving in front when Bruno Fornaroli nodded Carlo Armiento's left-wing cross in at the back post just after the hour mark.
The Perth goalkeeper made six saves but was finally beaten when Brimmer rose highest to head Adama Traore's cross into the bottom-right corner.
After Josh Rawlins handled a corner inside the box, Brimmer swept the spot-kick past Velaphi as the Victory avoided defeat at AAMI Park after a club-record three-game losing streak at the venue in the A-League.