Stormers had the majority of possession in Cape Town but failed to turn dominance into points, although they did opt to turn down a number of opportunities to kick for the posts.
Brumbies caught their opponents out with two tries inside the opening 12 minutes, Pete Samu forcing his way over in the corner for the first before Rory Arnold collected a loose ball to score after he had charged down Herschel Jantjies's box kick.
A penalty from Stormers fly-half Joshua Stander was sandwiched between the tries, with the home team having a score of its own chalked off for a knock-on.
However, it was awarded a penalty try on the stroke of half-time when Jantjies's progress was halted by a high tackle from Toni Pulu, who received a yellow card after a lengthy TMO (television match official) review.
Stormers capitalised on the man advantage to take the lead for the first time early after the break, Wilco Louw forcing his way over from close range with Stander adding the conversion.
But Brumbies hit back just before the hour mark, full-back Thomas Banks going over. Christian Lealiifano's conversion edged the visitors back in front and they held on to their slender advantage through to the final whistle, securing a sixth win in the past eight outings on South African soil.