Kolbe crossed either side of the break as he took his tally to 10 tries in as many games since the sport's restart in September.
For Ulster, it was a first loss at its Kingspan Stadium since October 2018.
The host's coach Dan McFarland made five changes from its Pro14 hammering of Edinburgh two weeks ago as hooker Rob Herring and three-quarter Jacob Stockdale returned from Ireland duty.
Australia lock Rory Arnold and New Zealand's Rugby World Cup winner Charlie Faumuina returned to Ugo Mola's starting line-up after last weekend's league victory at Bayonne.
McFarland's men led 12-0 after 16 minutes as Herring and five-eighth Ian Madigan crossed in greasy conditions.
Mola's side responded immediately as Kolbe chipped and chased to finish from 50 metres in a sublime individual effort as he beat three defenders.
The joint-four time record winner led 14-12 at the break as Antoine Dupont broke from halfway and sold opposition full-back Michael Lowry a dummy to slide over.
The home team dominated after the interval as scrum-half John Cooney kicked a penalty and Herring added a second either side of 2.13-metre Arnold's first try of the season for a 22-21 lead with a quarter of an hour to play.
Kolbe's second moment of brilliance came as the former Stormers winger received the ball on the left touchline.
He side-stepped Stockdale within centimetres of the whitewash and then found his way through for a try despite falling to all fours.
Thomas Ramos made up for the missed conversion with a 50-metre penalty goal to seal the win and a repeat result of last season's quarter-final.
Earlier, Argentina winger Santiago Cordero dived over with seven minutes remaining in Bordeaux-Begles's 16-12 victory at Northampton Saints.
Saints coach Chris Boyd said after the loss he will focus on the league and pick youngsters for next weekend's trip to Leinster.
On Sunday (AEDT), four-time winner Leinster heads to Montpellier as Toulon host Sale Sharks, which will be without head coach Steve Diamond after he resigned during the week.
On Monday (AEDT), champion Exeter Chiefs welcomes Glasgow Warriors with 16 players remaining from October's final win over Racing 92, which in turn hosts Connacht.