The league's bottom club had lost all 14 games this season prior to the match but snatched a 2-2 draw right at the death at its Ciro Vigorito home, leaving Gattuso and his Rossoneri players stunned.
Milan had won only two of its previous nine Serie A games heading into the encounter and looked short on confidence in the early stages of Gattuso's first game in charge since succeeding Vincenzo Montella as head coach.
Giacomo Bonaventura got the breakthrough for the visitor before half-time but it was pegged back by George Puscas before the hour mark.
The goal had Benevento fans dreaming of a first point in Italy's top flight until Kalinic struck six minutes later, but Alessio Romagnoli's red card gave the home fans a glimmer of hope of a famous result.
The opportunity looked to have passed them by until the fifth minute of second-half stoppage time, when Brignoli's brilliant header sent the crowd into raptures and dealt another blow to Milan's UEFA Champions League ambitions.
Gattusso started with a 3-4-3 set-up that saw Bonaventura and Suso playing off Kalinic in attack, with Andre Silva surprisingly consigned to the substitutes' bench.
It was Kalinic who came closest to breaking the deadlock in the opening quarter of an hour, drawing a good save from Brignoli in the Benevento goal before Suso aimed the rebound straight at his gloves.
Benevento missed a clear chance to break the deadlock when Vittorio Parigini headed wide at the far post from a Marco D'Alessandro cross, and they were made to pay seven minutes before half-time.
Bonaventura's header from Franck Kessie's cross was glanced on by Kalinic and cleared off the line, but the former Atalanta man stooped to nod the rebound into the bottom-right corner and put Milan ahead.
It was harsh on Benevento given their first-half display but they hit back only five minutes after the restart, with Puscas turning home the rebound after Gaetano Letizia saw a thumping volley from the edge of the box saved by Gianluigi Donnarumma.
Six minutes later, luck was again on Milan's side. Having been awarded a corner when it looked as though Benevento should have had a goal kick instead, the visitors took full advantage of some slack marking, with Kalinic rising unchallenged to nod in Bonaventura's cross.
Benevento heads were bowed but they soon began to threaten a second equaliser, with Donnarumma called upon to save well low to his left and keep out D'Alessandro's low drive from just inside the penalty area.
Milan's task became more difficult when Romagnoli earned a second booking for a lunging challenge that brought down Letizia but the visitors defended stoutly, led by captain Leonardo Bonucci, as Benevento pushed forward in the closing stages.
iT looked to have done enough until the fifth added minute at the end of the match, when Danilo Cataldi's free-kick from the left was steered superbly by Brignoli into the bottom-right corner, leaving Gattuso shell-shocked as the home dugout went wild.