Miralem Pjanic and Dembele each missed from the spot but the latter eventually made the breakthrough with a powerful strike.
The visitors made sure of the win in the very last moment through Braithwaite to ensure there would be no repeat of Wednesday's action, when Real Madrid suffered one of the biggest shocks in the competition's history by losing to Alcoyano.
Barca struggled to turn possession into chances, their first real effort of note coming via a Francisco Trincao volley from Pjanic's corner that was parried by Ramon Juan.
Pjanic had the chance to break the deadlock from the spot after Ronald Araujo took a high foot to the head, but his penalty was expertly saved.
They started the second half with more purpose and should have scored when Martin Braithwaite shot straight at Ramon after Antoine Griezmann diverted the ball into his path.
Griezmann then blasted over the bar with the goal at his mercy after a clever quick free-kick from Pjanic, and Neto had to make a flying save to parry an Agus Medina strike barely a minute later.
Substitute Dembele was given the chance to get the breakthrough 11 minutes from time after another high boot in the box, this time from Pol Moreno on Clement Lenglet, but Ramon held his nerve and blocked the penalty from the middle of his goal.
Braithwaite sliced another clear chance wide of the right-hand post and Ramon came to the rescue again to tip a Pjanic half-volley onto the crossbar and away.
Yet within two minutes of the beginning of extra time, Ramon's resistance was broken, the keeper getting two hands to Dembele's strike from 20 yards out but unable to keep the ball out of the net.
Konrad de la Fuente almost made it 2-0 immediately after coming off the bench, but Ramon threw himself to his left to make another fine stop, before Albert Estelles earned a second booking in the closing moments for a poor challenge on the excellent Araujo.
With the final move of the game, Dembele combined expertly with Pedri, who unselfishly teed up Braithwaite to drill into the unguarded net.
What does it mean? Barca avoid more disappointment – just
This was the first time in their history that Barca had three consecutive games go to extra time, having twice gone the distance in the Supercopa de Espana.
After losing the final to Athletic Bilbao, it was important for Koeman's changed side to respond here and he will be relieved that they eventually found a way through.
With the two Madrid giants already out, the Copa del Rey looks like Barca's safest bet for silverware this season.
Ramon made seven excellent saves, including the two penalties, and he will be furious he could only parry Dembele's shot into the top-left corner.
Still, were it not for their keeper, Cornella could well have suffered a straightforward defeat in normal time.
Riqui Puig has found it difficult to convince Koeman at times and this was not a performance to earn the trust of his coach.
The midfielder, booked for dissent seven minutes in, managed no shots, crosses or key passes and was taken off at half-time.
Barca are back in LaLiga action away to Elche on Sunday.