The Reds looked set to ease to victory after Sadio Mane marked his 100th league appearance for the club with his 50th goal five minutes before half-time, but Jurgen Klopp's men missed chance after chance to extend their lead.
Leicester, having scarcely threatened, struck through James Maddison with 10 minutes remaining as Foxes boss Brendan Rodgers threatened to end the Reds' streak on his first return to Merseyside since leaving the club in 2015.
However, there was time for late drama as Mane won a penalty having been clipped by Marc Albrighton moments after Mohamed Salah hobbled off with an ankle issue, allowing Milner to net a last-gasp winner.
Liverpool has now triumphed in 17 consecutive top-flight matches — one short of the all-time record — and leads Manchester City by eight points in the table ahead of the champion's Monday (AEDT) meeting with Wolverhampton Wanderers.
Trent Alexander-Arnold was involved in much of Liverpool's best attacking play from right-back and crafted early opportunities for Milner and Roberto Firmino, yet both were squandered.
It was a ball from Milner on the opposite flank that led to the breakthrough as Jonny Evans failed to cut out a pass in behind and was punished ruthlessly by Mane, who might have had a second before the break but shot straight at Kasper Schmeichel.
Salah was denied by Schmeichel shortly after the restart and Evans deflected the ball narrowly wide off Mane, while both Firmino and Andy Robertson went close.
This profligacy set up a tense finish and Maddison looked to have snatched an improbable point as he fired under Adrian after good work from Ayoze Perez.
Salah then departed after a rugged challenge from Hamza Choudhury, but Liverpool did not give up and Albrighton tangled with Mane in the area, with a VAR (video assistant referee) check siding with referee Chris Kavanagh on the award of a spot-kick, which Milner rifled home.
The full-time whistle was greeted with scenes of jubilation from Klopp and the Liverpool supporters, while Perez appeared to clash with Robertson, prompting a melee that granted a gripping contest a suitably chaotic ending.