Juve, held 2-2 at Sassuolo last time out, took the lead through Cristiano Ronaldo but was pegged back before the break by Luiz Felipe's header in a gripping Serie A encounter.
Juan Cuadrado's red card opened the door for Lazio and they charged through it, Luis Alberto setting up Milinkovic-Savic for a superb strike to edge in front.
There was time for more drama, though, Juve goalkeeper Wojciech Szczesny saving Ciro Immobile's penalty, and Caicedo added a third deep in added time to leave Inter two points clear at the top.
