Goals by midfielder Robert Andrich, his second of the night, and winger Diaby in a five-minute burst late in the game swung the result for hosts Leverkusen.
"We were shocked to be 2-1 down and needed a bit of time to regain control," admitted Leverkusen's double goal-scorer Andrich.
"We had to remain patient and deserved to get the result."
While the Germans reach the knock-out stages as Group G winners, the away defeat means Celtic will finish in third spot after second-placed Real Betis won 2-0 at Ferencvaros Budapest.
"(Conceding) two late goals was disappointing and we came up short," lamented Celtic's Australian coach Ange Postecoglou.
"We lost some composure at the end and didn't make good decisions."
Andrich's early header for Leverkusen was cancelled out when Josip Juranovic equalised with a first-half penalty before Jota was left unmarked in the area to fire Celtic ahead for the first time just after the break.
Leverkusen midfielder Andrich equalised for the hosts eight minutes from time before Diaby's volley sealed the win.
"It's not easy to finish top of the group with a game left. At the end we were rewarded for our hard work," said Leverkusen coach Gerardo Seoane.
The result preserves Leverkusen's record of having never lost at home in the Europa League group stages, while Celtic must wait for a first win on German soil in European competition.
Leverkusen struck early when Andrich outleapt his marker Stephen Welsh at a corner and powered his header past Celtic's former England goalkeeper Joe Hart.
Celtic started applying pressure
A great chance went begging when Japan striker Kyogo Furuhashi got in behind the defence and his speculative shot to the far post flew just wide of Jota.
The Scottish giants drew level five minutes from the break when Juranovic chipped in from the penalty spot after home goalkeeper Lukas Hradecky fouled Furuhashi went he went to punch clear a cross.
Leverkusen made a strong finish to the first-half as Florian Wirtz and Moussa Diaby both hit the woodwork, but it stayed 1-1 at the break.
Hart came to Celtic's rescue early in the second-half when he scrambled back to deny Jeremie Frimpong moments after blocking a Diaby shot.
At the other end, Jota, who is on loan from Benfica, fired Celtic in front when Furuhashi spotted the Portuguese unmarked at the edge of the area to blast his shot past Hradecky.
Leverkusen roared back as Andrich beat Hart for the second time five minutes before Diaby volleyed in the winner after being picked out unmarked by a Florian Wirtz pass.
Europa League results - collated
Galatasaray (TUR) 4 (Cicaldau 12, Caleta-Car 30-og, Feghouli 64, Babel 83) Marseille (FRA) 2 (Milik 69, 85)
Lokomotiv Moscow (RUS) 0 Lazio (ITA) 3 (Immobile 56-pen, 63-pen, Pedro Rodríguez 87)
Midtjylland (DEN) 3 (Sviatchenko 2, Isaksen 48, Evander 90'+3-pen) Braga (POR) 2 (Ricardo Horta 43, Galeno 85)
Red Star Belgrade (SRB) 1 (Ivanic 57) Ludogorets (BUL) 0
Bayer Leverkusen (GER) 3 (Andrich 16, 82, Diaby 87) Celtic (SCO) 2 (Juranovic 40-pen, Jota 56)
Real Betis (ESP) 2 (Tello 5, Canales 52) Ferencvaros (HUN) 0
Rapid Vienna (AUT) 0 West Ham (ENG) 2 (Yarmolenko 40, Noble 45+2-pen)
Dinamo Zagreb (CRO) 1 (Menalo 35) Genk (BEL) 1 (Ugbo 45'+1)
Brondby (DEN) 1 (Uhre 51) Lyon (FRA) 3 (Cherki 57, 66, Slimani 76)
Rangers (SCO) 2 (Morelos 15, 49) Sparta Prague (CZE) 0
Monaco (FRA) 2 (Volland 28, Fofana 38) Real Sociedad (ESP) 1 (Isak 35)
PSV Eindhoven (NED) 2 (Carlos Vinícius 45-pen, Bruma 56) Sturm Graz (AUT) 0
Leicester (ENG) 3 (Daka 11, Maddison 21, Ndidi 33) Legia Warsaw (POL) 1 (Mladenovic 26)
Eintracht Frankfurt (GER) 2(Kamada 13, Gonçalo Paciência 90'+4) Antwerp (BEL) 2 (Nainggolan 33, Samatta 88)
Olympiakos (GRE) 1 (Soares 90) Fenerbahce (TUR) 0
Spartak Moscow (RUS) 2 (Sobolev 3-pen, 28) Napoli (ITA) 1 (Elmas 64)