Harry Kane looked to have salvaged a point for Tottenham by cancelling out Tommy Horvat's opener at Fazanerija, but Amadej Marosa scored a dramatic winner with the final kick on Friday (AEDT).
Spurs made nine changes from their 2-1 Premier League win over Leeds United last weekend and played more than an hour with 10 men following Ryan Sessegnon's dismissal.
But the defeat – Tottenham's first under head coach Conte since his appointment earlier this month – will still go down as one of the worst in its European history against a side with a UEFA coefficient ranking of 341.
Spurs can now no longer finish top of Group G, meaning a two-legged play-off awaits in February should they first match Vitesse's result in the final round of group fixtures in a fortnight.
Conte, who has a record of two wins, a draw and a defeat from his four games since replacing Nuno Espirito Santo, accepted an improvement is required in every department ahead of hosting Rennes.
"This is a bad loss for us. The performance was poor," the former Inter boss said. "We know we have a lot of work ahead of us. This is a difficult moment for us.
"We need patience to improve, to be good again. We were poor not only technically and tactically but also mentally. It will take time to bring Tottenham back.
"I don't take this loss personally. I accepted this challenge to improve and build Tottenham to a better level. But it will take time.
"I want to thank our supporters for their support. We will analyse the game and I will talk to the players then."
Tottenham are seventh in the Premier League with 12 matches played, four points adrift of the top four, and are back in action against Burnley on Sunday.
"I think this team is very young but at the same time we have to work and improve because the details move the final result," Serie A-winning coach Conte added.
"I did not like what happened tonight, the performance. I must be honest and tell you that after three and a half weeks, I am starting to understand the situation. I can tell you that the situation is not simple.
"It is not simple because in this moment because for sure the level of Tottenham is not so high. There is an important gap to the top teams in England. Of this we must not be scared.
"I am here to work, here to improve the situation. I know we have to work hard but we have also to improve the quality of the situation, because if we want to be competitive."
Kane, one of two players to retain his place from the Leeds game, was on target for Spurs but otherwise looked off the pace – as did the rest of his team-mates.
"We started poorly, conceded an early goal and were left with ten men," said Kane, who has scored eight goals in his last four outings for club and country.
"In the second half we put on the pressure, scored but then let them get a goal in the last second. We're disappointed, we're angry and we have to learn from this."