Didier Deschamps' side appeared to be coasting towards the quarter-finals on Monday after overcoming a 1-0 half-time deficit to lead 3-1 through Karim Benzema's double and a sumptuous long-range effort from Pogba.
But Haris Seferovic's second of the game nine minutes from time preceded a last-gasp equaliser from Mario Gavranovic, the build-up to which saw Pogba lose the ball too easily in midfield.
During the game Pogba was seen clashing angrily with team-mate Adrien Rabiot, with the families of both players also reportedly remonstrating in the stands.
France went on to lose 5-4 in a penalty shoot-out, with Kylian Mbappe missing the crucial spot-kick, and former Arsenal and Barcelona midfielder Petit was at a loss to explain the world champions' implosion.
"I don't really understand what happened to France versus Switzerland," Petit told PaddyPower.
"During the game we only played well for 20 minutes after [Hugo] Lloris stopped the penalty. After it, we looked confident and like a good team.
"But then there was one incident – Pogba lost the ball for the second Swiss goal – and France collapsed.
"We lost everything. Concentration, team spirit, desire – and the tie. Worst of all, we started fighting on the pitch between ourselves.
"There was a big clash between Rabiot and Pogba, who lost the plot, after the second Swiss goal and from then on we had no control, no confidence.
"We looked average – it's hard to explain."
It was a strange outing for Pogba, who had largely impressed with an industrious performance in central midfield.
No France player won more than his 19 duels, while his 72 successful passes was also a team high and he had an impressive 93.6 per cent passing success rate in the opposition half.
By contrast, Mbappe's miss in the shoot-out was emblematic of a disappointing tournament in which the Paris Saint-Germain superstar failed to score.
None of his six shots against Switzerland managed to hit the target, including a glorious opportunity in extra time.
Petit alluded to the issues Mbappe had pre-tournament with Olivier Giroud, who had complained of a lack of service during a warm-up friendly against Bulgaria, leading to reports of a dispute between the two.
"Mbappe was asking for more responsibilities before the tournament, and there were stories about not passing the ball to Giroud," Petit added.
"There was even fighting in the stands between Rabiot, Mbappe and Pogba's families after the second goal.
"Then Mbappe's penalty is saved in the shootout. We looked so far away from the World Cup-winning team.
"It reminds me of our team in 2002, when we went out in the group stage of the World Cup. It's a real shame because it doesn't need to be like this."