Fenerbahce secured a 2-1 win over Kayserispor that saw them secure third place in the Turkish Super Lig and qualify for next season's UEFA Europa Conference League.