Potter was plucked from the Seagulls last month after Chelsea sacked Thomas Tuchel just seven games into the Blues' season.
Brighton sat fourth in the Premier League at the time, but have failed to win in five games since.
"Sometimes life throws you opportunities and you have to take them or not," said Potter at his pre-match press conference.
"The timing wasn't great for anybody but sometimes that happens.
"Then it is about the decision, the decision to work here was too good to turn down, I thought."
Potter added he is "not naive" to the prospect he may get a frosty reception from the home fans at the Amex on Saturday.
But he is more concerned the damage his old club could do on the pitch despite a poor run of results under new boss Roberto De Zerbi.
"I think they have been unlucky, that is the only thing I can say. The performances have been good, there are positives there. You need luck and I don't think they have had any," said Potter.
"I think Roberto has been in a few weeks now, you can see the ideas, the differences, there is definitely change there I would say.
"There is a new manager, new ideas and there is always something to learn, something to surprise."
Chelsea are unbeaten in nine games under Potter, but back-to-back draws in the Premier League have left them still seven points off the top in fifth.