Fisher signed his first professional contract at Blackburn Rovers in January 2016. On August 29th 2017 he made his first team debut, keeping a clean sheet in a 1–0 EFL Trophy group stage win over Stoke. 2 months later he signed an extended deal keeping him at the club until 2021. Over the following two seasons, Fisher was sent out on loan to FC United of Manchester, Northampton Town and MK Dons. In October 2020, Fisher re-joined MK Dons on a permanent deal for two years. In January 2022, he joined Swansea City on a permanent four-and-a-half-year deal for an undisclosed fee.