Phillips signed for the Charlton Athletic academy when he was 8 and went on to sign his first professional contract in summer 2013 aged 17. He joined National League club Cheltenham Town on a six-month loan in July 2015 and started every game for the "Robins", keeping 10 clean sheets in 26 games, and had his loan extended until the end of the 2015–16 season. He picked up an injury in March 2016, and missed the end-of-season run-in as Cheltenham secured promotion as National League champions. He signed a new two-year contract with Charlton in May 2016. In 2020, Phillips transferred to Cardiff City FC and then on the 27 July 2022, Phillips joined Belgian First Division A side Oostende on a season-long loan deal.