A firm that facilitated a phone subscription scam has received a record £600,000 fine from a UK watchdog.