In a retrospective session at New York Comic-Con, Peter Capaldi said that playing the title role in British TV juggernaut Doctor Who brought many pleasures but proved all-consuming. After four years, he said he felt it was best to leave before it ever felt like a routine.

“It fills up your life,” he said. “You don’t have a second where it’s not about Doctor Who. It’s a nice way to live.” And yet, he continued, “I really never wanted to get to a place where I knew how to do this because that’s not what being creative is. The actual amount of time we were spending on the show, I realized I was getting the hang of it. And that made me frightened.”

Asked by a fan whether he would consider returning for a special or in any small role down the road, he said, “I think it’s probably time for me to go.”

Capaldi stunned fans early this year with the surprise announcement that he was leaving the show after a well-reviewed, high-rated run. He became the 12th actor to play Doctor Who in the sci-fi series when he took over the role from Matt Smith in 2013 and was the fourth Doctor since the BBC relaunched the series in 2005 (Christopher Ecclestone and David Tennant also both had stints as the Doctor).

The TV mainstay made more news with the casting of Jodie Whitaker as Capaldi’s successor, the first time the title role has been played by a woman.

“It’s in good hands,” he said. “She is really great.” While the two had never met until recently discussing the transition, Capaldi said he discovered the two of them live close to each other in England.

“You’re coming up on a really nice time,” he said. “It’s always really wonderful when there’s a transition to a new doctor.”

A highlight reel of Capaldi’s time on the show preceded the session as well as another compiling fans’ farewell tributes, but no footage was shown of the highly anticipated Christmas special, which will screen in theatres in addition to its broadcast window. Asked about plot details, Capaldi spoke carefully so as not to spoil anything. But he said the special taps into the show’s legacy. “My doctor is refusing to regenerate,” he said. “He doesn’t want to turn into someone else. He has to be persuaded to do that. The person to do that is the first Doctor.”