Mr Burrell, who also served as a footman for the Queen prior to working for the late Princess Diana, made the revelation in an interview for Yahoo News’ weekly six-part series, The Royal Box.
Prince Charles, the eldest son of the Queen and Prince Philip, is first in line to the throne, followed by his eldest son Prince William and the Duke of Cambridge’s son, Prince George.
But Mr Burrell claimed Princess Diana told him in her own words that “my husband is not fit for the top job”.
The former butler added that Charles might be too old to take over from his mother and would be wise to let someone younger take his place.
He said: “I don’t think we will ever see King Charles and Queen Camilla on the throne of England.
“I know it’s his right, I know he’s heir to the throne, I know he’s waited his whole life to do it [but] his own wife said: “My husband is not fit for the top job”. Diana said that.
“I don’t think he will get there. I think by the time the Queen dies – and she will live to be Queen as long as there is breath in her body – she will die aged 101, 102, or 103, maybe. Her mother was almost 102 when she died. Longevity is in the family.
“I think, for the sake of the monarchy – by the time Charles gets there, he’s going to be 80.
“Why wouldn’t he say to the Commonwealth, and the people of the United Kingdom: “I’m sure you’d much prefer a younger king and a younger queen to sit on the throne, with those beautiful children, instead of this old thing”. Let’s hope he does that.
A poll from 2016 showed Prince Charles faces an uphill battle to win over the public, despite it being 20 years since he and Princess Diana divorced.
The Opinium Poll, representative of 2,000 adults, revealed that just a quarter of Britons want to see Charles succeed his mother as monarch.
In comparison, more than half (54 percent) said they would prefer to see the succession jump to Prince William.
This scenario would appear extremely unlikely, but Prince Charles’ popularity among the British public appears to have fallen dramatically over the past two decades.
Mr Burrell claimed Prince William and Kate, the Duchess of Cambridge, would one day make a “great” king and queen
He added: “I think William will be a great King and Catherine will be a great Queen and one day Mummy’s ring [Princess Diana’s engagement ring, worn by the Duchess of Cambridge] will be sat on the throne of England.”