(BUTNER, N.C.) -- Convicted Ponzi schemer Bernard Madoff was forced to "let...go" of his wife almost a year ago and is plagued by "horrible nightmares" as he sits in a North Carolina prison, he told ABC News' Barbara Walters in an exclusive interview.
Though he said he "can live with" the anger of people he defrauded out of billions of dollars, and he is adjusting to the rhythms of life in prison, even at 73 years old, he is troubled by anger and turmoil within his own family.
"Not seeing my family and knowing they hate me" is the worst thing about being in prison, he said. "I betrayed them."
Asked what he'd like to say to his grandchildren, he said, without apparent emotion, "I am sorry to have caused them pain."
As he sat across from Walters during a two-hour conversation at the Federal Correction Complex at Butner, N.C., on Oct. 14, Madoff often seemed to be trying as much as possible to feel no pain himself.
Madoff said he passes the time by reading, recently finishing a book about Wall Street robber barons.
The man who ran a Ponzi scheme of more than $60 billion has held six or seven different jobs in prison, he said, and he makes $170 a month.
He said he is relieved to be free from years of fear he'd be discovered as a fraud and finally has overcome thoughts of suicide.
Repeatedly throughout the interview he told Walters that he was guilty of the crimes that put him in prison, saying "I deserved to be punished. I deserved to go to jail."
"I feel safer here than outside," Madoff said. "Days go by. I have people to talk to and no decisions to make...I know that I will die in prison. I lived the last 20 years of my life in fear. Now I have no fear -- nothing to think about because I'm no longer in control of my own life."
Though Madoff has people to talk to in prison, his family situation is far more complicated. He has not spoken to his wife, Ruth Madoff, since after the suicide of their son, Mark Madoff, on Dec. 11, 2010. And Mark Madoff's widow, Stephanie Madoff Mack, has told ABC News she holds Bernie Madoff responsible for her son's death and, "I'd spit in his face," if she ever saw him again.
Madoff told Walters that his wife used to visit him at the prison weekly and they spoke on the phone daily. In order to visit Butner, N.C., Ruth Madoff would drive 12 hours alone, stay at a motel overnight and drive 12 hours back to Florida, which was hard on her.
But after their son's suicide, the couple had an emotional final meeting at the prison at which Ruth Madoff "asked me to let her go, which I understood," Madoff said.
Madoff told Walters he has not reached out to his wife since that final meeting.
Copyright 2011 ABC News Radio