By the 1960s, with his career reduced to bit-parts in films and guest spots on TV, he had been arrested for drug charges and sentenced to rehab. After his parole, he tried to establish himself as an avant-garde artist in New York City, including a stint as a member of Andy Warhol's group of artists, The Factory. His last known film role was in the Warhol-produced "Dirt". He left The Factory sometime in 1967 or early 1968, but his addiction to drugs took a toll on his life.
His body was found on March 30, 1968 in an abandoned building, his body was unclaimed and buried in Potter's Field on Hart Island. In 1969, his mother with the help of the Disney Studio tracked down his body and it was confirmed with a fingerprint match provided by NYPD. The public did not know of his death until it was revealed in 1972, when Song of the South was re-released in theaters.