Colley was born in Klamath Falls, Oregon, to Muriel and Pete Colley. He attended Klamath Union High School and actively played American football and track & field, which led to an unsuccessful try-out for the 1960 Summer Olympics. He later attended the University of Oregon and studied architecture.
Later, he became a theatre member and spent five years learning his craft in various productions in San Francisco.
Some of his better known roles include Gideon on Daniel Boone, Ongaro in Beneath the Planet of the Apes, Joshua in The Legend of N***r Charley, and Sheriff Little in the 1980s TV series The Dukes of Hazzard.