Robinson was born on January 31, 1919, into a family of share croppers in Cairo, Georgia. He was the youngest of five children born to Mallie (McGriff) and Jerry Robinson, after siblings Edgar, Frank,Mack, and Willa Mae. His middle name was in honor of former President Theodore Roosevelt, who died 25 days before Robinson was born. After Robinson's father left the family in 1920, they moved to Pasadena, California.

The extended Robinson family established itself on a residential plot containing two small houses at 121 Pepper Street in Pasadena. Robinson's mother worked various odd jobs to support the family. Growing up in relative poverty in an otherwise affluent community, Robinson and his minority friends were excluded from many recreational opportunities. As a result, Robinson joined a neighborhood gang, but his friend Carl Anderson persuaded him to abandon it.