Charles Wesley Jarvis, the second son of the portrait painter John Wesley Jarvis, was born in New York City. His mother died the following year, and he and an older brother, John, were raised by her relatives on Long Island. Jarvis' father spent many years away from home working as an itinerant painter.

Jarvis apparently received his earliest artistic training from Henry Inman, his father's former assistant. Jarvis apprenticed with Inman and worked with him in Philadelphia between 1831 and 1834. Jarvis then returned to New York and opened a studio the following year; within a few years he was exhibiting at the National Academy of Design. During the next two decades his patrons included members of prominent families. He maintained a studio in New York City despite moving his home to Newark, New Jersey.

Although Charles Jarvis' career seems to have been overshadowed by that of his more famous parent, recent scholarship has reattributed many works once thought to be his father's to his hand.