Huffman was born on December 9, 1962, in Bedford, New York, into a wealthy family, the daughter of Grace Valle (née Ewing) and Moore Peters Huffman (1910–1987), a banker and partner at Morgan Stanley. Her parents divorced a year after her birth, and she was raised mostly by her mother. She has six sisters, Mariah, Betsy, Grace, Isabel, Jessie, Jane (1949–2013), and a brother, Moore Jr. In the 1970s her mother left New York and bought property in Snowmass, Colorado, where Felicity and her siblings spent their youth. Her great-grandfather was Gershom Moore Peters (1843–1919), founder of the Peters Cartridge Company and prominent Baptist minister, author of The Master. Another great-grandfather, Frederick Berthold Ewing, graduated from Yale University and became a prominent St. Louis businessman. Her great-great-grandfather was Joseph Warren King (1814–1885), founder of the King Mills Powder Company. She has German, English, Scots-Irish, Scottish, and French-Canadian ancestry.