“Back and Forth”

Why do we always say, “I”m going back and forth” when

 in fact we really don’t go “back” until we have gone “forth”?

Shouldn’t it be … “We go forth and back”????????????????