It's always important to place ideas in their historical context. For Smith, Ricardo, Marx et al, the world was largely pre-industrial (psychologically, in the case of Marx, because he totally failed to understand the transformative effects of applying capital to the means of production). Thus labor was the obvious means whereby crops were harvested, baskets were woven, etc.
But the industrial revolution changed everything.
What do we say about labor when a single technician controls a factory full of robots that produce more automobiles in a day than a factory full of human workers could produce in a week?
What do we say about the value of labor when we contrast the hard-learned skills of a neurosurgeon against the labor exerted by a semi-skilled painter of walls?
The fact is, early economists had little understanding of the interaction between labor and productivity and hence their notions of value were naive in the extreme. Marx was a brilliant propagandist but a terrible economist.