The fitness refers ultimately and only to reproductive fitness according to 

Fitness is the end result of the ability to adapt and get selected by nature. Some organisms are better adapted to survive in an otherwise hostile environment.  The fitness, according to Darwin, refers ultimately and only to reproductive fitness. Hence, those who are better fit in an environment, leave more progeny than others.  Adaptive ability is inherited. It has a genetic basis.