In the present review, we summarize scientific knowledge concerning sex differences in world record performance and the influence of sport discipline and competition duration. In addition, we discuss how physiological factors relate to sex dimorphism. While cultural factors played a major role in the rapid improvement of performance of women relative to men up until the 1990’s, sex differences between the world’s best athletes in most events have remained relatively stable at approximately 8-12%. The exceptions are events in which upper-body power is a major contributor, where this difference is more than 12%, and ultra-endurance swimming, where the gap is now less than 5%. The physiological advantages in men include a larger body size with more skeletal muscle mass, a lower percentage of body fat, as well as greater maximal delivery of anaerobic and aerobic energy. The greater strength and anaerobic capacity in men normally disappears when normalized for fat-free body mass, whereas the higher hemoglobin concentrations leads to 5-10% greater maximal oxygen uptakes in men also with such normalization. The higher percentage of muscle mass in the upper-body of men results in a particularly large sex difference in power production during upper-body exercise. While the exercise efficiency of men and women is usually similar, women have a better capacity to metabolize fat and demonstrate better hydrodynamics and more even pacing, which may be advantageous in particular during long-lasting swimming competitions.