For as long as light and matter have partnered, impurities have played a role in optical system performance. This remains generally true for photonic heat engines and especially the case for optical refrigeration. Building on the history of light and glass, including the materials development of low loss telecom fibers, this paper briefly discusses the sources of heat generation in materials and all-material means for their reduction. Particularly attention will be paid to active optical fibers and connect thermal management to parasitic optical nonlinearities, both critical to high and low power amplifier and laser systems.