It is sometimes said that headphone amplifier output impedance doesn't matter much. The reasoning comes from basic electrical engineering and builds on the observation that headphone impedance is much flatter and higher than we see with speakers (a common point of reference). The problem is that while the reasoning is solid, the premises are wrong.
If a headphone has flat impedance, then we should see no difference in frequency response with different amplifier output impedances. The source of this is basically what is called a voltage divider, where the voltage drop from the amp output to ground is split between the amplifer impedance and the headphone or speaker impedance. Assuming the amp exhibits flat impedance with frequency, if the headphone has different impedances at different frequencies the voltage will be split differently at different frequencies. That will lead to a frequency response difference from what was encoded in the input signal. But, if there is no variation in headphone (or speaker) impedance at different frequencies, then the voltage split is constant and the amp output impedance has no effect on frequency response.
A secondary factor to take into consideration is that headphone impedances (30-600 ohms) tend to be much higher than speaker impedances (4-8 ohms). If headphone amps had low (less than 1 ohm) output impedances like stereo power amps, then even with some variation in headphone impedance, we