With your setup running through a Focusrite (I’m assuming Scarlett?) From what I’ve learned, it would be strongly recommended to push the signal through to a Headphone Amp. As I said, I’m impressed with the combination I am running now, but eventually I will invest into a headphone amp by Schiit (Asgard 2 or Valhalla 2, not sure which model).
When I was first researching amps affiliated with headphones I used Head-Fi forums and Gearslutz for real life experiences from other engineers who had similar issues. I remember a particular case where someone had a Focusrite Scarlett 2i2 and just like you was curious if he needed a headphone amp to drive the impedance level because the 2i2 and similar models have limited voltage and power capabilities.
If it’s permitted (I don’t know if it’s allowed to post links to other forums here) I can find it and post the case study, it’s very informative. If anyone has similar experiences with Scarlett audio interfaces and success without headphone amps, please correct me if I am wrong. 