It is important if:

1. Someone's life depends on your code. A function taking 25ms to execute in someone's heart rate monitor is probably a bad idea.

I personally take a two pronged approach -- there are micro-optimizations you can do that won't affect readability -- obviously you want to use those. But if it affects readability, hold off -- you won't get much benefit and it actually might take you longer to debug in the long haul.