I think the full quote in context is instructive. I'll copy from a post I made on Reddit on the topic:
There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.
Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.
-- Donald Knuth, Structured Programming with go to Statements, ACM Computing Surveys, Vol 6, No. 4, Dec. 1974, p.268
The point, and implication, is that there are more important things to worry about than putting your attention to optimization too early. Certainly, you should carefully consider your data structures and algorithms (this is in the 3%) but you shouldn't worry about whether subtraction is faster than modulo (this being in the 97%) until it becomes clear that low-level optimization is necessary.
The former is not necessarily optimization in the sense that your colleagues are thinking, but it is optimization in the sense that poorly-chosen algorithms and data structures are suboptimal.