Is it better to iterate over data once and do multiple complex operations, or to iterate multiple times with simpler operations?

Optimize for clarity. Iteration overhead tends to be low.

Some people think that this kind of code

for x in data:
  a(x)
  b(x)

is going to be much more efficient than

for x in data:
  a(x)

for x in data:
  b(x)

But that is not the case. Usually, there is very low overhead from running the for-loop, with the actual time being spent in the functions a() and b(). Then, there is no significant performance impact from choosing one option or the other. Performance does not provide an excuse to avoid clear code.

Exception: The two code snippets differ in their ordering of the a(x) and b(x) invocations. Sometimes that ordering is relevant. Also, I'm assuming you're iterating over data that already exists in memory. If you're iterating over a generator, then you can only do that once, which may require you to implement some kind of caching.

Your current code is very complex, with nested list comprehensions, mixing different tasks (fetching data, parsing it, and organizing it into a new structure), and using features like the walrus-operator for assignment := where I'm not immediately sure what the data flow for that is going to be.

Before modifying this code, I want to note that you are making network requests. Network requests tend to be orders of magnitude slower than the actual work done by your code. For example, the network request might take 50ms, but all the other parts of your code (iteration, formatting dates, creating dicts and lists) might complete in 5μs. Those numbers are made up, but my point is that there might very well be a factor of 10000× here. Focusing on iteration patterns is close to irrelevant because they might only change the total runtime by 0.01%.

What I would do in your case is to extract the processing for one agent into a separate function. With some additional changes, your code might look like:

tc_builds_by_agent = [full_build_info_for_agent(agent) for agent in agents_id]

def full_build_info_for_agent(agent):
  agent_id = agent['id']
  agent_name = agent['name']

  # TODO build URL using library to avoid injection/escaping issues
  locator = f'agent(id:{agent_id})'
  fields = 'count,build(id,number,status,state,buildTypeId,startDate,finishDate)'
  result = tc_rest_request(f'/app/rest/builds?locator={locator}&fields={fields}&count=10000')

  return {
    'agent_id': agent_id,
    'agent_name': agent_name,
    'builds': list(make_build_info(result))
  }

def make_build_info(result):
  for element in result:
    build = make_one_build_info(element)
    if build:
      yield build

def make_one_build_info(element):
  timestamps = list(element)
  if not timestamps:
    return None

  start = parse_timestamp(timestamps[0].text[:-5])
  end = parse_timestamp(timestamps[1].text[:-5])

  build = dict(element.attrib)  # TODO more processing here
  build['duration'] = str(end - start)
  return build

def parse_timestamp(s):
  datetime.strptime(s, "%Y%m%dT%H%M%S")

Note that if your tc_rest_request() function were async, then this entire code could now be made concurrent with very small changes, potentially allowing for much faster overall processing of all agents.

Even without async, we can now use a thread pool to easily make multiple requests at the same time:

import concurrent.futures

# TODO select a suitable limit on concurrency,
# to avoid overwhelming the server
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
  tc_builds_by_agent = list(executor.map(full_build_info_for_agent, agents_id))

Choosing a sensible structure for our code makes it easier to understand, much easier to extend, and also much easier to optimize it meaningfully. Micro-optimizations like avoiding another loop would be rarely worth it.

As @amon already said, optimize for clarity. Optimize for performance only if you know that you have a performance problem, and then optimize targetting exactly and only the symptom that you were able to measure, typically with a profiler tool. And I bet that the iterating code will be negligible.

Regarding clarity, my guideline is to view elements like classes, fields, methods etc. as vocabulary that you introduce, in order to create a language where you can express your algorithm in a straightforward, easily-readable way.

To use an example from plain English:

If nobody had ever coined the word "car", you'd have to say: "I'm going home in my machine with four wheels, an engine, some seats, and some doors...". Saying "I'm going home in my car" is so much more concise and elegant.

So, introduce vocabulary for abstractions that make sense in your application's context, and that provide more clarity for things that are otherwise hard to see.

• If the two things you do inside the one or two iterations have a common abstraction, i.e. do something that can naturally expressed as one task with one name, go for a single iteration, with this one task (using the proper name) as the body.

• If they are different things x and y, not serving a common purpose, keep them separate like do_x_with_agent(...) and do_y_with_agent(...). I'd prefer to have separate iterations then as well, allowing you e.g. to introduce names like do_x_for_all_agents(...) and do_y_for_all_agents(...) for the iterations.

So, you are stuck with two apparently equal choices :). Kind of things that may jam an engineer.

The way out is to obviously choose an option. No two things in world are exactly equal so there do exist a way out. Just choose the better option.

But how do you know which one is better? Other answers already ruled out performance concerns so there is only one parameter left - cognitive load - to concern.

When you put on socks and shoes which way do you go? Do you get done with one foot altogether then start the second foot? I doubt that. Most people put socks on both feet and only after that start putting on shoes.

The above however is deceiving. It shows preference of getting done with one object (foot) then moving on to next object. Its deceiving because we pick up pairs of socks in our hands so we go for getting done with this cognitive load, before loading the next cognitive load (pair of shoes).

In programming the heaviest cognitive load is the functions to performs. Once you have set up your loop you basically dont care how many objects you process. Dealing with 5 objects is no more a cognitive load than dealing with 5,000 objects as programming goes.

What you are most interested in, therefore focus on, therefore use most mental energy on is what you do with each object. Do you validate the objects before using their values? Should you do operation z as well as operations w, x and y?. These are questions you focus on. Not which objects you work on. It, after all, dont matter for schema rules how many rows are there in a table, to give an example.

Putting objects in a loop, iterating through each of them, running all the functions that have to be run on one of them and then only in next iteration on the next object is the simplest your code can be.

Its also the most "primitive" thing. What else can you do? Putting all your functions in some kind of higher-order lambda array and running loop on that? Dont it sound unnecessary complicated?

Mentally, I would first figure out which identical operations I need to perform on each object (I do that once), then the set of objects to operate on. The set of objects might be more complex than just a loop, so you don’t want to duplicate that.

The simplest code is:

<loop figuring out objects to operate on>
    <operations on that object>

Code complexity = complexity of the loop plus complexity of the operation. Better than complexity of the loop times number of operations, plus number of operations.

Now if you need three operations to figure out that you shouldn’t perform more operations on the same object, then multiple loops gets you into trouble. Use the loop with all operations inside.