Break for event loop processing without breaking loop

Question

I found myself trying to process a vast amount of data recently and I had to do some (lets call it) creative coding to get the desired result.

I knew the processing would be a multi day task so I wrote it in a way that could be interrupted and resumed. But what stumped me was finding a nice way to process SIGINT/SIGTERM style events without wanting to break my loop.

btw these are obviously not the real names for my functions

let closed = false;
function* syncGenerator() {
  yield* ~20-trillion-calculated-results;
}
function main() {
  try {
    for ( const value of syncGenerator() ) {
      syncWork(value);
      if (closed) {
        syncGracefulCleanup();
        break;
      }
    }
  } catch ( err ) {
    handleError(err);
  } finally {
    syncGracefulCleanup2();
  }
}
process.on('SIGINT', () => closed = true);

I had failed to realise that SIGINT would never be processed while I was still in the for-loop and so would only ever finish after the entire dataset was processed (a very foolish oversight, I think I was deluded to believing it would work the same was as Arduino Hardware Interrupts. D'oh. Lesson learned.

Now that I realised this, my solution was to put half of my code at the end of the tick using the microtask async-await trick.

let closed = false;
function* syncGenerator() {
  yield* ~20-trillion-calculated-results;
}
async function main() {
  try {
    for ( const value of syncGenerator() ) {
      syncWork(value);
      await new Promise(r => setImmediate(r)); // the new nasty
      if (closed) {
        syncGracefulCleanup();
        break;
      }
    }
  } catch ( err ) {
    handleError(err);
  } finally {
    syncGracefulCleanup2();
  }
}
process.on('SIGINT', () => closed = true);

Now, this works as I expected and the await allows the loop to be paused and allow SIGINT/SIGTERM to be processed and then picks up if closed is set to true. YAY.. But DAMN it looks nasty. I was hoping someone might have a better looking solution than this?

Can your syncGenerator possibly be an asynchronous generator instead? — noseratio
– noseratio, Commented Sep 3, 2019 at 22:08
@noseratio only by moving the await setImmediate code within that function. Same problem, different location. — WORMSS
– WORMSS, Commented Sep 3, 2019 at 22:23
What I mean is, can you actually yield "~20-trillion-calculated-results" asynchronously in a natural way, as they become available? And use for await loop to consume it? — noseratio
– noseratio, Commented Sep 3, 2019 at 22:49
@noseratio I just wrote a quick test for that scenario of just writing async infront of the generator and for-await-of . but sadly it works the way I thought it would. Without putting an actual asynconous task within the async generator it works exactly like a syncGenerator, so I would have to move await new Promise((r) => setImmidate(r)); into there. As I said, same problem, just different location. — WORMSS
– WORMSS, Commented Sep 3, 2019 at 23:19

Holli · Accepted Answer · 2019-09-03 22:39:01Z

1

You will need to keep the flag, but you can make your main logic simpler. Consider

let closed = false;
function* syncGenerator() {
  yield* ~20-trillion-calculated-results;
}

function main( generator = syncGenerator() ) 
{
    if (closed) 
    {
      syncGracefulCleanup();
    }
    else 
    {
        // assuming syncWork returns false when 
        // there are no more value
        if ( syncWork( generator.next() ) )
        {
            // next-ish off to the event loop. this is NOT recursive, 
            // even though it may look a bit like it
            setTimeout( () => main(generator), 0 );         
        }
    }
}

process.on('SIGINT', () => close = true);

edited Sep 3, 2019 at 22:39

answered Sep 3, 2019 at 22:15

Holli

5,08213 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

WORMSS Over a year ago

I don't believe that is how generators work. That would give me the first result of the generator each time. I would then need to store a second global variable to hold the iterator and call it.next(); I didn't consider this 'nicer'

Holli Over a year ago

Oh you doubting Thomasses :-) You are right of course. But also not, cause my update is still pretty, imho.

Holli Over a year ago

And no additional global ^^

WORMSS Over a year ago

You seem to also forget that you would need to test for generator.next().done, otherwise when the generator has ran out of values, the main code would be looping forever. At this point the code starts to look even more spaghetti than the original, and not taking into account that my code above is a very very cutdown version which I hadn't even mentioned the try/catch/finally that wraps the whole for-loop

Holli Over a year ago

Right, i'm not so familiar with generators. Can you make syncWork return true / false to indicate when the generators is done?

|

Collectives™ on Stack Overflow

Break for event loop processing without breaking loop

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related