1

I want to process array of length around 100 000, without putting too much load on CPU. I researched about streams and stumbled upon highlandjs, but i am unable to make it work.

I also tried using promises and processing in chunks but still it is putting very much load on CPU, program can be slow if needed but should not put load on CPU

Improve this question
3
  • What is the reason for low load on the CPU? Server responsiveness to other requests? Something related to your hosting plan? What? That would help us know what things to suggest. Also, please show code for what the processing is that you are doing? As this question stands now, there's no way to answer it. It takes a certain amount of CPU cycles to do your processing on the array. Nothing you do (other than rewriting the actual code that processes the array to be more efficient) changes that. Commented Jul 6, 2019 at 21:30
  • As written, this is an XY problem where you're describing your attempted method of solving a problem and not describing the underlying actual problem. We can help you better if you describe the actual underlying problem. Commented Jul 6, 2019 at 21:43
  • Upon high load on CPU, server stops responding to other requests and sometimes crashes, so i want to prevent that and about implementation I haven't posted because that contains basic array processing like finding element from other array, using it, etc Commented Jul 7, 2019 at 3:45

1 Answer 1

Reset to default
3

With node.js which runs your Javascript as single threaded, if you want your server to be maximally responsive to incoming requests, then you need to remove any CPU intensive code from the main http server process. This means doing CPU intensive work in some other process.

There are a bunch of different approaches to doing this:

  1. Use the child_process module to launch another nodejs app that is purposeful built for doing your CPU intensive work.
  2. Cluster your app so that you have N different processes that can do both CPU intensive work and handle requests.
  3. Create a work queue and a number of worker processes that will handle the CPU intensive work.
  4. Use the newer Worker Threads to move CPU intensive work to separate node.js threads (requires node v12+ for stable, non-experimental version of threads).

If you don't do this CPU intensive work very often, then #1 is probably simplest.

If you need scale for other reasons (like handling lots of incoming requests) and you don't do the CPU intensive stuff very often #2.

If you do the CPU intensive stuff pretty regularly and you want incoming request processing to always have the highest priority and you're willing to allow the CPU intensive stuff to take longer, then #3 (work queue) or #4 (threads) is probably the best and you can tune the number of workers to optimize your result.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.