How to Run many script python with low memory in EC2? [closed]

Question

Closed. This question needs details or clarity. It is not currently accepting answers.

Want to improve this question? As written, this question is lacking some of the information it needs to be answered. If the author adds details in comments, consider editing them into the question. Once there's sufficient detail to answer, vote to reopen the question.

Closed 11 days ago.

Improve this question

I have many python script that run in schedule (its crawlers). currently i'm using tmux to automate the script running in my ec2 instance. but since the number of scripts are growing, its getting consume much RAM. is there any solution for this kind of problem ? or architecture ?

What kind of scripts are these? Are they CPU intensive or disk IO or network IO bound? What is the size (as measured by Wheeler's sloccount ...) of your scripts? — Basile Starynkevitch
– Basile Starynkevitch, Commented Nov 10 at 9:50

Basile Starynkevitch · Accepted Answer · 2025-11-14 20:01:26Z

1

A possibility could be to use the batch(1) or at(1) (or prehaps crontab(1)) Linux commands (to run your Python scripts in sequence).

Another possibility might be to recode the slow and CPU intensive Python scripts using some practically faster programming language (like Ocaml or Rust or C++ -compiled by GCC- or SBCL).

You could even consider using program generating scripts. Maybe RefPerSys could be relevant.

edited Nov 14 at 20:01

answered Nov 10 at 9:42

Basile Starynkevitch

231k18 gold badges322 silver badges578 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

pambudi Nov 10 at 12:18

do you mean I just need 1 tmux window (currently i used around 20 window to run all of those scripts), and set the scheduller by using linux at instead ? so its reduce the tmux consume of ram memory ?

Basile Starynkevitch Nov 10 at 20:54

Read tutorials and documentations of these commands. Explain in your question why you have to use tmux and what your scripts are doing

pambudi Nov 13 at 12:15

im aiming to run python script regularly e.g (every week, every 3 days) with lowest ram memory usage. but I think for my use case, crontab will be the good choice. thanks for the insight. btw, is there's any technology that has functionality like crontab but its has better performance or more modern ?

Basile Starynkevitch Nov 13 at 17:05

Explain in your question why crontab or at or batch has bad peformance. Explain also what kind of python scripts are you running, and why can't you recode (in some practically faster language like Ocaml [on ocaml.org/] or C++ (with gcc.gnu.org ....) or Rust or Go or Common Lisp with sbcl.org ) those scripts that are CPU intensive. Explain why you need something more modern, and your definition of modern software.

pambudi 2 days ago

1. I haven't test crontab or at or batch before. its just my assumption. that assumption coming from my thought that : a. TMUX case : I execute python script (that has scheduler code) one time. then the script will idle if not in schedule to run. this is takes ram memory b. crontab or cronjob : the kernel execute it it when only in schedule. no idle. so the script totally shutdown 2. I don't say crontab or at or batch has bad performance. CMIIW

Collectives™ on Stack Overflow

How to Run many script python with low memory in EC2? [closed]

1 Answer 1

5 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Related