optimize this java code?

Question

I have a quartz scheduler job which scans a directory every 10 seconds and convert all pcl files to pdf files.

Here I put part of the code which scans the directory and loads all the files and for each of it it calls some methods to convert it.

    FileExtensionFilter extFilter = new FileExtensionFilter();
    extFilter.setAllowAll(false);
    extFilter.addExtension(_fExtension);
    // Filter out all files that match.
    File[] filteredFileList = fInbox.listFiles(extFilter);
    for (File fSrc : filteredFileList) {
       try {
          //call the methods to convert fSrc file
       }catch (Exception e){
         //threat exception's code
       }

All works nice if I have 30 pcl files but if I have 50 for instance, the processing goes step by step slower.

The question is: is there any way to improve this loop? Basically to not "wait" for the file to be processed but go further and take another one and so one?

I'm thinking about multithreading but I am not sure if this will work on this filesystem scan conversion task... Can you suggest something?

Thanks.

Ps: Please note that If I throw 30 files, and after that another 30 all works great. The performance is affected when I throw at once more files...

corsiKa · Accepted Answer · 2011-04-18 19:40:52Z

3

Throttle it so it only picks 20 files to do each 10 seconds. If you throw 50 files at it, it will do 20, and remove those, then do 20 more, then remove those, then do 10 more and remove those.

It may take a little longer (some downtime) but you won't screw up your scheduler.

answered Apr 18, 2011 at 19:40

corsiKa

82.8k26 gold badges162 silver badges210 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Cristian Boariu Over a year ago

Well..that's a nice approach!

corsiKa Over a year ago

You could also keep track of your time. In the loop, stop after 9500 milliseconds, and let the next "run" take care of it.

Byron Whitlock Over a year ago

+1. Also filesystem conversion tasks usually don't benenifit from multithreading. This is because the bottleneck isn't cpu, it is IO.

darioo · Accepted Answer · 2011-04-18 19:52:39Z

0

I'd suggest taking a look at jNotify. It will allow you to get notified at the exact time when a file is appears/is modified/deleted in a directory. This way, you can start processing when the file appears, and this could be better than processing all files at once.

This will be a part of JDK core in Java 7. This article will tell you more about it. The java.nio.file package has a WatchService API to support this.

Although, as others have said, your problem is more I/O bound than CPU bound. If you're doing a lot of conversions, a faster hard drive, possibly a SSD, will be of greater help.

answered Apr 18, 2011 at 19:52

darioo

47.4k10 gold badges79 silver badges104 bronze badges

Collectives™ on Stack Overflow

optimize this java code?

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related