Read from multiple files and write them into file file sequenction using multithread

Question

I want to read from multiple files and write them into single file using multithread. I have done some code which is reading content of files perfectly fine and writing them into another file. But my problem is data is not coming sequential. Like reading f1 and write into output file, then read f2 and write into out file and so on. Have a look on my code

public MultiFileReadingThread(BufferedWriter writer, BufferedReader reader, String name, java.util.List<File> files){
        this.writer = writer;
        this.reader = reader;
        this.name = name;
        this.files = files;
    }

    @Override
    public void run() {
        SimpleDateFormat formatter = new SimpleDateFormat("dd/mm/yyyy hh:mm:ss:ms");
        System.out.println("Start Time : " + name + formatter.format(System.currentTimeMillis()));
//      readFile();
        if(i == 1){

        }
        writeIntoSharedBufferedWriter();
        System.out.println("End Time : " + name +  formatter.format(System.currentTimeMillis()));
    }

public void writeIntoBufferedWriter(){

        for(File f : files){
            try{
            BufferedReader r = new BufferedReader(new FileReader(f));
            String line = null;
            while((line = r.readLine()) != null){
                writer.write(line);
                writer.write("\n");
            }
            } catch(Exception ex){
                ex.printStackTrace();
            }
        }
    }

Code to run the threads. There are five different task and worker for them.Again this code giving me the out but not correct. like f1, f2, f3

BufferedReader r1 = new BufferedReader(new FileReader(new File("/home/naveen/f1")));
        BufferedReader r2 = new BufferedReader(new FileReader(new File("/home/naveen/f2")));
        BufferedReader r3 = new BufferedReader(new FileReader(new File("/home/naveen/f3")));
        BufferedReader r4 = new BufferedReader(new FileReader(new File("/home/naveen/f4")));
        BufferedReader r5 = new BufferedReader(new FileReader(new File("/home/naveen/f5")));

        BufferedWriter writer = new BufferedWriter(new FileWriter(new File("/home/naveen/write")));
        Runnable task1 = new MultiFileReadingThread(null, writer, r1, "f");
        Runnable task2 = new MultiFileReadingThread(null, writer, r2, "s");
        Runnable task3 = new MultiFileReadingThread(null, writer, r3, "t");
        Runnable task4 = new MultiFileReadingThread(null, writer, r4, "fu");
        Runnable task5 = new MultiFileReadingThread(null, writer, r5, "fi");

        Thread wroker1 = new Thread(task1, "w1");
        Thread wroker2 = new Thread(task2, "w2");
        Thread wroker3 = new Thread(task3, "w3");
        Thread wroker4 = new Thread(task4, "w4");
        Thread wroker5 = new Thread(task5, "w5");

Can somebody help to achieve the taks. Thanks in advance.

VGaur · Accepted Answer · 2014-10-13 19:03:53Z

1

The better way is to use producer consumer pattern. The producer threads will read the files and send the content to consumer by wrapping it in some class. Which will write the combined file into the disk.

On consumer side you can use priority queue and let producer set the priority.

answered Oct 13, 2014 at 19:03

VGaur

7152 gold badges9 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

keepmoving Over a year ago

But i want while reading thread1 and all thread should be waiting. once Thread1 has finished then it should send the notification to thread2 and thread1 along with other thread should be in waiting stage. can i use the stack kind of system from where all thread will be in waiting stage. Then one thread will be fetched, processed and send it again in stack. but it will be queued. I just want to know the approach to follow. or if you can share any link for this.

VGaur Over a year ago

Yes you can. I would suggest look into blocking queue and barrier .

Justin · Accepted Answer · 2014-10-13 19:16:24Z

Well, there's a lot of unnecessary code duplication here.

Each instance of MultiFileReadingThread should have it's own reader and writer which is never accessed from outside the thread. It seems like you have multiple threads using the same writer, in this case that's not going to work out very well.

Try this:

public MultiFileReadingThread(String name, List<File> inputFiles, File outputFile) {
    this.writer = new BufferedWriter(new FileWriter(outputFile));
    this.name = name;
    this.files = inputFiles;
}

@Override
public void run() {
    SimpleDateFormat formatter = new SimpleDateFormat("dd/mm/yyyy hh:mm:ss:ms");
    System.out.println("Start Time : " + name + formatter.format(System.currentTimeMillis()));
    //readFile();
    if(i == 1){

    }
    writeIntoSharedBufferedWriter();
    System.out.println("End Time : " + name +  formatter.format(System.currentTimeMillis()));
}

public void writeIntoBufferedWriter(){
    for(File f : files){
        try{
        BufferedReader r = new BufferedReader(new FileReader(f));
        String line = null;
        while((line = r.readLine()) != null){
            writer.write(line);
            writer.write("\n");
        }
        } catch(Exception ex){
            ex.printStackTrace();
        }
    }
}

Don't forget that you'll also need to close your BufferedReader and BufferedWriter.

Threading can be difficult and unfortunately there's no single tutorial or resource to provide all the information you need. I'd suggest reading every one you can find, multiple times. Trying to implement multithreading before you have a strong grasp on the subject can lead to lots of problems. Sometimes threading problems don't become apparent until late in an applications life cycle when major changes are required to correct them.

Collectives™ on Stack Overflow

Read from multiple files and write them into file file sequenction using multithread

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related