How to process multiple files with multi-threads?

Question

What I meant was

If there are 100 files, 100 threads will be created to read the file and process the contents of the file,

When there are 100 files, one thread is turned every second, and no threads are generated for the remaining 99.

One thread for one file is running, and 99 are off.

If you have 100 files, you have to create 100 multi-threads.

If there are 100 files, what are the ways to process them simultaneously?

        public static void CollectData(CancellationTokenSource cts)
        {
            Task.Run(async () =>
            {
                while (true)
                {
                    try
                    {
                        string shareIp = IniFileHandler.GetPrivateProfile("MOVE_PATH", "PATH", string.Empty, iniFileName);

                        if (shareIp == null || shareIp == "")
                        {
                            log.Error(iniFileName + " 의 폴더명을 기입하시기 바랍니다.");
                            break;
                        }
        
                        FileRead read = new FileRead();
                        await read.getDataTableFromFile(shareIp);                     
        
                    }
                    catch (Exception ex)
                    {
                        log.Error(ex.Message);
                    }
                    finally
                    {
                        await Task.Delay(1000); 
                    }
        
                }
            });
        }

        public async Task getDataTableFromFile(string path)
        {
            try
            {
                string successPath = IniFileHandler.GetPrivateProfile("SUCCESS_PATH", "PATH", string.Empty, iniFileName);
                string failPath = IniFileHandler.GetPrivateProfile("FAIL_PATH", "PATH", string.Empty, iniFileName);
                // string originalPath = IniFileHandler.GetPrivateProfile("SUCCESS_PATH", "PATH", string.Empty, iniFileName);
                DirectoryInfo directoryInfo = new DirectoryInfo(path);
                string fileName = string.Empty;
                string withoutFileName = string.Empty;
                DataTable dt = new DataTable();

                FileInfo[] files = directoryInfo.GetFiles();

                if (directoryInfo.GetFiles().Length > 0)
                {
                    // Console.WriteLine(directoryInfo.GetFiles());
                    foreach (FileInfo file in files)
                    {
                        fileName = file.FullName;
                        withoutFileName = files[0].Name;

                        string fileExtension = Path.GetExtension(fileName);
                        string strLine = string.Empty;
                        string[] strAry = new string[] { };

                        FileStream fs = new FileStream(fileName, FileMode.Open, System.IO.FileAccess.Read, FileShare.Read);

                        try
                        {
                            using (StreamReader reader = new StreamReader(fs, UTF8Encoding.UTF8, true))
                            {
                                if (reader == null)
                                {

                                }

                                if (fileExtension == ".txt")
                                {
                                    strLine = await reader.ReadLineAsync();
                                    strAry = strLine.Split('\t');
                                }
                                else if (fileExtension == ".csv")
                                {
                                    strLine = await reader.ReadLineAsync();
                                    strAry = strLine.Split(',');
                                }
                                else if (fileExtension == ".Log")
                                {
                                    strLine = await reader.ReadLineAsync();
                                    strAry = strLine.Split('\t');
                                    //Console.WriteLine(strAry);
                                }

                                foreach (string str in strAry)
                                {
                                    dt.Columns.Add(str);
                                }
                                while (reader.Peek() >= 0)
                                {
                                    //Console.WriteLine(sr.Peek());
                                    if (fileExtension == ".txt")
                                    {
                                        strLine = await reader.ReadLineAsync();
                                        strAry = strLine.Split('\t');
                                    }
                                    else if (fileExtension == ".csv")
                                    {
                                        strLine = await reader.ReadLineAsync();
                                        strAry = strLine.Split(',');
                                    }
                                    else if (fileExtension == ".Log")
                                    {
                                        strLine = await reader.ReadLineAsync();
                                        strAry = strLine.Split('\t');
                                        //Console.WriteLine(strAry);
                                    }
                                    dt.Rows.Add(strAry);
                                }

                                if (reader.Peek() < 0)
                                {

                                }

                                if (reader.EndOfStream == true)
                                {
                                    reader.Close();

                                    if (InsertDb(dt, fileName, path, successPath, fileExtension) == true)
                                    {
                                        Console.WriteLine(fileName + " 이 " + successPath + " 으로 이동합니다.");
                                        log.Info(fileName + " 이 " + successPath + " 으로 이동합니다.");
                                        moveToSuccess(fileName);
                                    }
                                    else
                                    {
                                        Console.WriteLine(fileName + " 이 " + failPath + " 으로 이동합니다.");
                                        log.Info(fileName + " 이 " + successPath + " 으로 이동합니다.");
                                        moveToFail(fileName);
                                    }

                                }
                            }
                        }
                        catch (Exception ex)
                        {
                            //if (fileName == "") 
                            //{
                            //    Console.WriteLine("폴더에 파일이 없습니다 , 파일을 기입해주세요.");
                            //}2

                        }
                    }
                    //fileName = files[0].FullName;
                }
                
                if (fileName == "")
                {
                    Console.WriteLine("폴더에 파일이 없습니다 , 파일을 기입하세요.");
                    return; 
                }

                //await Task.Delay(1000);
            }
            catch (Exception ex)
            {
                log.Error(ex.Message);
            }
        }

tasks and threads are not the same thing; but: what is the question here? I can't see where you are dealing with 100 of anything - do you mean the foreach (FileInfo file in files) ? — Marc Gravell
– Marc Gravell, Commented Jan 23, 2024 at 9:07
that's still not a question; but - perhaps await Parallel.ForEach(files, file => {...}); ? — Marc Gravell
– Marc Gravell, Commented Jan 23, 2024 at 9:12
Note that file access tend to prefer large sequential read/writes. Processing files in parallel may, depending on your specific context, reduce overall performance due to random IO. This is most significant with HDDs, but may also affect SSDs to a lesser extent. — JonasH
– JonasH, Commented Jan 23, 2024 at 9:25
Also note that async methods may not necessarily be faster than the synchronous variant. And if you are concerned about performance you should start by measuring. Otherwise you have little idea if a change made an improvement. — JonasH
– JonasH, Commented Jan 23, 2024 at 9:32

Lajos Arpad · Accepted Answer · 2024-01-24 09:56:34Z

This is a trial & error domain. You cannot determine whether doing this operation on n files simultaneously is feasible or not without trying. So you should try. Measure server load and performance and try to do something on the server in the meantime to see whether the state into which you draw the server is acceptable.

If so, then all good, because then it's a paper tiger after all.

But if not, then try reducing the number of threads you are running at the same time gradually until you reach an acceptable state.

Let's presume that your server can handle properly 10 such threads at a time.

This would mean that you will always have 10 threads running at a time (except for the end of the process, of course) and, if there are further threads, they are enqueued, so whenever a thread is executed, the next one in the queue will start.

A well-known way to do this is to repeatedly sleep the main thread and between the sleeps check for the number of running threads that completed in the meantime and start that number of threads from the queue.

However, a better way to handle this is to use WaitHandle (see https://learn.microsoft.com/en-us/dotnet/api/system.threading.waithandle.waitone?view=net-8.0) combined with WaitOne, so at the next thread's finish the main thread will resume its operation and start the next thread and then WaitOne again, until the queue is empty.

Collectives™ on Stack Overflow

How to process multiple files with multi-threads?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related