0

I have to read a large CSV file with about 700,000 records and compare the CSV data to an API response. I was able to use OpenCSV and make the code work. However, the deserialization process is extremely slow. It takes about an hour just to deserialize the data. I have been using the following code to read and deserialize my CSV.

     List<ProjectVO> csvValue = new CsvToBeanBuilder(new FileReader("project.csv"))
       .withType(ProjectVO.class).build().parse();

Is there any other efficient method to replace it?

My PersonVO class looks like this:

.
.
.
@JsonIgnoreProperties(ignoreUnknown = true)
public class ProjectVO {
@JsonProperty("actualCompletionDate")
@CsvBindByName(column = "actualCompletionDate")
private String actualCompletionDate;
.
.
.

I am comparing my CSV data and JSON response something like following:

assertEquals("The value for column 'actualCompletionDate' has the same data in both files for the ID: "
   + jsonValue.getId(), csvValue.getActualCompletionDate(), jsonValue.getActualCompletionDate());
Improve this question
7
  • You should read using a streaming parser than the entire collection (eg List) at once. Commented Dec 24, 2022 at 4:42
  • What's the performance like when you do a "bare bones" experimental version with the absolute bare minimum to parse (ignoring all functional requirements, just to get a baseline benchmark)? Commented Dec 24, 2022 at 4:42
  • 2
    Your first step should be profiling your code to figure out what's taking the time. Commented Dec 24, 2022 at 4:42
  • Re Anand's suggestion, here's an example of what he's talking about stackoverflow.com/questions/39673372/… Commented Dec 24, 2022 at 4:44
  • stackoverflow.com/questions/19486077/…, @YAMM , anwser, have test read text file, run: BufferedReader.readLine() into LinkedList, lines: 1000000, estimatedTime: 0.105118655, maybe you can read file into LinkedList or ArrayList, then use multi thread read it from ArrayList and pass it to CsvToBeanBuilder Commented Dec 24, 2022 at 5:47
Related questions
19
Fast CSV parsing
0
Creating a Java class on the fly from a CSV structure
0
how to work with large csv file
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.