java xml parsing a large file and db relation

Question

I have a 10+ MB xml file with consists of node(about 10K to 20K) with relations.

<.....>
<Emplyoyee>
    <name>Jack</name>
    <age>35</age>
    <supervisor></supervisor>
    <....>
</Emplyoyee>
<.....>
<.....>
<.....>
<Emplyoyee>
    <name>Smith</name>
    <age>20</age>
    <supervisor>Jack</supervisor>
    <....>
</Emplyoyee>
<.....>

Now, I want to parse this file and store all the details in DB with "Employee" table which has a field(ID) called "supervisorID". Until now I have tried to make a List of all the employees and then iterating the List for finding supervisor relation.

Please suggest me a memory efficient and faster way to do this. What libraries can I use to handle this type of problems.

you mean you have parsed it using, lets say the Documentfactory in java, but you need a more efficient way? — samach
– samach, Commented Sep 19, 2012 at 19:51
What happens when you have two "emplyoyees" [sic] with the same name and both are supervisors? How to you link their subordinates to the correct one? — Jim Garrison
– Jim Garrison, Commented Sep 19, 2012 at 20:02
@JimGarrison The employee names correspond to xxxx in [email protected] so are unique. I've put these names for simplicity. — itsraja
– itsraja, Commented Sep 19, 2012 at 20:19

Jiri Kremser · Accepted Answer · 2012-09-19 21:33:19Z

1

Look at MOXy framework provided by EclipseLink. It uses JAXB implementation behind the scenes in fact. But does also the ORM stuff with JPA.

answered Sep 19, 2012 at 21:33

Jiri Kremser

13k8 gold badges49 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ilya · Accepted Answer · 2012-09-20 08:11:48Z

1

You can convert data from XML file to Java Objects using JAXB and insert Java objects to database using Hibernate + JPA.
You can create 2 DTO
Emplyoyee - with all info about Emplyoyee (name, age, ...)
and
Emplyoyees with List<Emplyoyee> for JAXB unmarshalling

EDIT: WITHOUT JAXB and JPA

You can parse file using javascript and send SQL queries usinf Ajax

var xmlDoc = new ActiveXObject("MSXML.DOMDocument");  
xmlDoc.async = false;
xmlDoc.preserveWhiteSpace = true;
xmlDoc.load(pathToFile);
var nodes = xmlDoc.selectNodes("/Emplyoyee");  
for (var node = nodes.nextNode(); node != null; node = nodes.nextNode())
{
   // get another nodes, create SQL query and sent it to server usinj Ajax  
}

edited Sep 20, 2012 at 8:11

answered Sep 19, 2012 at 19:54

Ilya

29.9k19 gold badges119 silver badges161 bronze badges

3 Comments

feldoh Over a year ago

When using JAXB xjc does most of the work for you. There are many tools that will create an XSD for you given a file sample which will get you most of what you need to use with JAXB (I tend to use the XML tools in intelliJ: jetbrains.com/idea/webhelp/…). JAXB is more than capable of doing this for a 10MB file, I perform a similar task on a 30MB file to get daily supplier updates. Oh and here is a good JAXB primer oracle.com/technetwork/articles/javase/index-140168.html

bdoughan Over a year ago

You can also start from the JPA entities and add JAXB annotations to map those classes to the desired XML format.

itsraja Over a year ago

I could not use Hibernate for some reasons, and I want it to be done with in 5 secs (Parsing + Persistence) and thought of threads to make it faster persistence but the fact that I need to parse all the nodes before searching for relations. Also the number of employees entities may grow to 50K-60K.

Collectives™ on Stack Overflow

java xml parsing a large file and db relation

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related