0

I have a flat data file in the form of xml, but there isn't a real Windows viewer for the file, currently. I decided to create a simple application with Node-WebKit, just for basic viewing - the data file won't need to be written to by the application.

My problem is, I don't know the proper way to read a large file. The data file is a backup of phone SMS's and MMS's, and the MMS entries contain Base64 image strings where applicable - so, the file gets pretty big, with large amounts of images (generallly, around 250mb). I didn't create/format the original data in the file, so I can't modify it's structure.

So, the question is - assuming I already have a way to parse the XML into JavaScript objects, should I, a) Parse the entire file when the application is first run, storing an array of objects in memory for the duration of the applications lifetime, or b) Read through the entire file each time I want to extract a conversation (all of the messages with a specific outgoing or incoming number), and only store that data in memory, or c) Employ some alternate, more efficient, solution that I don't know about yet.

Improve this question
22
  • Is this happening in the front end or server side? Commented Aug 30, 2014 at 0:58
  • Can you create a server side script (like perl) to break the file into chunks for your front end? Commented Aug 30, 2014 at 0:58
  • Sorry, I should've been more clear - the entire application is on the client-side, via node-webkit - so, data transfer from a server isn't really applicable - the file will be opened from the local filesystem by the user. I'm more so wondering if so much data stored in the memory would lead to bad preformance, or if the process of reading through a file that large so many times (each time I wanted to extract a conversation) would be wasteful. Commented Aug 30, 2014 at 1:03
  • 1
    @vol7ron no, node-webkit is a bare bones "browser" featuring webkit and node.js in one process. It is meant as a "shell" to deliver HTML5-NodeJS apps as stand alone applications. Commented Aug 30, 2014 at 1:09
  • 1
    Comments are not for extended discussion; this conversation has been moved to chat. Commented Aug 30, 2014 at 16:28

1 Answer 1

Reset to default
1

Convert your XML data into an SQLite db. SQLite is NOT memory based by default. Query the db when you need the data, problem solved :)

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Not memory based? Meaning the data won't be persistently stored, and should be released at the application's close?
Means that it doesn't store the whole dataset on the memory, which is what you said you wanted to avoid.
Ok, makes sense - what would be the best place to store the database?
Does not matter. What matters is that it is somewhere on your client 's computer.
Ok, I think I'll try a temporary sqlite database then - all the benefits of SQL along with temporary, easily-accessible storage.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.