2

I am working on a project where I read data from a file and I need to manipulate some of this data.

The data is binary with some ASCII encoded text in it. The data is also saved Big Endian if that matters.

What I would like to accomplish is to find a pattern with in this data and manipulate parts of that pattern.

Example: (09 49 6E 76 65 6E 74 6F 72 79 0A 00 00 00 02 01 00)

This represents the number 9 for how many letters are in Inventory, followed by ASCII "Inventory" without quotes. "0A" Marks the end of that ASCII text, followed with 00 00 00 02 to mark the size of our inventory "2". "01 00" Marks the end of The entire inventory region.

Example2: (04 53 6C 6F 74 00 02 00)

This represents the number 4 for how many letters are in Slot, followed by ASCII "Slot" without quotes. "00" is a space between ASCII text and Slot number "02", followed by end of region "00".

I need to locate these patterns and several others within the file. I then need to modify parts of the pattern and write to disk.

ExampleModify: (04 53 6C 6F 74 00 02 00) "From above" to (04 53 6C 6F 74 00 07 00) changing "slot number '02' to '07'.

Another caveat is that while I have several patterns that I need to search in the file of varying size, length, and data contained, is that there can be multiple sections of these patterns containing different data that needs to be modified separately as a whole.

To clarify: (Inventory, Slot, id, Count) - is to be treated as information for one person.

There can be multiple copies of (Inventory, Slot, id, Count) for each person that is recorded.

I would like to display this information to the user, and give them the option to modify each of the elements in the group.

I am not a good programmer and want to learn, if you have examples I appreciate that, if you have advice please give it. If you can dumb it down that is even better, thanks. I have a work-in-progress going now, but I am stuck now. If you want to see what I have please let me know.

Summary of what I have: Read file into byte[] then display the whole array to console. That's about it. with a little bit of formatting and some debug info for locating that chunk I read into the array.

Here is a link to my code on pastebin. LINK

I realize that I do not get all occurrences of (Inventory, Slot, id, Count) I need to fix that as well.

EDIT: Example (09 49 6E 76 65 6E 74 6F 72 79 0A 00 00 00 02 01 00) this is a fixed length chunk of binary data within the file I'm reading. As stated 09 represents the length of the string. what follows the string is (0A 00 00 00 02 01 00) the important part of that is (02) this is because that is the only byte(s) that change in that slice of binary data. "02" represents "2" meaning there are 2 instances of (Slot, id, Count) for that particular persons record.

(File)
    (09 Inventory 0A 00 00 00 02 01 00) // Start of person 1's record with 2 instances.
        (Slot)
        (id)
        (Count)
        (Slot)
        (id)
        (Count)
    (Rotation)  // End of person 1's record

    (09 Inventory 0A 00 00 00 04 01 00) // Start of person 2's record with 4 instances.
        (Slot)
        (id)
        (Count)
        (Slot)
        (id)
        (Count)
        (Slot)
        (id)
        (Count)
        (Slot)
        (id)
        (Count)
    (Rotation)  // End of person 2's record
(File End)

The idea is that I want to edit the "id" or increase the number of slots by adding to the inventory and add in more instances of (Slot, id, Count).

"id" - contains item id

"Slot" - contains inventory slot number

"Count" - contains how many of item is in that slot. EDIT Note: Please let me know if I am not being clear, thank you again.

Improve this question
1
  • 1
    I'm not really sure what to try. I will however post my code that I have, in hopes that someone can point out how foolish it is for me not knowing any better. I will add a link to the code in an update to the post. Commented Jul 7, 2012 at 21:23

2 Answers 2

Reset to default
3

You said you're not a good programmer. I'll make some basic suggestions then.

First, for any problem, break it down. Try to break it up into simpler sub-problems that you can solve, or think will be easier to solve. You said you wanted to check for patterns in a file, and then maybe modify some of the data and write it back out. As a first step, I'd say these are the major sub-problems you need to solve in order to get to a complete solution.

  • read the file
  • search for patterns in the binary data
  • make the modifications
  • write the modified data out.

You focused mostly on "search for patterns" part, so I will focus further comments on that. All those other pieces can also be tricky, but you can either solve them yourself or look for hints on stackoverflow or in other places for how other people solved them.

ok, now, regarding "search for patterns" . In my reading of your description,

It seems to me the first byte in your byte array is the text length. In C# code, a data array that exhibits your pattern could be declared explicitly like this: 

byte[] data = new byte[] { 9, 0x49, 0x6E, 0x76, 0x65, 0x6E, 0x74,
               0x6F, 0x72, 0x79, 0x0A, 0, 0, 0, 2, 1, 0 } ;

of course, you won't be declaring the array explicitly. You'll be creating or filling array via the file-read operation. The file-read is still an open problem at this point, but that's ok - you can solve it separately. And, the effect is the same - when you have successfully read the file, you will have a byte array that looks like the array that is explicitly declared above.

ok, now how do you manipulate that data? The first byte is the length of string data. The next N bytes are the string data. Then you have some other stuff.

you didn't describe exactly what "modifications" you want to make, but with the information you have provided, it should be easy to walk through the data.

If you want to get a "slice" of the data in an array, you can do this . 

int length = (int) data[0];
byte[] s = new byte[length];
Array.Copy(data, 0, s, 0, length);
// now, s contains N bytes, representing the string

The next thing you might want to do is get an actual string out of those bytes you've sliced out. To do that, use the Encoding class:

String word = System.Text.Encoding.UTF8.GetString(s);

I don't understand the pattern for the data that follows the string data. In some cases it's a 00, in some cases it's 0A.

But maybe you know the pattern. Follow along the same way and you can walk through all the data for one "record". When you get to the end of the record, then start again and process the next one. The key to getting to the next "Record" is knowing how large a slice of data to take out of the data you have read from the file.

This brings us back to the "reading the file" sub-problem. One way to get the right amount of data from the file is to read only the first byte, then read N bytes further in the file. (See System.IO.Stream.Read).

byte[] size = new byte[1];
var fileStream = File.OpenRead(path);
int offset = 0;
int n;
n = fileStream.Read(size, offset, 1);
// size[0] now contains the byte of data indicating the number of bytes to follow

At this point you can read the next N bytes: 

int length = (int)size[0];
byte[] stringData = new byte[length];
n = fileStream.Read(stringData, offset, length);

Now you can get the string. 

String word = System.Text.Encoding.UTF8.GetString(stringData);

...and so on. Read all the trailing bytes. At that point the cursor that is implicitly held by the fileStream points to the next record in the file.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

I appreciate your reply. Thank you for being positive in your response. I will update the bottom of my post with some hopeful clarification.
I won't do a full code review of what you have but I will make some additional comments. It looks like you are reading the data. You are reading chunkSize and chunkEnd, and so on. You're further along than I thought you would be, based on the original question. You have a bunch of code there, but you haven't identified a specific problem you are experiencing. What's the roadblock?
Im a beginner at programming, and what I have so far is just from me wanting to do this project and reading up on things yesterday to get where I am at now. But the logic of how to go about finding my solution to being able to "slice up" and manipulate that data in a meaningful way is beyond my knowledge level. I am in farther than my knowledge level and probably should learn more basic things first, but they don't keep my attention, so that is why I'm working on this project. Does that makes sense?
ya, it makes sense and I think the way you're going about it is a good way to learn. I still think breaking up the problem is a good way to do things. you seem to be on your way. If there's a roadblock, more than likely you will be able to get help here. If it's just a general question like "I want to read this file then modify it, can you help me?" then... you will get less concrete help.
0

I agree with what @Cheeso said above and that's a good start. I'm going to describe an alternative solution:

  1. Define opcodes for all known sections (one starts with 0x4 0x56, one with 0xA 0x2, etc.).
  2. Read the file in memory.
  3. Apply a matching algorithm on the byte array in memory and extract the start positions of each matched pattern. For performance reasons I recommend using one of the well known searching algorithms (or their variations), such as Knuth-Morris-Pratt or Boyer-Moore.
  4. At this moment you should cache in the application all start positions for all patterns you've found. You can use for this a dictionary of lists (where the key is the pattern type), but you can contain this to a custom class (with pattern id and pattern offsets within the file) and then just use a list and access everything with LINQ.
  5. Access and decode the actual patterns when needed, so not immediately after you read the file. I imagine that you have different uses for them and not all come up right after someone loads a file.

So this gives you already a list of all pattern occurrences within the file. You can use it to display some kind of list to the user.

I had a quick look at your code. It's really big, all stuffed in a few big functions. You need to structure it a bit. One class for each task needed to load and parse the file and then a lot of smaller functions.

When you work with a pattern in the app, for example when you make it available to the user to modify it, it should be fairly easy with this mechanism:

  • Read the data at the offset you already have.
  • You know the offset and you also know the length. When the time comes to write back a modification you can position yourself in the loaded stream at that specific position and overwrite.

Having to work constantly with data in the file for various modifications I would say that it's better to hold it in memory and just save it when needed, still keeping your modified version in memory. Maybe sometimes pattern offsets need to be updated in your in-memory cache, so you can apply the initial parsing process only to a subsection of the file (for speeding up the process).

If you need the pattern data as well immediately after you load a file then you can apply a pattern decoding process when you traverse the file the first time (when you extract the offsets). I suggest keeping only what's important in a separate collection, for displaying to the users (like only IDs and whatever texts you need to display). Load the entire entry when you actually need it.

Improve this answer

3 Comments

I understood most of what you said and appreciate your advice and will look into both your answer and @Cheeso. If it helps to add more clarification. The total size of the file I'm reading in is less than 25Mb. Out of that I use no more than 3.5Kb for my array. I'm not sure what lINQ is and will try to read up some. Also as far as restructuring the code, I agree. I just don't have the knowledge yet on how to do that. I have code in there I don't really understand. I just got it to do what I needed it to do for that part of the process. Thank you again for your response.
Finding start positions is probably the the easiest way to accomplish what needs to be done, in my mind.
@DrewBirmingham: Yes, you can do that with any pattern matching algorithm. The good part is that you don't have to decode the pattern at that time, just store it's offset. So file loading will be faster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.