Extracting date and time from a binary data string using

Question

I have a third party application that stores time series data as a packed binary file. I am trying to create a tool to convert the data stored in this file.

Shown below is a snapshot of this data.

41 16 00 00 01 00 D7 11 00 00 01 00 E8 55 A6 20 08 1E D0 08 00 00 00 60 59 D5 
86 40 03 E8 F5 2C 22 08 1E D0 08 00 00 00 00 C0 0B 87 40 01 E8 95 B3 23 08 1E 
D0 08 00 00 00 40 1E 00 87 40 01 E8 35 3A 25 08 1E D0 08 00 00 00 60 13 F8 86
40 01 E8 D5 C0 26 08 1E D0 08 00 00 00 40 65 09 87 40 01 E8 75 47 28 08 1E D0 
08 00 00 00 20 8A F6 86 40 01 E8 15 CE 29 08

I know that this block of data corresponds to the following values.

5/13/2013 17:46:11.558  730.6686401
5/13/2013 17:46:14.118  737.46875
5/13/2013 17:46:16.678  736.0147705

I can extract the values: they are of type double. For instance, the 8 bytes 00 00 00 60 59 D5 86 40 correspond to 730.6686401.

But I am stumped by how to extract the datetime format. I know that it is buried in this string somewhere. How can I figure out what format the time is in?

I have been using Python's struct module for the type conversion.

Anybody have any ideas?

Judging by the other 2 float values, encoded as doubles, there are only 9 bytes between these to encode the date-time value. — Martijn Pieters
– Martijn Pieters, Commented Jun 20, 2014 at 21:01
Also, the sequence [08 1E D0 08] repeats 6 times, always 13 bytes apart. Are there only 3 values in this block, or are there more than 3? — aruisdante
– aruisdante, Commented Jun 20, 2014 at 21:02
The sequence [E8 X5] also repeats (7 in this case, but looks like this block may be truncated and there would have been another 08 1E sequence), where X is a varying number, 2 bytes before the 08 1E squence — aruisdante
– aruisdante, Commented Jun 20, 2014 at 21:17

Armin Rigo · Accepted Answer · 2014-06-21 18:39:56Z

4

If you take the 8 bytes immediately before one of the double, and consider it as an integer (low-endian, like the double), then you get the following numbers:

635040567715583464
635040567741183464
635040567766783464

If you divide these numbers by 10**7, then you get the date as a number of seconds (and fractional seconds). At least it corresponds in the minutes, seconds, and fractions of seconds. For the hours I get an off-by-two error (timezone?). For the complete date, the day number 735000(*) corresponds to the 5/13/2013 as follows. It's the number of days from the year 1:

>>> datetime.date(1,1,1) + datetime.timedelta(735000)
datetime.date(2013, 5, 13)

(*) that's any of these numbers divided by 10**7 * 60 * 60 * 24

Or in a single step:

>>> x = 635040567715583464 / 10.**7 / 86400
>>> datetime.datetime(1,1,1) + datetime.timedelta(x)
datetime.datetime(2013, 5, 13, 15, 46, 11, 558353)

edited Jun 21, 2014 at 18:39

answered Jun 21, 2014 at 8:37

Armin Rigo

13.1k41 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Martijn Pieters Over a year ago

datetime supports ordinals (datetime.datetime.fromordinal(735000)). So the numbers are offsets from a 1/1/1 00:00:00 epoch instead of the UNIX epoch?

Martijn Pieters Over a year ago

so ordinal, rem = divmod(635040567715583464, 10**7 * 86400), then datetime.datetime.fromordinal(ordinal) + datetime.timedelta(microseconds=rem/10.0) gives you the correct date datetime.datetime(2013, 5, 12, 15, 46, 11, 558346), in UTC presumably.

Armin Rigo Over a year ago

I'm unsure what exactly fromordinal(735000) does, but the result is one day off (the 12th instead of the 13th). Of course the final result is still 2 hours off, so maybe we should really use fromordinal() and consider the result to be 22 or 26 hours off instead.

Martijn Pieters Over a year ago

fromordinal() interprets the integer as days since 1/1/0001. If there is an off-by-one in there that should be easily corrected.

d0m1n0 Over a year ago

I dug into the 2 hour difference. Out could be due to the fact that the data was collected during European Summer Time. @Armin Rigo, does the time conversion give us the time in UTC?

|

Collectives™ on Stack Overflow

Extracting date and time from a binary data string using

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related