9

I want to store a list of datetimes in a binary file in Python.

EDIT: by "binary" I mean the best digital representation for each datatype. The application for this is to save GPS trackpoints composed by (unix-timestamp, latitude, longitude, elevation), so the whole structure is little-endian "Long, float, float, float", with four bytes to each value.

NOTE: I don't use "unix-timestamp" due to any affection to the Unix platform, but only as an unequivocal way to represent the value of a datetime.

Currently, I am doing like the code below, but besides some timezone confusion that I'm still working out (my timezone is -3), I believe converting to int and back might not be the right way, since datetime and datetime64 are native types in python/numpy, if I'm not mistaken. Thus, a datetime64 would need eight bytes instead of the four I am using for the (long)unix-timestamp.

import datetime
import calendar
import struct

now = datetime.datetime.now()
print now

stamp = calendar.timegm(now.utctimetuple())
print stamp

binarydatetime = struct.pack('<L', stamp)
recoverstamp = struct.unpack('<L', binarydatetime)[0]
print recoverstamp

recovernow = datetime.datetime.fromtimestamp(recoverstamp)
print recovernow

So the main question is: "is this the pythonic way to converting naive datetime to binary and back?"

And the aditional question is: "if everything in this code is supposed to be naive, why do I have a timezone offset?"

Thanks for reading!

Improve this question
5
  • Define 'binary' here. What other software do you expect to interoperate with? There is no standard, per se, of storing datetime in binary formats, most software pick their own preferred format, which may or may not be based on the UNIX epoch. Commented Aug 29, 2013 at 16:56
  • @MartijnPieters I would like the datatypes to be written to file in a language and platform independent way. See my edit! Commented Aug 29, 2013 at 19:42
  • 1
    consider sqlite database as a portable binary format to store gps trackpoints using its native datetime type for utc (or gps) time. Commented Sep 2, 2013 at 7:24
  • Did you solve this problem? ("reading" timestamp from bytes). I have the same. Commented Dec 31, 2020 at 11:17
  • @garrythehotdog see my own answer below Commented Jan 5, 2021 at 16:45

3 Answers 3

Reset to default
6

I have found a way using the Unix timestamp and storing it as an integer. This works for me because I don't need a subsecond resolution, but I think long integers would allow for microsecond resolution with some modifications of the code.

The changes from my original consist in replacing calendar.timegm by time.mktime and also utctimetuple by timetuple, to keep everything naive.

This:

import datetime
import struct
import time

now = datetime.datetime.now()
print now

stamp = time.mktime(now.timetuple())
print stamp

recoverstamp = datetime.datetime.fromtimestamp(stamp)
print recoverstamp

binarydatetime = struct.pack('<L', stamp)
recoverbinstamp = struct.unpack('<L', binarydatetime)[0]
print recoverbinstamp

recovernow = datetime.datetime.fromtimestamp(recoverbinstamp)
print recovernow

Returns this:

2013-09-02 11:06:28.064000
1378130788.0
2013-09-02 11:06:28
1378130788
2013-09-02 11:06:28

From this, I can easily write the packed binarydatetime to file, and read it back later.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

For me microseconds are important what do you mean by using long integers ?
@le_lemon the unix timestamp is too large to be stored on a 32 bit signed integer, so in order to properly store it with microsecond resolution you would need a larger number format, such as uint, long, or even better ulong (unsigned 64 bit integer). Please check all this, perhaps what I just said is not totally correct, but these are the principles.
@heltonbiker looks like stamp needs to be converted to integer before packing: binarydatetime = struct.pack('<L', int(stamp))
2

By far the most simple solution is to use Temporenc: http://temporenc.readthedocs.org/

It takes care of all the encoding/decoding and allows you to write a Python datetime object to a file:

now = datetime.datetime.now()
temporenc.pack(fp, now)

To read it back, this suffices:

dt = temporenc.unpack(fp)
print(dt)
Improve this answer

Comments

0

If you want to save an object to a binary file, you may think about pickle

As far as I know, it writes a byte stream, so it's binary enough, but the result is

Python-specific

Also, I think it should work with datetime.

And you'll have this for saving:

with open('your_file', 'wb') as file:
    pickle.Pickler(file).dump(your_var)

And this for recovering

with open('your_file', 'rb') as file:
     recovered=pickle.Unpickler(file).load()
Improve this answer

4 Comments

Actually I want a language- and platform- independent way to store the datetime data type. See my edit.
Ah, I had the thought pickle won't do. See, I cannot figure out another way to do this, so for now it's surely good enough. ''Pythonic'' is a very abstract notion ( as is PEP 20 ). I only
You can use str.encode('encodage') to have hexadecimal, utf-8 or whatever representation of your string, but it's a bit off topic. (the 5 minutes edit session is way too short) Cannot help you.
Actually I am not storing a string representing a date and time, but actually a "native" datetime.datetime object: docs.python.org/2/library/datetime.html#datetime.datetime

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.