Changing byte-width midstream in Common LISP

Question

Suppose I have some utility functions defined as:

(defun write-bytes-to-file (bytes file-path bits)
  (with-open-file (stream file-path
                           :direction :output
                           :if-does-not-exist :create
                           :if-exists :append
                           :element-type (list 'unsigned-byte bits))
     (dolist (b bytes))
       (write-byte b stream))))

(defun read-file-bytes-to-list (file-path bits)
  (with-open-file (stream file-path :direction :input :element-type (list 'unsigned-byte bits))
     (read-bytes-to-list stream nil)))

Furthermore, let's say I run:

(write-bytes-to-file '(65 65 65) "foo.txt" 8)
(write-bytes-to-file '(512) "foo.txt" 9)

Now this leaves me with a variable byte-width file with three 8-bit bytes, and one with 9-bit width. Normally I'd use my read-file-bytes-to-list function with a bit-width input to read the file with the correct alignment, but in this case I can't really do that. Is there a built-in way I can change the byte alignment mid-stream in Common LISP? Essentially I'd like to read the integers back as intended (65, 65, 65, and 512). Thanks for your help. I'm using the SBCL implementation.

EDIT: If there is no convenient way to handle this using the standard LISP libraries/utilities, I realize that I most likely will have to handle this with bit-wise operations; that's fine. I was just wondering if there is a better way to do it, before doing so.

coredump · Accepted Answer · 2020-05-18 07:39:44Z

First of all, please consider using lisp-binary when encoding/decoding binary formats.

I modified your writing function to fix the parentheses and print the actual stream element type:

(defun write-bytes-to-file (bytes file-path bits)
  (with-open-file (stream file-path
                           :direction :output
                           :if-does-not-exist :create
                           :if-exists :append
                           :element-type (list 'unsigned-byte bits))
    (print (stream-element-type stream))
    (dolist (b bytes)
      (write-byte b stream)))))

With this function, I built a binary file:

USER> (when-let (file (probe-file "/tmp/test.data"))
        (delete-file file))
T
USER> (write-bytes-to-file '(1 1 1 1) "/tmp/test.data" 9)

(UNSIGNED-BYTE 16) 
NIL
USER> (write-bytes-to-file '(1 1 1 1) "/tmp/test.data" 8)

(UNSIGNED-BYTE 8) 
NIL
USER>

As you can see, encoding (unsigned-byte 9) elements is done by opening a stream that uses bytes of size 16. If you look at the generated file with hexdump:

$ hexdump /tmp/test.data
0000000 0001 0001 0001 0001 0101 0101          
000000c

You can see that the first 4 ones are encoded on 16 bits words, followed by 4 ones encoded on 8 bits. In order to decode the data, you need to open it several times (as for writing) and seek the appropriate position in the file. The following was tested on SBCL, and there is not guarantee it will work portably:

(defun read-file-bytes-to-list (file-path bits count &optional (bits-offset 0))
  (with-open-file (stream file-path :direction :input :element-type (list 'unsigned-byte bits))
    (destructuring-bind (_ bits) (stream-element-type stream)
      (declare (ignore _))
      (loop
         initially (file-position stream (/ bits-offset bits))
         repeat count
         collect (read-byte stream) into bytes
         finally (return (values bytes (* bits (file-position stream))))))))

At the end of the function, we return both the decoded bytes and the file position, expressed in bits, after reading is done. Returning the amount of bits is necessary, since file-position returns a multiple of the stream element's size.

When we reopen the file, we give the last offset as bits, and the new stream element type is used to compute the offset to give to file-position.

For example:

USER> (read-file-bytes-to-list "/tmp/test.data" 9 4)
(1 1 1 1)
64
USER> (read-file-bytes-to-list "/tmp/test.data" 8 4 64)
(1 1 1 1)
96

Thank you for the very lucidly explained answer. This approach seems very sensible, and idiomatic too.

Collectives™ on Stack Overflow

Changing byte-width midstream in Common LISP

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related