Java.io.IOException, "bad file number" USB connection

I'm setting up a USB accessory connection between my Android phone and another device. Just sending bytes back and forth for now to test. I get some definite communication going at first, but it always ends up dying with Java.io.IOException: write failed: EBADF (Bad file number)" after a second or so. Sometimes the reading stays alive but the writing dies; others both die.

I'm not doing anything super fancy, reading and writing just like the Google documentation:

Initial connection (inside a broadcast receiver, I know this part works at least initially):

if (action.equals(ACTION_USB_PERMISSION))
{
    ParcelFileDescriptor pfd = manager.openAccessory(accessory);
    if (pfd != null) {
        FileDescriptor fd = pfd.getFileDescriptor();
        mIn = new FileInputStream(fd);
        mOut = new FileOutputStream(fd);
    }
}

Reading:

Thread thread = new Thread(new Runnable() {
    @Override
    public void run() {
        byte[] buf = new byte[BUF_SIZE];
        while (true)
        {
            try {
                int recvd = mIn.read(buf);
                if (recvd > 0) {
                    byte[] b = new byte[recvd];
                    System.arraycopy(buf, 0, b, 0, recvd);
                    //Parse message
                }
            }
            catch (IOException e) {
                Log.e("read error", "failed to read from stream");
                e.printStackTrace();
            }
        }
    }
});
thread.start();

Writing:

synchronized(mWriteLock) {
    if (mOut !=null && byteArray.length>0) {
        try {
            //mOut.flush();
            mOut.write(byteArray, 0, byteArray.length);
        }
        catch (IOException e) {
            Log.e("error", "error writing");
            e.printStackTrace();
            return false;
        }
    }
    else {
        Log.e(TAG, "Can't send data, serial stream is null");
        return false;
    }
}

Error stacktrace:

java.io.IOException: write failed: EBADF (Bad file number)
W/System.err(14028):     at libcore.io.IoBridge.write(IoBridge.java:452)
W/System.err(14028):     at java.io.FileOutputStream.write(FileOutputStream.java:187)
W/System.err(14028):     at com.my.android.transport.MyUSBService$5.send(MyUSBService.java:468)
W/System.err(14028):     at com.my.android.transport.MyUSBService$3.onReceive(MyUSBService.java:164)
W/System.err(14028):     at android.app.LoadedApk$ReceiverDispatcher$Args.run(LoadedApk.java:781)
W/System.err(14028):     at android.os.Handler.handleCallback(Handler.java:608)
W/System.err(14028):     at android.os.Handler.dispatchMessage(Handler.java:92)
W/System.err(14028):     at android.os.Looper.loop(Looper.java:156)
W/System.err(14028):     at android.app.ActivityThread.main(ActivityThread.java:5045)
W/System.err(14028):     at java.lang.reflect.Method.invokeNative(Native Method)
W/System.err(14028):     at java.lang.reflect.Method.invoke(Method.java:511)
W/System.err(14028):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:784)
W/System.err(14028):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:551)
W/System.err(14028):     at dalvik.system.NativeStart.main(Native Method)
W/System.err(14028): Caused by: libcore.io.ErrnoException: write failed: EBADF (Bad file number)
W/System.err(14028):     at libcore.io.Posix.writeBytes(Native Method)
W/System.err(14028):     at libcore.io.Posix.write(Posix.java:178)
W/System.err(14028):     at libcore.io.BlockGuardOs.write(BlockGuardOs.java:191)
W/System.err(14028):     at libcore.io.IoBridge.write(IoBridge.java:447)
W/System.err(14028):     ... 13 more

I have logging all over the place and thus I know it's not anything too obvious, such as another permission request being received (and thus the file streams being reinitialized mid-read). The streams aren't closing either, because I never have that happen anywhere in my code (for now). I'm not getting any detached or attached events either (I log that if it happens). Nothing seems too out of the ordinary; it just dies.

I thought maybe it was a concurrency issue, so I played with locks and sleeps, nothing worked that I tried. I don't think it's a throughput issue either because it still happens when I sleep every read (on both ends), and read one single packet at a time (super slow bitrate). Is there a chance the buffer is being overrun on the other end somehow? How would I go about clearing this? I do have access to the other end's code, it is an Android device as well, using Host mode. In case that it matters, I can post that code too - standard bulk transfers.

Does the phone just have lackluster support for Android Accessory Mode? I've tried two phones and they both fail similarly, so I doubt it's that.

I'm wondering what causes this error in general when writing or reading from USB on Android?