How to structure TypeScript event shapes to allow for event-specific middleware arguments?

I am working on an event-driven application framework that leverages a middleware pattern to process events dispatched from my service to third party applications. My service models many different kinds of events that third party applications may wish to register for and receive.

As event payloads are received by the framework, they are 'augmented' with middleware-specific additions like a next() function and user-definable context - relatively standard issue middleware stuff. Some events, however, have further specific augmentations available only on particular events. For example, message events have a message property, foo events may have a foo property, and so on - though the event type/name and the event-specific augmentation(s) may not map perfectly one-to-one and/or they may have multiple augmentations.

The way these event-specific augmentations are currently typed in the app framework is problematic and causes errors often; as a result, the codebase uses type assertions (as) a lot - probably a bad sign. Admittedly I am not a TypeScript expert and the framework code predates my involvement; I just thought 'this is how TS projects are built I guess.' After a couple years and learning more about TypeScript, I think now that there must be a better way!

Here is a TypeScript playground for the example, that I also copied below. The wrapMiddleware method errors with:

Argument of type 'MiddlewareArgs<"message">' is not assignable to parameter of type 'MiddlewareArgs<string>'.
  Types of property 'message' are incompatible.
    Type 'MsgEvent' is not assignable to type 'undefined'.

import { expectAssignable } from 'tsd';

// A couple of events, and a union of all events (there are more in actuality)
interface MsgEvent {
  type: 'message';
  text: string;
  channel: string;
  user: string;
}
interface JoinEvent {
  type: 'join';
  channel: string;
  user: string;
}
type AllEvents = MsgEvent | JoinEvent;

// Utility types to 'extract' event payloads out based on the `type` property
type KnownEventFromType<T extends string> = Extract<AllEvents, { type: T }>;
type EventFromType<T extends string> = KnownEventFromType<T> extends never ? { type: T } : KnownEventFromType<T>;

// A contrived example of arguments passed to middleware
interface MiddlewareArgs<EventType extends string = string> {
  event: EventFromType<EventType>;
  message: EventType extends 'message' ? this['event'] : undefined; // <-- problematic; there must be a better way, right?
}

// Augmenting events with additional middleware things
type AllMiddlewareArgs = {
  next: () => Promise<void>;
}
function wrapMiddleware<Args extends MiddlewareArgs>(
  args: Args,
): Args & AllMiddlewareArgs {
  return {
    ...args,
    next: async () => {},
  }
}

// And now for the actual example:
const messageEvt: MsgEvent = {
  type: 'message',
  channel: 'random',
  user: 'me',
  text: 'hello world',
}
const messageEvtArgs: MiddlewareArgs<'message'> = {
  event: messageEvt,
  message: messageEvt,
}
const joinEvt: JoinEvent = {
  type: 'join',
  channel: 'random',
  user: 'me'
}
const joinEvtArgs: MiddlewareArgs<'join'> = {
  event: joinEvt,
  message: undefined, // <-- bonus points if we can get rid of having to set an undefined message!
}

// Some test cases
expectAssignable<AllMiddlewareArgs>(wrapMiddleware(messageEvtArgs));
expectAssignable<MiddlewareArgs<'message'>>(wrapMiddleware(messageEvtArgs));
expectAssignable<AllMiddlewareArgs>(wrapMiddleware(joinEvtArgs));
expectAssignable<MiddlewareArgs<'join'>>(wrapMiddleware(joinEvtArgs));
// Wrapping random untyped events should fallback
expectAssignable<AllMiddlewareArgs>(wrapMiddleware({ event: { type: 'garbage' }}));
expectAssignable<MiddlewareArgs<'garbage'>>(wrapMiddleware({ event: { type: 'garbage' }}));

I understand why I get the error I do: the wrapMiddleware method accepts the wider, non-message-specific MiddlewareArgs type, which sets the generic parameter to be string, so the interface's message property could be either a message object or undefined as per the conditional type for the message property. My question is: what's a better way to structure this approach so that it scales to more events, with different event-specific middleware argument shapes?