Remove object from nested array of objects

I got this user schema

const UserSchema = new Schema({
  email: {
    type: String,
    required: true,
    unique: true,
  },
  groups: [
    {
      groupName: {
        type: String,
        required: true,
      },
      groupMembers: [{ type: Schema.Types.ObjectId, ref: "GroupMember" }],
    },
  ],
});

And I want to delete a group from the ‘groups’ array based on given ‘userId’ and ‘groupId’, my attempt (with express and mongoose):

router.delete(  
  "/:userId/:groupId",
  catchAsync(async (req, res) => {
    const { userId, groupId } = req.params;
    const updatedUser = await User.findByIdAndUpdate(
      userId,
      { $pull: { "groups.$._id": groupId } },
      { new: true }
    );
    res.send({ updatedUser });
  })
);

The response of the request: I get an error: “The positional operator did not find the match needed from the query.”

Edit: After I delete a group I need to remove all the group members in the groupMembers array. User collection structure example:

{
"_id" : "111",
"email" : "[email protected]",
"username" : "michael098",
"groups" : [
    {
        "_id" : "222"
        "groupName" : "family",
        "groupMembers" : [
            {
                "_id" : "333"
                "name" : "Liam"
            },
            {
                "_id" : "444"
                "name" : "Noah"
            }
        ]
    },
    {
        "_id" : "555"
        "groupName" : "friends",
        "groupMembers" : [
            {
                "_id" : "666"
                "name" : "Oliver"
            }
        ]
    }
  ]
}

Inside every group there is group members and I have a collection for the group members that I ref in the UserSchema : groupMembers: [{ type: Schema.Types.ObjectId, ref: "GroupMember" }] GroupMember collection structure example:

{
    {
        "_id" : "333"
        "name" : "Liam"
    },
    {
         "_id" : "444"
         "name" : "Noah"
    },
    {
         "_id" : "666"
         "name" : "Oliver"
    }
 }

For example when I get the params of userId="111" and groupId="222" I will delete the whole 'family' group and the whole group members in the groupMembers array (Liam and Noah) from the GroupMember collection.

GroupMember collection after deleting the group with _id="222":

 {
        {
             "_id" : "666"
             "name" : "Oliver"
        }
  }