How do I get replaceRoot to return an object instead of an array?

The problem doesn't have to do with $replaceRoot directly. In the question it's noted that the field with an unexpected type is cancelPolicy. The cancelPolicy field, not to be confused with cancelPolicies or cancelPolicyId, contains the result of the $lookup. That is to say that cancelPolicy is the as parameter of the $lookup which is defined as follows in the documentation (emphasis added):

Specifies the name of the new array field to add to the input documents. The new array field contains the matching documents from the from collection. If the specified name already exists in the input document, the existing field is overwritten.

There is nothing that you can do inside of the $lookup, including in its nested pipeline, to coerce the type into anything other than an array.

The solution is to modify that field in these new documents in a subsequent stage. Most commonly this is achieved with an $unwind stage. So the aggregation would look something like this:

$lookup: {
  from: "properties",
  let: { propertyId: "$propertyId", cancelPolicyId: "$cancelPolicyId" },
  pipeline: [
    { 
      $match: { $expr: { $eq: [ "$_id", "$$propertyId" ] } }
    },
    {
      $unwind: {
        path: "$cancelPolicies",
        preserveNullAndEmptyArrays: true
      }
    },
    { 
      $match: { $expr: { $eq: [ "$cancelPolicies._id", "$$cancelPolicyId" ] } }
    },
    {
      $replaceRoot: { newRoot: "$cancelPolicies" }
    },
  ],
  as: "cancelPolicy"
},
{
  $unwind: "$cancelPolicy"
}
]

You can include the preserveNullAndEmptyArrays parameter in there as well depending on whether you are looking to perform a logical inner join or a left join.