Correctly escape a trailing backslash in a string within XML to JSON function?

This function isn't needed at all. You can generate JSON from query results in various forms using FOR JSON and produce nested results if needed.

For example, this query (fiddle)

select 
    root.value('Book[1]','nvarchar(50)') as Book,
    root.value('Publisher[1]','varchar(50)') as Publisher,
    root.value('PublisherId[1]','varchar(50)') as PublisherId,
    root.value('Release[1]','varchar(50)') as Release,
    root.value('UserId[1]','varchar(50)') as UserId    ,
    ed.value('Name[1]','varchar(20)')  as [Edition.Name],
    ed.value('Color[1]','varchar(20)') as [Edition.Color],
    ed.value('Price[1]','varchar(20)') as [Edition.Price],
    ed.value('file[1]','varchar(20)')  as [Edition.File],
    TransactionId.value('.','varchar(20)') as [TransactionId.TransactionId]
from 
@xml.nodes('/root') x(root)
outer apply root.nodes('Edition') q(ed)
outer apply root.nodes('TransactionId') t(TransactionId)
for JSON PATH,WITHOUT_ARRAY_WRAPPER

Produces this result :

{"Book":"Book1","Publisher":"Amazon","PublisherId":"1","Release":"","UserId":"1234","Edition":{"Name":"Ed1","Color":"Red","Price":"100","File":"C:\\Books\\Book1\\Book1"},"TransactionId":{"TransactionId":"abc123"}}

or, pretty-printed :

{
  "Book": "Book1",
  "Publisher": "Amazon",
  "PublisherId": "1",
  "Release": "",
  "UserId": "1234",
  "Edition": {
    "Name": "Ed1",
    "Color": "Red",
    "Price": "100",
    "File": "C:\\Books\\Book1\\Book1"
  },
  "TransactionId": {
    "TransactionId": "abc123"
  }
}

The Fiddle output displays the unescaped strings, which can lead to confusion.

The data doesn't contain any arrays so it's unclear why Edition and TransitionId should appear as arrays. That json:Array="true" means nothing. http://james.newtonking.com/projects/json is a sample namespace from a JSON.NET sample project.

If the XML actually contained an array of objects, eg :

DECLARE @xml xml = N'<root>
    <Book>Book1</Book>
    <TransactionId  xmlns:json="http://james.newtonking.com/projects/json" json:Array="true">abc123</TransactionId>
    <Publisher>Amazon</Publisher>
    <Editions>
        <Edition>
            <Name>Ed1</Name>
            <Color>Red</Color>
            <Price>100</Price>
            <file>C:\Books\Book1\Book1.pdf\</file>
        </Edition>
        <Edition>
            <Name>Ed2</Name>
            <Color>Green</Color>
            <Price>200</Price>
            <file>C:\Books\Book1\Book2.pdf\</file>
        </Edition>
    </Editions>
    <PublisherId>1</PublisherId>
    <UserId>1234</UserId>
    <Release />
</root>
';

You could generate an array of Editions in the output with :

select 
    root.value('Book[1]','nvarchar(50)') as Book,
    root.value('Publisher[1]','varchar(50)') as Publisher,
    root.value('PublisherId[1]','varchar(50)') as PublisherId,
    root.value('Release[1]','varchar(50)') as Release,
    root.value('UserId[1]','varchar(50)') as UserId    ,
    (
     select
        ed.value('Name[1]','varchar(20)')  as Name,
        ed.value('Color[1]','varchar(20)') as Color,
        ed.value('Price[1]','varchar(20)') as Price,
        ed.value('file[1]','varchar(20)')  as [File]
        from root.nodes('Editions/Edition') q(ed) FOR JSON PATH
    ) as Editions,
    TransactionId.value('.','varchar(20)') as TransactionId
from 
@xml.nodes('/root') x(root)
outer apply root.nodes('TransactionId') t(TransactionId)
for json auto,WITHOUT_ARRAY_WRAPPER;

This produces :

{
  "Book": "Book1",
  "Publisher": "Amazon",
  "PublisherId": "1",
  "Release": "",
  "UserId": "1234",
  "Editions": [
    {
      "Name": "Ed1",
      "Color": "Red",
      "Price": "100",
      "File": "C:\\Books\\Book1\\Book1"
    },
    {
      "Name": "Ed2",
      "Color": "Green",
      "Price": "200",
      "File": "C:\\Books\\Book1\\Book2"
    }
  ],
  "TransactionId": "abc123"
}

The results of the subquery became the Editions array attribute

I think the problem is that you don't need the extra replace, but that DbFiddle is fooling you:

See:

https://dbfiddle.uk/CWdL_2BU

When running, the second column seem to miss an extra \ even though the LEN thing returns 6:

When running your function code on my local computer but without the REPLACE(..., '', '\') i get proper results:

CREATE OR ALTER FUNCTION dbo.XmlToJson(@XmlData xml)  
RETURNS nvarchar(max)
WITH RETURNS NULL ON NULL INPUT
AS  
BEGIN  
    DECLARE @m nvarchar(max);

    WITH XMLNAMESPACES (
        N'http://james.newtonking.com/projects/json' AS json
    )
    SELECT @m = '{' + STRING_AGG(
      '"' + STRING_ESCAPE(name, 'json') + '":' + value,
      ','
    ) + '}'
    FROM (
        SELECT
            v.name,
            CONCAT(
              CASE WHEN COUNT(*) > 1 OR MAX(isArray) = 1 THEN '[' END,
              STRING_AGG(
                ISNULL(
                  '"' + STRING_ESCAPE(x.a.value('text()[1]', 'nvarchar(max)'), 'json') + '"',
                  dbo.XmlToJson(x.a.query('./*'))
                ),
                ','
              ),
              CASE WHEN COUNT(*) > 1 OR MAX(isArray) = 1 THEN ']' END
           ) AS value
        FROM @XmlData.nodes('./*') x(a)
        CROSS APPLY (SELECT
            x.a.value('local-name(.)', 'nvarchar(4000)') AS name,
            x.a.value('xs:int(xs:boolean(@json:Array))', 'int') AS isArray
        ) v
        GROUP BY
            v.name
    ) grouped;
    SET @m = ISNULL(@m, 'null');
    SET @m = REPLACE(@m, '\/', '/');
    RETURN @m;
END;

GO

...

SELECT dbo.XmlToJson(@xml.query('/root/*'));

Outputs:

{"Book":"Book1","Edition":[{"Color":"Red","file":"C:\\Books\\Book1\\Book1.pdf\\","Name":"Ed1","Price":"100"}],"Publisher":"Amazon","PublisherId":"1","Release":null,"TransactionId":["abc123"],"UserId":"1234"}