Note, I don't think about performance, only about kotlin.

Iluminate branch

in your when, you check if the rightIndex >= string.length+1.
this means in the next branches the size is maximal string.length. Substring is exclusive, so it allows str.length.
this means the second and third branches are identical.

Use Set

It would make more sense to me to use toSet instead of toList.distinct, because sets are meant to hold distinct versions.

personal Choices:

Single-expression function

A single-expression-function is a function with one expression 🙄.
In kotlin, you can write this in an easier way.
Instead of writing

fun lengthOfLongestSubstring(str: String): Int {
    return when {
        ...
    }
}

you can write

fun lengthOfLongestSubstring2(str: String): Int = when {
    ...
}

or even without return-type:

fun lengthOfLongestSubstring2(str: String) = when {
    ...
}

I personally like to write these like:

fun helper(
    str: String,
    leftIndex : Int,
    rightIndex : Int,
    max: Int
) = when {
    ...
}

function with receiver

If I'm right functional means:

• The function knows about the type and the type doesn't know about the function
• If a function receives the same input, it should return the same output

(or only the later)

The first way is often expressed by having a function that only takes parameters, but is not called on an object: b(a) instead of a.b()

Kotlin allows to let a function like b(a) look like a.b(), if you define it that way. This is called a function with receiver.
Note, inside de function-body, you refer to the receiver using this.

isDistinct("hi")
private fun isDistinct(
    substring: String
): Boolean = substring.toSet().size == substring.length

can be rewritten to

"hi".isDistinct()
private fun String.isDistinct() : Boolean = this.toSet().size == this.length

And as you don't have to call this inside a scope:

"hi".isDistinct()
private fun String.isDistinct() : Boolean = toSet().size == length

Or just:

"hi".isDistinct()
private fun String.isDistinct() = toSet().size == length

Under the scenes it's still the same function. Inside kotlin, you can't call it the old way anymore

New code

class Solution {
    fun lengthOfLongestSubstring(
        str: String
    ) = when {
        str.length == 1 -> 1
        str.isEmpty() -> 0
        else -> helper(str, 0, 1, 0)
    }

    private tailrec fun helper(
        str: String,
        leftIndex: Int,
        rightIndex: Int,
        max: Int
    ): Int = when {
        rightIndex >= str.length + 1 -> max
        str.substring(leftIndex, rightIndex).isDistinct() ->
            helper(str, leftIndex, rightIndex + 1, max(rightIndex - leftIndex, max))
        else -> helper(str, leftIndex + 1, leftIndex + max + 1, max)
    }

    private fun String.isDistinct() = toSet().size == length
}