This problem is not specific to nullable enums. Even if you use a non-nullable enum, it won't work. That's because while specific enum types are value types, System.Enum is a reference type as explained here. So, you need to add a struct constraint in addition to the Enum constraint.
The following should work:
public static T? ToEnum<T>(this string enumName)
where T : struct, Enum
=> Enum.TryParse<T>(enumName, out T val) ? val : default(T?);
Also, note that you can't use null in the second branch of the ternary operator because the compiler can't infer the type. Use default(T?) instead.
Example of usage:
enum MyEnum { a, b, c }
static void Main(string[] args)
{
var e1 = "b".ToEnum<MyEnum>();
var e2 = "d".ToEnum<MyEnum>();
Console.WriteLine("e1 = " + (e1?.ToString() ?? "NULL")); // e1 = b
Console.WriteLine("e2 = " + (e2?.ToString() ?? "NULL")); // e2 = NULL
}
where T: struct. This has been the case for over a decadeEnumis a class and that's what you're using in your constraint, so that's presumably the issue.System.Enumhas to be a class instead of a struct? @user18387401