Azure Durable Functions - diagnose failures

Question

I am testing how durable functions can fan in/out and how I can scale my code. for this example I am simulating many short running, cpu intensive operations.

Not all activities seem to complete and I am not sure why or where to look to find the log for failures.

Please see the code below:

public static class ParallelLoadDurable
    {
        [FunctionName("ParallelLoadDurable")]
        public static async Task<string> RunOrchestrator(
            [OrchestrationTrigger] DurableOrchestrationContext context, ILogger log)
        {
            DateTime StartTimer = DateTime.Now;

            int counter = 0;
            var parallelTasks = new List<Task<string>>();
            var retryOptions = new RetryOptions(
                 firstRetryInterval: TimeSpan.FromSeconds(5),
                 maxNumberOfAttempts: 5);
            for (int i = 0; i < 1000; i++)
            {
                counter += 1;
                DurablePassModel DPM = new DurablePassModel()
                {
                    LoopNum = counter,
                    StartedOn = StartTimer
                };
                Task<string> task = context.CallActivityWithRetryAsync<string>("ParallelLoadDurable_Hello", retryOptions, DPM);
                parallelTasks.Add(task);
            }
            await Task.WhenAll(parallelTasks);

            DateTime CompleteTime = DateTime.Now;
            TimeSpan TS = CompleteTime.Subtract(StartTimer);

            string ret = $"PROCESS COMPLETED: {counter} times for: {TS.TotalMilliseconds} ms.";
            log.LogInformation(ret);
            return ret;
        }

        [FunctionName("ParallelLoadDurable_Hello")]
        public static string SayHello([ActivityTrigger] DurablePassModel val, ILogger log)
        {
            log.LogInformation($"Starting child function num {val.LoopNum.ToString()}.");
            DateTime StartTimer = DateTime.Now;

            var endTime = DateTime.Now.AddSeconds(10);

            while (true)
            {
                if (DateTime.Now >= endTime)
                    break;
            }

            DateTime CompleteTime = DateTime.Now;
            TimeSpan TS = CompleteTime.Subtract(val.StartedOn);
            TimeSpan TSThis = CompleteTime.Subtract(StartTimer);

            string ret = $"Ran this for: {TSThis.TotalSeconds}s - LoopNum: {val.LoopNum} - total time: {TS.TotalSeconds}s.";
            log.LogInformation(ret);

            return ret;
        }

        [FunctionName("ParallelLoadDurable_HttpStart")]
        public static async Task<HttpResponseMessage> HttpStart(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")]HttpRequestMessage req,
            [OrchestrationClient]DurableOrchestrationClient starter,
            ILogger log)
        {
            // Function input comes from the request content.
            string instanceId = await starter.StartNewAsync("ParallelLoadDurable", null);

            log.LogInformation($"Started orchestration with ID = '{instanceId}'.");

            return starter.CreateCheckStatusResponse(req, instanceId);
        }
    }

What I get in almost every case is about 96% of the expected activities completed. this is from the results in the history table where EventType = TaskCompleted

Also in the "instances" table the RuntimeStatus just remains on "running"

Where could I look to find the list of failures?

Thanks for any help

Nick

HariHaran · Accepted Answer · 2019-05-21 04:25:39Z

1

Surround your code with a try catch block and then log the exception using the Ilogger.

It can be achieved by like this

try
{
   //Do something
}
catch(Exception ex){
   log.LogError($"Error in function: {ex.Message}");
}

Then you can view the error messages in the logs or as well as Application insights if you have one.

answered May 21, 2019 at 4:25

HariHaran

4,4592 gold badges26 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Marc · Accepted Answer · 2019-05-21 11:29:37Z

I suggest you configure Application Insights for Azure Functions: https://learn.microsoft.com/en-us/azure/azure-functions/functions-monitoring Although there is some cost involved for the storage of the data it does help a lot when investigating issues.

In addition (or when you don't want Application Insights) you can query the status of the orchestration either via C# or HTTP. This also works very nicely when you run & debug on your local machine!

Using the HTTP API you could do the following query to determine the failed instances:

@functionAppUrl = https://{{appName}}.azurewebsites.net
@code = YOUR_HOST_KEY
@taskHub = YOUR_HUB_NAME (see host.json)
@createdFrom = 2019-02-01T20:00:00Z

GET {{functionAppUrl}}/runtime/webhooks/durabletask/instances
        ?taskHub={{taskHub}}
        &code={{code}}
        &createdTimeFrom={{createdFrom}}
        &runtimeStatus=Failed

I noticed you're using DateTime.Now in your orchestration code. It is advised to use the CurrentUtcDateTime property from the DurableOrchestrationContext since the behavior in the orchestration function should be deterministic. See this section about orchestrator code constraints.

Collectives™ on Stack Overflow

Azure Durable Functions - diagnose failures

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related