0

Hello all who read this,

We have written a router function on azure in an app plan that receives messages from iothub and depending the message type we route our message to another eventhub.

Previously we had 6 out bindings to eventhubs in this function Recently we added 3 more message type so 3 more out binding to 3 more eventhubs

No processing of the messages happen in this function but what we see now is that we spend 16 times more time in the routing function.

Is there a performance issue about having multiple output bindings. We don't see an increase in load of the incoming messages.

We are running on azure functions 1.0 (Runtime version: 1.0.12205.0 (~1))

Regards Ben

Simplified Sample code of the routing function 

public static class IotHubRouterFunction
{
    [FunctionName("IotHubRouterFunction")]
    public static void Run([EventHubTrigger("%iothub%", Connection = "IothubRouterListen")]EventData myEventHubData,
        [EventHub("%msg1-eventhub%", Connection = "msg1event")] ICollector<EventData> eventHub4Dmsg1Event,
        [EventHub("%msg2-eventhub%", Connection = "msg2event")] ICollector<EventData> eventHub4Dmsg2Event,
        [EventHub("%msg3-eventhub%", Connection = "msg3event")] ICollector<EventData> eventHub4Dmsg3Event,
        //...  like 6 more bindings like this
        ILogger logger
    )
    {
        try
        {
            var messageType = GetValue(myEventHubData.Properties, "type");

            // routing
            switch (messageType)
            {
                case "msg1event":
                {
                    eventHub4DevicesStatusChanged.Add(eventHub4Dmsg1Event);
                    break;
                }
                case "msg2event":
                {
                    eventHub4MeasurementLog.Add(eventHub4Dmsg2Event);
                    break;
                }
                case "msg3event":
                {
                    eventHub4DeviceDiscovered.Add(eventHub4Dmsg3Event);
                    break;
                }
                //6 more cases like this
                default:
                {
                    logger.LogError("Unrouteable message of type: {messageType}", messageType);
                    break;
                }
            }
        }
        catch (Exception ex)
        {
            //removed
        }
    }        
}

With 6 bindings the message fly through the router function at 50ms With 9 bindings the message crawl through the router function at 800ms

CPU raised with 30% as well on the applan (we scaled extra so we have it under control but why so much what is causing this)

Improve this question

2 Answers 2

Reset to default
1

A little late with the follow up of what happened

In the end we found out what was going on We have several instances of our app plan but the old monitoring solution showed the average of the cpu and memory overall the instances of the applan.

Basically with switching to the newer metrics and azure monitoring we were able to drill down in the separate instances of the app plan and the instances of the functions.

We found out that one instance of a function which was running three times two of them norammly but the third function had crashed it's internal apppool and consumed all cpu power it got hold off and did absolutely nothing.

We restarted the function and all issues were gone.

Still wondering if it was something in our code that made it go through the roof or that something happened in azure that made it go crazy.

:-s

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

When you are using Azure Function under App service plan then you have to watch out for performance parameters like scaling. Have you investigated your function is not getting overloaded ?

On the other hand , As part of your design this approach is wrong to me. With this many bindings there could be potential performance issues , and what if you are supposed to add more bindings in future ? If you are not performing any operation then you shouldn't be taking overhead of redirecting messages.

Event Grid

We can use event grids for that. Based on topic the IoT hub publishes the event to a topic and events are consumed by subscribers in your case other event hubs. You also get advantage of micro billing (serverless) and auto scaling as well. https://learn.microsoft.com/en-us/azure/event-grid/overview

Improve this answer

8 Comments

No the system is not overloaded, there is no increase in load. When we started this solution event grid was not available this approach was verified by microsoft consultants we are planning to route over 50 different messages types in this way. I want to know why you say "With this many bindings there could be potential performance issues". Because that was my question is there a performance hit adding so many bindings and why. Because that means we were very Ill advised.
Eventgrid is on the radar but that is not something we can just directly roll out in our current production setup are product is running and working but seems to have a decent performance hit by adding an extra bindings.
Out of box solution should be fine but When i say could be problem i assume you are redirecting based on your custom code which might have some issue like blocking calls etc. How actually are you redirecting them ? can you share code sample? The delay you mention is way too high. Also share numbers like how long it takes to process in terms of seconds and how many messages.
I extended the post
Thanks for sharing the code. Looking at this i can't find the reason for this much delay. I think you would have to look deeper into the logs around what's going on. If you haven't already i would suggest to use "Application Insights" to drill down into the function and share your results.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.