1

I need to read JSON data from a web service.

I've created a C# Script Task in SSIS to do this. I'm able to connect to the service, authenticate and get result of the web service back.

The JSON is formatted like this:

{
    "20": {
        "kpiData": [
            {
                "date": "2018-10-01",
                "interval": "DAILY",
                "kpiFigure": 80,
                "symbol": null,
                "additionalInfo": {}
            },
            {
                "date": "2018-10-02",
                "interval": "DAILY",
                "kpiFigure": 58,
                "symbol": null,
                "additionalInfo": {}
            }
        ],
        "average": 69,
        "weightedAverage": 0,
        "max": 80,
        "min": 58,
        "total": 138,
        "symbol": ""
    },
    "24": {
        "kpiData": [
            {
                "date": "2018-10-01",
                "interval": "DAILY",
                "kpiFigure": 133,
                "symbol": null,
                "additionalInfo": {}
            },
            {
                "date": "2018-10-02",
                "interval": "DAILY",
                "kpiFigure": 130,
                "symbol": null,
                "additionalInfo": {}
            }
        ],
        "average": 132,
        "weightedAverage": 0,
        "max": 133,
        "min": 130,
        "total": 263,
        "symbol": ""
    }
}

How do I deserialize this in a C# script task?

I need to get it to output buffer in a "flattened" structure like this:

SiteID ¦ date       ¦ interval ¦ kpiFigure
20     ¦ 2018-10-01 ¦ DAILY    ¦ 80
...
9
  • No, the JSON deserialization is part of it, but you have to mix in all the SSIS script task goop. Commented Nov 14, 2018 at 16:40
  • In SSIS, a Script Task can manipulate SSIS state, and SSIS variables. A Script Transformation component (that you place on a Data Flow surface) can transform input columns into output columns, on a row, by row basis. However, what I'm thinking you are looking for is something that acts as a row source (like a connection manager). Take a look at this: marketplace.visualstudio.com/…. It provides a JSON Source object, which I think is what you are looking for. Caveat, I have no experience with this. Commented Nov 14, 2018 at 16:54
  • thanks! i've seen the 3rd party components, but i don't want to use a commercial component for this. i already have a custom script task almost finished - i'm getting the web service response into a c# string. now i would only need to deserialize the structure i've posted - e.g. traverse through it in C# and put it into an output buffer. i'm not so experienced with c# unfortunately. :/ Commented Nov 14, 2018 at 17:08
  • Then all you need to do is add in the NewtonSoft JSON NuGet package, create a type that matches the JSON data and deserialize the data. There are lots of examples of how to do that on SO. Commented Nov 14, 2018 at 17:10
  • i was trying with nuget, but that seems to be not supported in ssis script task. whenever i add it, the reference seems to be lost when it's built and integrated in ssis. :( Commented Nov 14, 2018 at 17:19

1 Answer 1

1

This isn't the C# route, based on my comment and your response you might want to explore doing this in T-SQL. If you haven't before, here is a high level example of how you hook that up in SSIS. As long as you are SQL Server version 2016+

  1. Have a variable in SSIS, we'll just call JsonResponse, Type=String
  2. You already have your C# Script task to call the web service, add the JsonResponse variable as ReadWriteVariables
  3. Modified your script task and write the response to that variable, I'll usually have separate variable the response is streamed to and then assign that to my ReadWriteVariables:

            using (var response = (HttpWebResponse)request.GetResponse())
            {
                using (Stream responseStream = response.GetResponseStream())
                {
                    using (StreamReader streamReader = new StreamReader(responseStream))
                    {
                        responseString = streamReader.ReadToEnd();
                    }
                    Dts.Variables["User::JsonResponse"].Value = responseString;
                }
            }
    
  4. After the script task in your Control Flow, add an Execute SQL Task

  5. Here you'll have a stored procedure you call that has a @JsonResponse NVARCHAR(MAX) parameter.
  6. You'll be passing the above JsonResponse to this stored procedure and with that proc you can then parse that JSON using OPENJSON and do what you will with it. Insert into another table, etc...

Here is an example of what that T-SQL looks like on how you could parse the example you provide. You'd have to obviously make the necessary adjustments to turn that into a stored procedure and adjust accordingly to what you are specifically trying to accomplish:

DECLARE @JsonResponse NVARCHAR(MAX);

SET @JsonResponse = N'
{
    "20": {
        "kpiData": [
            {
                "date": "2018-10-01",
                "interval": "DAILY",
                "kpiFigure": 80,
                "symbol": null,
                "additionalInfo": {}
            },
            {
                "date": "2018-10-02",
                "interval": "DAILY",
                "kpiFigure": 58,
                "symbol": null,
                "additionalInfo": {}
            }
        ],
        "average": 69,
        "weightedAverage": 0,
        "max": 80,
        "min": 58,
        "total": 138,
        "symbol": ""
    },
    "24": {
        "kpiData": [
            {
                "date": "2018-10-01",
                "interval": "DAILY",
                "kpiFigure": 133,
                "symbol": null,
                "additionalInfo": {}
            },
            {
                "date": "2018-10-02",
                "interval": "DAILY",
                "kpiFigure": 130,
                "symbol": null,
                "additionalInfo": {}
            }
        ],
        "average": 132,
        "weightedAverage": 0,
        "max": 133,
        "min": 130,
        "total": 263,
        "symbol": ""
    }
}
';

SELECT [a].[Key] AS [SiteID]
     , [b].*
FROM   OPENJSON(@JsonResponse) [a]
CROSS APPLY OPENJSON([a].[Value], '$.kpiData')
           WITH (
                    [date] DATE '$.date'
                  , [interval] NVARCHAR(100) '$.interval'
                  , [kpiFigure] INT '$.kpiFigure'
                ) [b];

Just an option/example of how you can get that JSON over to T-SQL and parse it from there.

Sign up to request clarification or add additional context in comments.

1 Comment

This looks exactly what I need. Thank you very much! I didn't think of JSON in T-SQL - I thought the deserialization in C# would be trivial to do. :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.