Merging multiple json arrays in the same file

Question

I'm currently downloading a ton of jira issues to generate a report. Currently the 'full data' file has a ton of individual records like this:

{
    "key": "645",
    "type": "Bug",
    "typeid": "1",
    "status": "Closed",
    "summary": "Crash when saving document",
    "closedDate": "2014-10-03T09:01:23.000+0200",
    "flag": null,
    "fixVersionID": "123",
    "fixVersionName": "2.7"
}

However, because I'm downloading multiple versions and appending to the same file I end up with this kind of structure.

[
  {
    "key": "645",
    "type": "Bug",
    "typeid": "1",
    "status": "Closed",
    "summary": "Crash when saving document",
    "closedDate": "2014-10-03T09:01:23.000+0200",
    "flag": null,
    "fixVersionID": "123",
    "fixVersionName": "2.7"
  }
]
[
  {
    "key": "552",
    "type": "Bug",
    "typeid": "1",
    "status": "Closed",
    "summary": "Graphical Issue",
    "closedDate": "2014-10-13T09:01:23.000+0200",
    "flag": null,
    "fixVersionID": "456",
    "fixVersionName": "2.8"
  }
]

What I want to do is to count the number of records with a specific date and then doing the same looping through a starting date to an end date using jq

But, I can't figure out how to:

Flatten the records so that they are one array not two
Strip the T09:01:23.000+0200 from the closedDate value
Count the number of objects with a specific date value such as 2014-10-13

Actually, I figured out how to strip T09:01:23.000+0200. when you map the jira issue do: map({closedDate: .fields.resolutiondate[0:10]}) that'll give you just the first ten characters. — TheMightyLlama
– TheMightyLlama, Commented Feb 12, 2015 at 8:09
grep '2014-10-13' some.json | wc -l will it not be enough to get the count? — pratZ
– pratZ, Commented Feb 12, 2015 at 8:22
@pratZ it will, but I wanted a slightly cleaner solution. I know that jq has a max and a min function and was looking for something similar. But if grep does it, then so be it. The real issue is the flattening of the arrays. — TheMightyLlama
– TheMightyLlama, Commented Feb 12, 2015 at 8:28

Jeff Mercado · Accepted Answer · 2015-02-12 17:21:44Z

1

You have multiple independent inputs. To be able to combine them in any meaningful way, you'll have to slurp up the input. The inputs will be treated as an array of the inputs. Then you could combine them into a single array by adding them.

Since the dates are all in a certain fixed format, you can take substrings of the dates.

"2014-10-13T09:01:23.000+0200"[:10] -> "2014-10-13"

Given that, you can then filter by the date you want and count using the length filter.

add | map(select(.closedDate[:10]=="2014-10-13")) | length

e.g.,

$ cat input.json
[
  {
    "key": "645",
    "type": "Bug",
    "typeid": "1",
    "status": "Closed",
    "summary": "Crash when saving document",
    "closedDate": "2014-10-03T09:01:23.000+0200",
    "flag": null,
    "fixVersionID": "123",
    "fixVersionName": "2.7"
  }
]
[
  {
    "key": "552",
    "type": "Bug",
    "typeid": "1",
    "status": "Closed",
    "summary": "Graphical Issue",
    "closedDate": "2014-10-13T09:01:23.000+0200",
    "flag": null,
    "fixVersionID": "456",
    "fixVersionName": "2.8"
  }
]
$ jq -s 'add | map(select(.closedDate[:10]=="2014-10-13")) | length' input.json
1

edited Feb 12, 2015 at 17:21

answered Feb 12, 2015 at 8:33

Jeff Mercado

136k34 gold badges267 silver badges282 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Jeff Mercado Over a year ago

Meh, insert desired method of input as needed. For a file, yes, useless, but I have no idea what the actual source is, but whatever it is, it probably can be piped in.

TheMightyLlama Over a year ago

@JeffMercado source of input is a curl call to jira which has an upper limit on the number of objects retrieved. So I have to do sequential calls and write them to a file. I wish I knew why that was useless though. It seems useful to me!

Jeff Mercado Over a year ago

@TheMightyLlama: Search "UUOC". The idea is that using cat to pipe to another process is a useless, inefficient use of `cat. It's better to redirect instead or use some other form of input. Personally I find no problem with its use.

soflare · Accepted Answer · 2015-02-12 10:59:20Z

1

For question 1 and 2:

$ echo -e "[\n$(sed '/^[][]$/d;/closedDate/s/\(T[^"]*\)//g' json)\n]" > flat-json

To count the number for special day:

$ grep "closedDate" flat-json | grep "2014-10-13" | wc -l

answered Feb 12, 2015 at 10:59

soflare

8116 silver badges17 bronze badges

Collectives™ on Stack Overflow

Merging multiple json arrays in the same file

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related