Bash/*NIX: split a file into multiple files on a substring

^{If using string parsing rather than proper JSON parsing to extract the county name is acceptable - brittle in general, but would work in this simple case - consider Sam Tolton's GNU awk answer, which has the potential to be by far the simplest and fastest solution.}

To complement chepner's excellent answer with a variation that focuses on performance:

jq -Rrn '[inputs[:-1]|fromjson] | .properties.county + "|" + (.|tostring)' file |
  awk -F'|' '{ print $2 > ($1 "_bins_2016.json") }'

Shell loops are avoided altogether, which should speed up the operation.

The general idea is:

• Use jq to trim the trailing , from each input line, interpret the trimmed string as JSON, extract the county name, then output the trimmed JSON strings prepended with the county name and a distinct separator, |.

• Use an awk command to split each line into the prepended county name and the trimmed JSON string, which allows awk to easily construct the output filename and write the JSON string to it.

Note: The awk command keeps all output files open until the script has finished, which means that, in your case, 100 output files will be open simultaneously - a number that shouldn't be a problem, however.

In cases where it is a problem, you can use the following variation, in which jq first sorts the lines by county name, which then allows awk to immediately close the previous output field whenever the next county is reached in the input:

jq -Rrn '
  [inputs[:-1]|fromjson] | sort_by(.properties.county)[] | 
    .properties.county + "|" + (.|tostring)
' file | 
   awk -F'|' '
    prevCounty != $1 { if (outFile) close(outFile); outFile = $1 "_bins_2016.json" }
    { print $2 > outFile; prevCounty = $1  }
  '

A simpler version of chepner's answer:

while IFS= read -r line
do 
    countyName=$(jq --raw-output '.properties.county' <<<"${line: : -1}")
    jq <<< "${line: : -1}" >> "$countyName"_bins_2016.json
done<file

The idea is to filter the county name using a jq filter after stripping the , from each line of your input file. Then the line is passed to jq as plain stream to produce a JSON file in prettified format.

If you are from a relatively older version of bash (< 4.0) use "${line%?}" over "${line: : -1}"

For example with the change above, one of your county becomes,

cat ALAMANCE_bins_2016.json
{
  "type": "Feature",
  "properties": {
    "county": "ALAMANCE",
    "vBLA": 0,
    "vWHI": 0,
    "vDEM": 0,
    "vREP": 0,
    "vUNA": 0,
    "vTOT": 0
  },
  "geometry": {
    "type": "Polygon",
    "coordinates": [
      [
        [
          -79.527429,
          35.843303
        ],
        [
          -79.532428,
          35.843303
        ],
        [
          -79.532428,
          35.848302
        ],
        [
          -79.527429,
          35.848302
        ],
        [
          -79.527429,
          35.843303
        ]
      ]
    ]
  }
}

Note: The current solution could be performance intensive as reading file line by line is an expensive operation, and equally invoking jq for each of the lines.

This will do what you want minus getting rid of the last comma:-

gawk 'match($0, /"county":"([^"]+)/, array){ print >array[1]"_bins_2016.json" }' INPUT_FILE

This will output files in the current path with a filename in the format COUNTRY NAME_bins_2016.json.

The script goes line by line and uses a regex to match the exact term "country":" followed by 1 or more characters that aren't a ". It captures the characters within the quotes and then uses it as part of the filename to append the current line to.

To remove the trailing comma from all .json files in the current path you could use:-

sed -i '$ s/,$//' *.json

If you were certain that the last char was always a comma, a faster solution would be to use truncate:-

truncate -s-1 *.json

Last part taken from this answer: https://stackoverflow.com/a/40568723/1453798

Here is a quickie script that will do the job. It has the virtue of working on most systems without having to install any other tools.

IFS=$'\n'
counties=( $( sed 's/^.*"county":"//;s/".*$//' counties.txt ) )
unset IFS

for county in "${!counties[@]}"
do
  county="${counties[$i]}"
  filename="$county".out.txt
  echo "'$filename'"
  grep "\"$county\"" counties.txt > "$filename"
done

The setting of IFS to \n allows the array elements to contain spaces. The sed command strips off all the text up to the start of the county name and all the text after it. The for loop is the form that allows iterating over the array. Finally, the grep command needs to have double quotes around the search string so that counties that are substrings of other counties don't accidentally get put into the wrong file.

See this section of the GNU BASH Reference Manual for more info.