You could use the function below. For the input given in the question:

p[a,b,c,d]|q[small,large]|r[small,large]|s|t[w[x,y],z[a,b,c]]

...it produces this object:

{
  "p": [
    "a",
    "b",
    "c",
    "d"
  ],
  "q": [
    "small",
    "large"
  ],
  "r": [
    "small",
    "large"
  ],
  "s": true,
  "t": {
    "w": [
      "x",
      "y"
    ],
    "z": [
      "a",
      "b",
      "c"
    ]
  }
}

function toObject(x) {
    // Turn custom format into JSON text format, and then parse it.
    // In that object, find nested objects that could be turned into array.
    return (function flagsToArray(obj) {
        // Collect keys with nested objects.
        var nested = Object.keys(obj).filter(key => obj[key] !== true);
        // For those, call this function recursively
        nested.forEach(key => obj[key] = flagsToArray(obj[key]));
        // If no nesting, then turn this into an array
        return nested.length ? obj : Object.keys(obj);
    })(JSON.parse('{' + 
        x.replace(/\|/g, ',') // treat '|' as ','
         .replace(/"/g, '\"') // escape any double quotes
         .replace(/([^,|\[\]]+)/g, '"$1"') // wrap terms in double quotes
         .replace(/"\[/g, '":[') // insert colon for assignment of arrays
         .replace(/"([,\]])/g, '":true$1') // insert `true` assignment for atomic term
         .replace(/\[/g, "{").replace(/\]/g, "}") // replace array notation with object notation
        + '}'));
}

// Sample input
var x = 'p[a,b,c,d]|q[small,large]|r[small,large]|s|t[w[x,y],z[a,b,c]]';
// Convert
var obj = toObject(x);
// Output
console.log(obj);

The function makes several replacements to convert the custom format into a JSON text format, turning everything into nested objects (no arrays). Then in a second process, a recursive one, objects are identified that have no nested objects, i.e. they only consist of members with true as value. Those objects are then replaced by their array "equivalent", i.e. the array with the object's keys.

Your query string is essentially a flat representation of a tree whose nodes are defined by:

• either a name alone
• or a name and a list of child nodes

Note that I don't see any obvious difference between , and |, so I'm going to assume that they actually have the same meaning.

You can't easily store this structure by using only arrays, and it would also be unnecessarily complicated to use a mix of arrays and objects.

Therefore, I'd suggest to use only objects with the following conventions:

• key = name of node
• value = either true¹ or a child object

1 _{This is a placeholder. You may also consider using an empty object.}

With these assumptions, your example string "abc|pqr[abc,xyz[abc,def]]" would be decoded as:

tree = {
  "abc": true,
  "pqr": {
    "abc": true,
    "xyz": {
      "abc": true,
      "def": true
    }
  }
}

Such a structure is quite easy to manipulate.

For instance, if you'd like to get the child nodes of root > pqr > xyz, you could do:

Object.keys(tree.pqr.xyz)

which will return:

["abc", "def"]

Implementation

Below is a possible implementation:

function parse(query) {
  var n, tree = {}, node = tree, stk = [],
      sym = '', sz = (query += ',').length;

  for(n = 0; n < sz; n++) {
    switch(query[n]) {
      case '|':
      case ',':
        sym && (node[sym] = true);
        break;

      case '[':
        stk.push(node);
        node = node[sym] = {};
        break;

      case ']':
        sym && (node[sym] = true);
        node = stk.pop();
        break;

      default:
        sym += query[n];
        continue;
    }
    sym = '';
  }
  return tree;
}

console.log(parse("abc|pqr[abc,xyz[abc,def]]"));