JavaScript (ES6), 52 bytes

f=s=>s>(s=s.replace(/\|(.*?[\d)])\|/,"($1)"))?f(s):s

Try it online!

Method

At each iteration, we look for the first substring consisting of:

• \| a leading pipe
• .*? the shortest possible string
• [\d)] either a digit or a closing parenthesis
• \| a trailing pipe

and replace the pipes with ( and ) respectively.

If a replacement occurs, the new string is less than the original string in lexicographical order, because the leading pipe is turned into an opening parenthesis. Hence the test s > (s = s.replace(...)) which triggers a recursive call if true.

NB: A safer way would be to use [^|]* instead of .*?. If you find a test case where .*? fails, please let me know.

Retina 0.8.2, 22 16 bytes

T`|`)`\d\|+
\|
(

Try it online! Link includes test cases. Explanation:

T`|`)`\d\|+

Replace all |s that appear after digits with )s.

\|
(

Replace the remaining |s with (s.

K (ngn/k), 27 22 bytes

Based on Arnauld's answer, but without regex.

{`c$(84-0=\"0|"'x)!'x}

Try it online!

Any closing parenthesis in the output directly follows a digit or another closing parenthesis, and none of the opening ones do. This means we can replace all consecutive |'s following a digit by closing parentheses. The remaining |'s are replaced by opening ones.

K (ngn/k), 36 bytes

My first idea, tests all ways of replacing | with parentheses using try/eval.

{c@*<.[.:;;`]','c:+`c$(83+!2&x)!''x}

Try it online!

It is not very easy to show this is correct, as (1+) is a valid expression in K. However after toying around with this for quite some time, I'm convinced that you can't construct a valid input where this actually becomes a problem

Charcoal, 16 bytes

ＦＳ≡ι|§)(‹ψ0«ι≔ιψ

Try it online! Link is to verbose version of code. Explanation:

ＦＳ≡ι

Loop over the input characters.

|§)(‹ψ0

If the current characters is a | then output a ) or ( depending on whether the last non-| was less than 0.

«ι≔ιψ

Otherwise output the character and save it as the last non-|.

JavaScript (Node.js), 44 bytes

s=>s.replace(d=/./g,c=>c>{}?++d?')':'(':d=c)

Try it online!

Look last non-pipe character before each pipe. Converting pipe to ) only when last non-pipe is a digit, and converting to ( otherwise. Does not work with the optional additional requirement.

05AB1E, 13 bytes

ε'|Qižu®dèëy©

Port of @Neil's Charcoal answer, so make sure to upvote that answer as well!

Outputs as a list of characters.

Try it online or verify all test cases.

Explanation:

ε           # Map over the characters of the (implicit) input:
 '|Qi      '#  If the current character is a "|":
     žu     #   Push builtin string "()<>[]{}"
       ®    #   Push variable `®` (which is -1 by default)
        d   #   Check whether it's a non-negative (>=0) number
         è  #   Use that check (0 or 1) to (0-based) index into string "()<>[]{}"
    ë       #  Else:
     y      #   Simply push the current character
      ©     #   And store it in variable `®` (without popping)
            # (after which the mapped list of characters is output implicitly as result)

Haskell + hgl, 26 bytes

gkx"(/d(/|{)_)+|/|{(_|.)+"

This solution uses a "regex" to replace each of the |s with either ( or ).

A closing paren must be preceded by a valid statement, and valid statements can only end in either digits, or a closing paren. Meanwhile a open paren can never come after a digit or a closing paren, so this classification is complete.

We translate these rules to the "regex":

(                  )+  -- at least once
 /d                    -- a digit
   (     )+            -- at least one of
    /|                 -- a |
      {)_              -- replaced with )
           |           -- or
            /|         -- a |
              {(_      -- replaced with (
                 |     -- or 
                  .    -- any character

Without "regex", 36 bytes

gkY$dgS<>sO(hh"|"")")++hh"|""("++hdS

This is the same idea as the algorithm above, but implemented using parser combinators.

This is considerably longer, as parser combinators tend to be at a disadvantage to the "regex" when it comes to challenges involving the specific behavior of specific characters.

36 bytes

rA"|""("<gkY(dgS<>sO(hh"|"")")++hdS)

This is another solution which is also the same length, but which uses a mix of parser and non-parser functions.

Reflection

Although these solutions don't exactly feel good, there is remarkably little I see in the way of improvement.

Regex

There's very little that can be done to improve the "regex" based solution. I have no plans to change the syntax of "regex" in the near future, and to be honest I don't even know what I would do.

• The only thing I can think of is to make a combined version of gkx and gkY. This would pull the (...)+ out and save 3 bytes.

Non-regex

I have more thoughts here, but even with all my suggestions this doesn't come close to catching up with the regex.

• The time has probably come for a combined version of <> and sO.
• hh"|"")" is annoyingly long. There are two ways to solve this:
  • Simplest is that like other characters we could just add a shortcut to hh"|". Of course next time we are going to want to do something like hh"@""*" and this will have been completely useless, but if we keep adding these eventually we will get there.
  • A more principled solution would be to implement something like xx but taking a single string argument and unconsing it to get two arguments. e.g. xxh(x:xs)=xx x xs. This would allow it to be xxh"|)" which saves only a byte.