APL(NARS), 389 chars

r←f w;a;b;i;l;c;t;p;q;np;G
G←⍬
w←,w⋄r←G⍳⊂w⋄→4×⍳r≤≢G⋄G,←⊂w⋄→6
r←0⋄→0
r←1⋄→0
r←2⋄→0
i←1⋄r←a←b←''⋄l←≢w⋄np←1⋄p←q←''
→3×⍳i>l⋄a←w[i]⋄i+←1⋄→10×⍳a='('
→3×⍳i>l⋄c←w[i]⋄i+←1
→11×⍳∼c='('⋄p←'('⋄q←')'
→3×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→12×⍳np=0⋄b,←c⋄→10
b←,c
t←b⋄→14×⍳a='('⋄→5×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→5×⍳(a='T')∧'ST'≡b
t←{a='N':''⋄a='S':b⋄a='D':b,p,b,q⋄a='T':b,p,b,q,p,b,q⋄'Fail'}
w←t,(i-1)↓w⋄→2

// +/ 27 4 30 7 7 7 30 31 20 24 59 5 61 62 15=389

f function has as input one string of chars, and return 0 for program terminate, 1 if there is a loop with a repeat state, 2 if the loop has the conseguence of out of memory for the program in a future time.

f function puts in the global array G, the state each input, that would be the string array in w in each loop inside f function. If some state is repeated than retun 1 because there is detected an infinite loop. If else function terminate it return 0. Else return 2 in the case it finds strings TT or T(ST) at beginning of w variable.

    r←f w;a;b;i;l;c;t;p;q;np;G
1:  G←⍬
2:  w←,w⋄r←G⍳⊂w⋄→4×⍳r≤≢G⋄G,←⊂w⋄→6
3:  r←0⋄→0
4:  r←1⋄→0
5:  r←2⋄→0
6:  i←1⋄r←a←b←''⋄l←≢w⋄np←1⋄p←q←''
7:  →3×⍳i>l⋄a←w[i]⋄i+←1⋄→10×⍳a='('
8:  →3×⍳i>l⋄c←w[i]⋄i+←1
9:  →11×⍳∼c='('⋄p←'('⋄q←')'
10:     →3×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→12×⍳np=0⋄b,←c⋄→10
11: b←,c
12: t←b⋄→14×⍳a='('⋄→5×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→5×⍳(a='T')∧'ST'≡b
13: t←{a='N':''⋄a='S':b⋄a='D':b,p,b,q⋄a='T':b,p,b,q,p,b,q⋄'Fail'}
14: w←t,(i-1)↓w⋄→2

APL(NARS), 380 chars

r←f w;a;b;i;l;c;t;p;q;np;G
G←⍬
w←,w⋄r←G⍳⊂w⋄→4×⍳r≤≢G⋄G,←⊂w⋄→6
r←0⋄→0
r←1⋄→0
r←2⋄→0
np←i←1⋄p←q←r←a←b←''⋄l←≢w
→3×⍳i>l⋄a←w[i]⋄i+←1⋄→10×⍳a='('
→3×⍳i>l⋄c←w[i]⋄i+←1
→12×⍳∼c='('
p←'('⋄q←')'
→3×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c=p⋄np-←c=q⋄→13×⍳np=0⋄b,←c⋄→11
b←,c
t←b⋄→15×⍳a='('⋄→5×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→5×⍳(a='T')∧'ST'≡b
t←{a='N':''⋄a='S':b⋄a='D':b,p,b,q⋄a='T':b,p,b,q,p,b,q⋄'Fail'}
w←t,(i-1)↓w⋄→2

// +/ 27 4 30 7 7 7 25 31 20 12 12 55 5 61 62 15=380

f function has as input one string of chars, and return 0 for function terminate, 1 if function meets a repeat state (this means a loop), 2 if the function find a loop with no repeat state, this means that if f is not stopped it will go of out of memory in a future time.

f function puts in an array G, the state, that is each input, that would be the string array in w in each loop inside f function. If some state is repeated (find the w is already in G) than retun 1 because there is detected an infinite loop. If else function terminate it return 0. Else return 2 in the case it finds in the start of w the strings TT or T(ST) that it seems generate always differents states tha will have as effect the go out of memory in the future. I don't know if these string are the only one that generate always differents states.

    r←f w;a;b;i;l;c;t;p;q;np;G
 1: G←⍬
 2: w←,w⋄r←G⍳⊂w⋄→4×⍳r≤≢G⋄G,←⊂w⋄→6
 3: r←0⋄→0
 4: r←1⋄→0
 5: r←2⋄→0
 6: np←i←1⋄p←q←r←a←b←''⋄l←≢w
 7: →3×⍳i>l⋄a←w[i]⋄i+←1⋄→10×⍳a='('
 8: →3×⍳i>l⋄c←w[i]⋄i+←1
 9: →12×⍳∼c='('
10: p←'('⋄q←')'
11: →3×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c=p⋄np-←c=q⋄→13×⍳np=0⋄b,←c⋄→11
12: b←,c
13: t←b⋄→15×⍳a='('⋄→5×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→5×⍳(a='T')∧'ST'≡b
14: t←{a='N':''⋄a='S':b⋄a='D':b,p,b,q⋄a='T':b,p,b,q,p,b,q⋄'Fail'}
15: w←t,(i-1)↓w⋄→2

APL(NARS), 503 chars

Garray←⍬

r←f w;a;b;z;i;l;c;t;p;q;np
w←,w⋄r←Garray⍳⊂w⋄→3×⍳r≤≢Garray⋄Garray,←⊂w⋄→5
r←0⋄→0
r←1⋄→0
r←2⋄→0
i←1⋄r←a←b←''⋄l←≢w
→2×⍳i>l⋄a←w[i]⋄i+←1⋄np←1⋄→14×⍳a='('
→2×⍳i>l⋄c←w[i]⋄i+←1⋄p←q←''
→10×⍳∼c='('⋄p←'('⋄q←')'
    →2×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→11×⍳np=0⋄b,←c⋄→9
b←,c
z←(i-1)↓w⋄→4×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→4×⍳(a='T')∧'ST'≡b
t←a{⍺='N':''⋄⍺='S':⍵⋄⍺='D':⍵,p,⍵,q⋄⍺='T':⍵,p,⍵,q,p,⍵,q⋄'Fail'}b
w←t,z⋄→1
b←''
→2×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→16×⍳np=0⋄b,←c⋄→15
w←b,(i-1)↓w⋄→1

r←g w
Garray←⍬⋄r←f w

// +/ 9 27 45 7 7 7 18 36 27 24 62 5 56 64 9 5 59 15 6 15=503

It puts in the global array Garray, the state each input, that would be the string array in w in each loop. If some state is repeated than retun 1 because there is a infinite loop. If program terminate return 0. Else return 2 in the case it finds strings TT or T(ST) in the start of the array w.

  g 'T(NTDS)T'
0
  g 'S(TSD(TSD))'
1
  g 'ST(ST)'
2
  g¨('N')('NT')('SS')('S(TNT)')('D(D(NN)N)')
0 0 0 0 0 
  g¨('TN')('TS')('TNT')('T(NTDS)T')('T(T(T(NTDS)))')
0 0 0 0 0 
  g¨('DD')('D(SD)')('TDD')('D(NSDD)')
1 1 1 1 
  g¨('DD(TT)')('T(TN)')('S(TSD(TSD))')
1 0 1 
  g¨('TT')('ST(ST)')('NTTT')('NT(DT)')('ST(ST)N')('TTD')('D(T(TT)T)NSNS')
2 2 2 2 2 2 2 
  g 'TTD'
2

APL(NARS), 389 chars

r←f w;a;b;i;l;c;t;p;q;np;G
G←⍬
w←,w⋄r←G⍳⊂w⋄→4×⍳r≤≢G⋄G,←⊂w⋄→6
r←0⋄→0
r←1⋄→0
r←2⋄→0
i←1⋄r←a←b←''⋄l←≢w⋄np←1⋄p←q←''
→3×⍳i>l⋄a←w[i]⋄i+←1⋄→10×⍳a='('
→3×⍳i>l⋄c←w[i]⋄i+←1
→11×⍳∼c='('⋄p←'('⋄q←')'
→3×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→12×⍳np=0⋄b,←c⋄→10
b←,c
t←b⋄→14×⍳a='('⋄→5×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→5×⍳(a='T')∧'ST'≡b
t←{a='N':''⋄a='S':b⋄a='D':b,p,b,q⋄a='T':b,p,b,q,p,b,q⋄'Fail'}
w←t,(i-1)↓w⋄→2

// +/ 27 4 30 7 7 7 30 31 20 24 59 5 61 62 15=389

f function has as input one string of chars, and return 0 for program terminate, 1 if there is a loop with a repeat state, 2 if the loop has the conseguence of out of memory for the program in a future time.

f function puts in the global array G, the state each input, that would be the string array in w in each loop inside f function. If some state is repeated than retun 1 because there is detected an infinite loop. If else function terminate it return 0. Else return 2 in the case it finds strings TT or T(ST) at beginning of w variable.

  f¨('N')('NT')('SS')('S(TNT)')('D(D(NN)N)')
0 0 0 0 0 
  f¨('TN')('TS')('TNT')('T(NTDS)T')('T(T(T(NTDS)))')
0 0 0 0 0 
  f¨('DD')('D(SD)')('TDD')('D(NSDD)')
1 1 1 1 
  f¨('DD(TT)')('T(TN)')('S(TSD(TSD))')
1 0 1 
  f¨('TT')('ST(ST)')('NTTT')('NT(DT)')('ST(ST)N')('TTD')('D(T(TT)T)NSNS')
2 2 2 2 2 2 2 
  f 'TTD'
2
  

f ungolfed with label numbers

    r←f w;a;b;i;l;c;t;p;q;np;G
1:  G←⍬
2:  w←,w⋄r←G⍳⊂w⋄→4×⍳r≤≢G⋄G,←⊂w⋄→6
3:  r←0⋄→0
4:  r←1⋄→0
5:  r←2⋄→0
6:  i←1⋄r←a←b←''⋄l←≢w⋄np←1⋄p←q←''
7:  →3×⍳i>l⋄a←w[i]⋄i+←1⋄→10×⍳a='('
8:  →3×⍳i>l⋄c←w[i]⋄i+←1
9:  →11×⍳∼c='('⋄p←'('⋄q←')'
10:     →3×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→12×⍳np=0⋄b,←c⋄→10
11: b←,c
12: t←b⋄→14×⍳a='('⋄→5×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→5×⍳(a='T')∧'ST'≡b
13: t←{a='N':''⋄a='S':b⋄a='D':b,p,b,q⋄a='T':b,p,b,q,p,b,q⋄'Fail'}
14: w←t,(i-1)↓w⋄→2

APL(NARS), 503 chars

Garray←⍬

r←f w;a;b;z;i;l;c;t;p;q;np
w←,w⋄r←Garray⍳⊂w⋄→3×⍳r≤≢Garray⋄Garray,←⊂w⋄→5
r←0⋄→0
r←1⋄→0
r←2⋄→0
i←1⋄r←a←b←''⋄l←≢w
→2×⍳i>l⋄a←w[i]⋄i+←1⋄np←1⋄→14×⍳a='('
→2×⍳i>l⋄c←w[i]⋄i+←1⋄p←q←''
→10×⍳∼c='('⋄p←'('⋄q←')'
    →2×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→11×⍳np=0⋄b,←c⋄→9
b←,c
z←(i-1)↓w⋄→4×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→4×⍳(a='T')∧'ST'≡b
t←a{⍺='N':''⋄⍺='S':⍵⋄⍺='D':⍵,p,⍵,q⋄⍺='T':⍵,p,⍵,q,p,⍵,q⋄'Fail'}b
w←t,z⋄→1
b←''
→2×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→16×⍳np=0⋄b,←c⋄→15
w←b,(i-1)↓w⋄→1

r←g w
Garray←⍬⋄r←f w

// +/ 9 27 45 7 7 7 18 36 27 24 62 5 56 64 9 5 59 15 6 15=503

It puts in the global array Garray, the state each input, that would be the string array in w in each loop. If some state is repeated than retun 1 because there is a infinite loop. If program terminate return 0. Else return 2 in the case it finds strings TT or T(ST)

The only test fail for 'T(TN)' that here result 0 instead than 1.

Here expand 'T(TN)' as

T(TN) TN(TN)(TN) NNN(TN)(TN) N(TN)(TN) (TN) TN NNN N

test:

  g 'T(NTDS)T'
0
  g 'S(TSD(TSD))'
1
  g 'ST(ST)'
2
  g¨('N')('NT')('SS')('S(TNT)')('D(D(NN)N)')
0 0 0 0 0 
  g¨('TN')('TS')('TNT')('T(NTDS)T')('T(T(T(NTDS)))')
0 0 0 0 0 
  g¨('DD')('D(SD)')('TDD')('D(NSDD)')
1 1 1 1 
  g¨('DD(TT)')('T(TN)')('S(TSD(TSD))')
1 0 1 
  g¨('TT')('ST(ST)')('NTTT')('NT(DT)')('ST(ST)N')('TTD')('D(T(TT)T)NSNS')
2 2 2 2 2 2 2 
  g 'TTD'
2

APL(NARS), 503 chars

Garray←⍬

r←f w;a;b;z;i;l;c;t;p;q;np
w←,w⋄r←Garray⍳⊂w⋄→3×⍳r≤≢Garray⋄Garray,←⊂w⋄→5
r←0⋄→0
r←1⋄→0
r←2⋄→0
i←1⋄r←a←b←''⋄l←≢w
→2×⍳i>l⋄a←w[i]⋄i+←1⋄np←1⋄→14×⍳a='('
→2×⍳i>l⋄c←w[i]⋄i+←1⋄p←q←''
→10×⍳∼c='('⋄p←'('⋄q←')'
    →2×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→11×⍳np=0⋄b,←c⋄→9
b←,c
z←(i-1)↓w⋄→4×⍳(np=1)∧(a='T')∧'T'=↑,b⋄→4×⍳(a='T')∧'ST'≡b
t←a{⍺='N':''⋄⍺='S':⍵⋄⍺='D':⍵,p,⍵,q⋄⍺='T':⍵,p,⍵,q,p,⍵,q⋄'Fail'}b
w←t,z⋄→1
b←''
→2×⍳i>l⋄c←w[i]⋄i+←1⋄np+←c='('⋄np-←c=')'⋄→16×⍳np=0⋄b,←c⋄→15
w←b,(i-1)↓w⋄→1

r←g w
Garray←⍬⋄r←f w

// +/ 9 27 45 7 7 7 18 36 27 24 62 5 56 64 9 5 59 15 6 15=503

It puts in the global array Garray, the state each input, that would be the string array in w in each loop. If some state is repeated than retun 1 because there is a infinite loop. If program terminate return 0. Else return 2 in the case it finds strings TT or T(ST) in the start of the array w.

The only test fail for 'T(TN)' that here result 0 instead than 1.

Here expand 'T(TN)' as

T(TN) TN(TN)(TN) NNN(TN)(TN) N(TN)(TN) (TN) TN NNN N

test:

  g 'T(NTDS)T'
0
  g 'S(TSD(TSD))'
1
  g 'ST(ST)'
2
  g¨('N')('NT')('SS')('S(TNT)')('D(D(NN)N)')
0 0 0 0 0 
  g¨('TN')('TS')('TNT')('T(NTDS)T')('T(T(T(NTDS)))')
0 0 0 0 0 
  g¨('DD')('D(SD)')('TDD')('D(NSDD)')
1 1 1 1 
  g¨('DD(TT)')('T(TN)')('S(TSD(TSD))')
1 0 1 
  g¨('TT')('ST(ST)')('NTTT')('NT(DT)')('ST(ST)N')('TTD')('D(T(TT)T)NSNS')
2 2 2 2 2 2 2 
  g 'TTD'
2