problem in using awk

Your quoting problem is because once you start a back-quoted command, it continues until the next back-quote. This is your code as shown above, except I've removed the blank lines.

a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F"U" '\
{
var2=`echo $var1 | awk -F"a"
{print " "$2}'`
}\
fi

(Back-quotes '`' are hard to show in in-line Markdown.)

The line var1= line starts a back-quoted expression, which stops at the next unescaped back-quote, which is the one after var2=. It then reads the rest of that line, and on the following line, encounters a single quote. When it looks for the following single quote, there is none - so it reports an error. You can demonstrate this is what goes on in steps:

a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F"U" '\
{
var2=\`echo $var1 | awk -F"a"
{print " "$2}'`
}\
fi

The script above has an escape (backslash) before the back-quote after var2=, so now the command in back-quotes extends to the back-quote after the print line. This still isn't valid shell; the line with }\ combines with the fi to make a command name }fi, so you still get an unexpected EOF error - because the fi for the end of the if is missing. Modify the script again:

a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F"U" '\
{
var2=\`echo $var1 | awk -F"a"
{print " "$2}'`
#}\
fi

This comments out the close brace, and the shell script is now 'valid'; it is awk's turn to start complaining about the invalid script it is given.

++ awk -FU '{
var2=`echo $var1 | awk -F"a"
{print " "$2}'
awk: syntax error at source line 2
 context is
     >>> var2=` <<< 
awk: illegal statement at source line 2
awk: illegal statement at source line 2
    missing }

Other people have given you roughly what you need as an answer. I'd probably use Perl to do the splitting up (and I suspect I could lose the intermediate array @x if I spent enough time on it, producing a script of line noise):

a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]
then var1=$(echo $a | perl -ne '@x=split /[aU]/; print "$x[1]\n"')
fi

However, you can also do it in one line with awk, thus:

a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]
then var1=$(echo $a | awk -Fa '{sub("ULL","",$2); print $2}')
fi

This splits the input on the 'a' instead of the 'U'; it then removes the 'ULL' from the second field and prints it. If you want to split on 'U', then you use:

a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]
then var1=$(echo $a | awk -FU '{sub("[0-9]+a", "", $1); print $1}')
fi

The regular expression in the sub is marginally more complex this way.

Not sure exactly what you are trying to do, but this slight re-write printed out the middle part of a (which is I think what you wanted)

> cat moo.sh
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
        var1=`echo $a | awk -F"U" '{print $1}'`
        var2=`echo $var1 | awk -F"a" '{print " "$2}'`
        echo $var2
fi
> sh moo.sh
1504606846976

You can't do shell commands inside awk except to make system calls. without you telling us what you are trying to achieve

#!/bin/bash
a=115292a1504606846976ULL
b=2
case "$b" in
 2 ) 
    a=${a%U*}
    echo ${a#*a} # I assume you want to get 1504606846976
esac

a=115292a1504606846976ULL
b=2

if [ "$b" = "2" ]; then
  var1=`echo $a | awk -F "U" '{print $1}' | awk -F "a" '{print $2}'`
fi