You are close, but you are confused in your use of return from a function in bash. The return is an integer, not a string. To accomplish what you are trying to do, simply printf or echo $count at the end of the function:
#!/bin/bash
cot() {
local count=0
local fname="${1:-test1}"
while read -r line
do
((count++))
done <"$fname"
printf $count
}
a=$(cot)
printf "lines read from test1: %s\n" $a
You can also use the local keyword before variable names in your function to eliminate ambiguity and the potential for name collisions and insure the variables are unset when the function returns. In the above function, you can pass the function any filename to read and return the line count of (the default if no filename is given is test1)
Examples
$ wc -l <test1 ## confirm test1 is a 94 line file
94
$ bash count_cot.sh ## calling above function
lines read from test1: 94
To read the line count of any file passed as the first argument to the script, simply pass $1 to cot:
a=$(cot "$1")
name=${1:-test1} ## intentionally placed after call for illustration
printf "lines read from file '%s': %s\n" "$name" $a
Example
$ wc -l <wcount2.sh
62
$ bash count_cot.sh wcount2.sh
lines read from 'wcount2.sh': 62