Based on personal--if not overly informed--experience, I'm pretty certain you (aka the "outside user") cannot control how much memory SQL allocates to CLR processes.
Further information that may or may not help here: there are limts, ratios, and (our big headache) fragmentation of memory assigned over time (that's days of regular use). Our issues could only address by stopping and restarting the SQL service. Again, I'm pretty certain that it doesn't matter how much memory is available on the box, so much as the internal means by which SQL addresses and allocates it. The problems that we were having back when were tangled, confused, recurrent, and very irritating... and then, based on my research, we upgraded to 64-bit edition (SQL 2008), which has very different means of addressing and allocating all that memory we had installed on the box. All of our problems went away, and I haven't had to consider the situation ever since.