
Radii of enclosing circle for pentagon, red and green circles being $R $, $ R_1 $ and $R_2$, we have:
$$ R_1 = R \sin{A} $$
$$ 2 R_2 = R + R \cos{A} - R_1 $$
$$ \Rightarrow R_2 = \dfrac {R(1 + \cos{A} - \sin{A})}{2}$$
Looking at the limiting case for $R_2 > R_1 $:
$$ \Rightarrow \dfrac {R(1 + \cos{A} - \sin{A})}{2} > R \sin{A}$$
$$ 1 + \cos{A} - \sin{A} > 2 \sin{A} $$
$$ 1 + \cos{A} > 3 \sin{A} $$
$$ \sec{A} + 1 > 3 \tan{A} $$
$$ \text{Squaring:} \;\;\;\;\; \sec^2{A} + 2 \sec{A} + 1 > 9 \tan^2{A} $$
$$ \text{Hence: } \;\;\;\;\; 8 \sec^2{A} - 2 \sec{A} - 10 < 0 $$
$$ \text{Or: } \;\;\;\;\; 4 \sec^2{A} - \sec{A} - 5 < 0 $$
$$ \text{Factorizing:} \;\;\;\;\; (4 \sec{A} - 5)\;(\sec{A} + 1)\; < \;0 $$
For a physical pentagon only a first quadrant angle, i.e. $\sec{A} < \dfrac{5}{4} $, is possible.
Thus $\cos{A} > \dfrac{4}{5} $ and $\sin{A} < \dfrac{3}{5} $. (Very interesting how the 3-4-5 triangle comes into this situation.)
Now the question remaining is whether $ \cos{A} > \dfrac{4}{5} $.
From the regular pentagon's geometry, we have $ A = \dfrac{\pi}{5} = 36^\circ $
Oscar Lanzi says that $ \cos{A} = \dfrac{(1 + \sqrt{5})}{4} $.
Those unfamiliar with the latter expression may explore its derivation in the following spoiler.
[ DIGRESSION . . .
Let the pentagon edge length be $1$.
From the regular pentagon's symmetry:
$$|BEG| \parallel |FA| \;\;\;\; |AEC| \parallel |FB| $$
Hence $AFBE$ is a rhombus with
$$ |BE| = |AF| = |AE| = |FB| = 1 $$
Let pentagon diagonal length = $D$.
Similar triangles, $\triangle ABC$ and $\triangle BCE$:
$$ \implies \dfrac{|AB|}{|BC|} = \dfrac{|BC|}{|CE|} $$
$$ \implies \dfrac{D}{1} = \dfrac{1}{D-1} $$
$$ \implies D^2 - D - 1 = 0 $$
Leaving $ D = \dfrac{(1 + \sqrt{5})}{2} $ as the only positive solution to the ratio of pentagon diagonal to edge length.
$ \dfrac{(1 + \sqrt{5})}{2} $ is referred to as the golden mean, $\phi$.
We also see that $|FI|$ bisects $|AB|$, $|CG|$ and $\angle AEB $.
Thus in $\triangle AHC $:
$$\cos{36^\circ} = \dfrac{|AH|}{|AE|} = \dfrac{\dfrac{D}{2}}{1} $$
$$ \boldsymbol{ \therefore \;\; \cos{36^\circ} \; = \; \frac{1 + \sqrt{5}}{4} } $$
. . . END OF DIGRESSION ]
$$ \text{If} \;\;\; \cos{A} > \dfrac{4}{5} $$
$$ \Rightarrow \dfrac{1 + \sqrt{5}}{4} > \dfrac{4}{5} $$
$$ \Rightarrow \sqrt{5} > \dfrac{11}{5} $$
$$ \text{Squaring, this means that: } \;\;\;\;\; 5 = \dfrac{125}{25} > \dfrac{121}{25} $$
And it is.
So the size of the regular pentagon semi-angle ($\dfrac{\pi}{5}$) means that the green circle will be marginally bigger than the red one.