Given data

The circle has center $z_0=\frac{1}{2}(1+3i)=\frac12+\frac32 i$ and radius $r=1.$

Also, $z_1=1+i.$

We are given that $z_0,z_1,z_2$ are collinear and $|z_1-z_0|\,|z_2-z_0|=1.$ Since $z_2$ is outside the circle, we must have $|z_2-z_0|>1.$

Compute $|z_1-z_0|$

$z_1-z_0=\left(1-\frac12\right)+\left(1-\frac32\right)i=\frac12-\frac12 i.$ Hence,

Find $|z_2-z_0|$

From $|z_1-z_0|\,|z_2-z_0|=1,$ we get $\frac{1}{\sqrt2}\,|z_2-z_0|=1$ so $|z_2-z_0|=\sqrt2.$

This is indeed greater than $1$, so $z_2$ is outside the circle.

Use collinearity

Since $z_0,z_1,z_2$ are collinear, the point $z_2$ lies on the line through $z_0$ and $z_1$.

First note the direction from $z_0$ to $z_1$: $z_1-z_0=\frac12-\frac12 i.$ Its length is $\frac1{\sqrt2}$.

A unit vector in this direction is \frac{z_1-z_0}{|z_1-z_0|}=rac{\frac12-\frac12 i}{1/\sqrt2}=\frac{1-i}{\sqrt2}.

Since $|z_2-z_0|=\sqrt2$, the two possible points on this line are $z_2=z_0\pm \sqrt2\cdot \frac{1-i}{\sqrt2}=z_0\pm(1-i).$

Therefore, $z_2=\left(\frac12+\frac32 i\right)+(1-i)=\frac32+\frac12 i$ or $z_2=\left(\frac12+\frac32 i\right)-(1-i)=-\frac12+\frac52 i.$

Compute $|z_2|^2$ for both cases

• For $z_2=\frac32+\frac12 i$, $|z_2|^2=\left(\frac32\right)^2+\left(\frac12\right)^2=\frac94+\frac14=\frac{10}{4}=\frac52.$

• For $z_2=-\frac12+\frac52 i$, $|z_2|^2=\left(-\frac12\right)^2+\left(\frac52\right)^2=\frac14+\frac{25}{4}=\frac{26}{4}=\frac{13}{2}.$

Smaller value

Hence the smaller value of $|z_2|^2$ is $\boxed{\frac52}.$

Option check

$\frac52$ corresponds to Option B.