Interpret the notation

Here $A'$ denotes the transpose of $A$. So the given condition is $A^T=\alpha A+I,$ where $\alpha\in \mathbb{R}\setminus\{-1,1\}$.

Write a general $2\times 2$ matrix

Let $A=\begin{pmatrix}a&b\\ c&d\end{pmatrix}.$ Then $A^T=\begin{pmatrix}a&c\\ b&d\end{pmatrix},\qquad \alpha A+I=\begin{pmatrix}\alpha a+1&\alpha b\\ \alpha c&\alpha d+1\end{pmatrix}.$

Equating corresponding entries from $A^T=\alpha A+I$: \begin{align*} a&=\alpha a+1,\ d&=\alpha d+1,\ c&=\alpha b,\ b&=\alpha c. \end{align*}

Solve these relations

From the first two equations, $a(1-\alpha)=1,\qquad d(1-\alpha)=1.$ Since $\alpha\ne 1$, $a=d=\frac{1}{1-\alpha}.$

From the other two equations, $c=\alpha b,\qquad b=\alpha c.$ Substituting $c=\alpha b$ into $b=\alpha c$ gives $b=\alpha(\alpha b)=\alpha^2 b,$ so $b(1-\alpha^2)=0.$ Since $\alpha\ne \pm 1$, we get $b=0,$ hence $c=\alpha b=0.$

Therefore, $A=\frac{1}{1-\alpha}I.$

Compute $A^2-A$

Let $k=\frac{1}{1-\alpha}.$ Then $A=kI$, so $A^2-A=(k^2-k)I.$

Hence, $\det(A^2-A)=(k^2-k)^2.$

Given that $\det(A^2-A)=4,$ therefore $(k^2-k)^2=4.$ So $k^2-k=\pm 2.$

Solve for $k$

• If $k^2-k=2$, then $k^2-k-2=0$ $\Rightarrow (k-2)(k+1)=0$ $\Rightarrow k=2 \text{ or } k=-1.$

• If $k^2-k=-2$, then $k^2-k+2=0,$ whose discriminant is $1-8=-7<0,$ so no real solution.

Thus real values are $k=2,-1.$

Convert back to $\alpha$

Since $k=\frac{1}{1-\alpha},$ we get $\alpha=1-\frac{1}{k}.$

• For $k=2$: $\alpha=1-\frac12=\frac12.$
• For $k=-1$: $\alpha=1-(-1)=2.$

So possible values of $\alpha$ are $\frac12,\ 2.$

Find their sum

$\frac12+2=\frac52.$

Option check

The correct option is $\boxed{\frac52}$ i.e. Option D.