Interpret the given conditions

We are given two $3\times 3$ matrices $M,N$ such that $MN=NM,$ and $M\ne N^2, \qquad M^2=N^4.$

Since $M$ and $N$ commute, $M$ also commutes with every polynomial in $N$. In particular, $MN^2=N^2M.$

Use the relation $M^2=N^4$

Note that $N^4=(N^2)^2.$ Hence $M^2-(N^2)^2=0.$

Because $M$ and $N^2$ commute, we can factor this as $M^2-(N^2)^2=(M-N^2)(M+N^2)=0.$

So, $(M-N^2)(M+N^2)=0.$

Rewrite the matrix in the options

Consider $M^2+MN^2.$ Since $M$ and $N^2$ commute, factor out $M$: $M^2+MN^2=M(M+N^2)=(M+N^2)M.$

Now multiply $(M-N^2)(M+N^2)=0$ on the right by $M$: $(M-N^2)(M+N^2)M=0.$

Using $(M+N^2)M=M^2+MN^2$, we get $(M-N^2)(M^2+MN^2)=0.$

Use the condition $M\ne N^2$

Since $M\ne N^2$, the matrix $U=M-N^2$ is a non-zero $3\times 3$ matrix.

Also, because everything commutes, $(M^2+MN^2)(M-N^2)=M(M+N^2)(M-N^2)=M(M^2-N^4)=M(0)=0.$

Thus there exists a non-zero matrix $U=M-N^2$ such that $(M^2+MN^2)U=0.$

Therefore, Option B is true.

Conclude about determinant

Let $A=M^2+MN^2.$ We found a non-zero matrix $U$ such that $AU=0.$

If $A$ were invertible, then multiplying by $A^{-1}$ would give $U=0$, contradiction. Hence $A$ is not invertible, so $\det(A)=0.$

Therefore, $\det(M^2+MN^2)=0.$

So Option A is true.

Check Options C and D

• Option C: $\det(M^2+MN^2)\ge 1$

  But we found $\det(M^2+MN^2)=0,$ so this is false.

• Option D: If $(M^2+MN^2)U=0$, then $U=0$

  This is exactly the statement that $M^2+MN^2$ is invertible. But it is not invertible, and we already exhibited a non-zero matrix $U=M-N^2$ with $(M^2+MN^2)U=0.$ Hence D is false.

Final answer

Correct options are: $\boxed{A,\ B}$