Let the eigenvalues of the $2\times 2$ matrix $A$ be the roots of $|A-xI|=0.$ These are given as $-1$ and $3$.

The sum of the diagonal elements of a matrix is its trace. We need $\operatorname{tr}(A^2).$

If the eigenvalues of $A$ are $\lambda_1$ and $\lambda_2$, then the eigenvalues of $A^2$ are $\lambda_1^2$ and $\lambda_2^2$.

Here, $\lambda_1=-1,\qquad \lambda_2=3.$

So the eigenvalues of $A^2$ are $(-1)^2=1, \qquad 3^2=9.$

The trace of a matrix equals the sum of its eigenvalues. Therefore, $\operatorname{tr}(A^2)=1+9=10.$

Hence, the sum of the diagonal elements of $A^2$ is $\boxed{10}.$