We need to count the number of $2\times 2$ matrices $A=\begin{bmatrix}a_{11}&a_{12}\\ a_{21}&a_{22}\end{bmatrix}$ with each entry in $\{0,1,2,3,4\}$ such that the sum of all four entries is a prime number in $(2,13)$.

1. Possible prime sums

Primes in the interval $(2,13)$ are $3,5,7,11.$ So we must count the number of ordered quadruples $(a_{11},a_{12},a_{21},a_{22}), \qquad a_{ij}\in\{0,1,2,3,4\}$ with total sum equal to $3,5,7,$ or $11$.

Thus the problem reduces to finding the coefficient of $x^3+x^5+x^7+x^{11}$ in $(1+x+x^2+x^3+x^4)^4.$

We will count each sum separately.

2. Count solutions for sum $3$

We need the number of nonnegative integer solutions of $x_1+x_2+x_3+x_4=3,$ with each $x_i\le 4$.

Since the sum is only $3$, the upper bound $\le 4$ is automatically satisfied.

Number of solutions: $\binom{3+4-1}{4-1}=\binom{6}{3}=20.$

So, $N(3)=20.$

3. Count solutions for sum $5$

We need the number of solutions of $x_1+x_2+x_3+x_4=5, \qquad 0\le x_i\le 4.$

First count all nonnegative solutions: $\binom{5+4-1}{3}=\binom{8}{3}=56.$

Now subtract those where some variable is at least $5$.

If $x_1\ge 5$, write $x_1'=x_1-5\ge 0$. Then $x_1'+x_2+x_3+x_4=0,$ which has exactly $1$ solution. Similarly for each of the 4 variables.

No two variables can both be at least $5$ because the total sum is $5$.

Hence $N(5)=56-4=52.$

4. Count solutions for sum $7$

We need the number of solutions of $x_1+x_2+x_3+x_4=7, \qquad 0\le x_i\le 4.$

Total nonnegative solutions: $\binom{7+4-1}{3}=\binom{10}{3}=120.$

Subtract solutions with some $x_i\ge 5$.

For a fixed variable, say $x_1\ge 5$, let $x_1'=x_1-5$. Then $x_1'+x_2+x_3+x_4=2.$ Number of solutions: $\binom{2+4-1}{3}=\binom{5}{3}=10.$ There are 4 choices of variable, so subtract $4\cdot 10=40.$

Now check intersections: if two variables are each at least $5$, their sum is at least $10>7$, impossible. So no intersections occur.

Thus $N(7)=120-40=80.$

5. Count solutions for sum $11$

We need the number of solutions of $x_1+x_2+x_3+x_4=11, \qquad 0\le x_i\le 4.$

Total nonnegative solutions: $\binom{11+4-1}{3}=\binom{14}{3}=364.$

Subtract solutions with at least one variable $\ge 5$

For a fixed variable, say $x_1\ge 5$, let $x_1'=x_1-5$. Then $x_1'+x_2+x_3+x_4=6.$ Number of solutions: $\binom{6+4-1}{3}=\binom{9}{3}=84.$ With 4 choices: $4\cdot 84=336.$

Add back intersections of two variables each $\ge 5$

Suppose $x_1\ge 5$ and $x_2\ge 5$. Put $x_1'=x_1-5,\quad x_2'=x_2-5.$ Then $x_1'+x_2'+x_3+x_4=1.$ Number of solutions: $\binom{1+4-1}{3}=\binom{4}{3}=4.$ Number of pairs of variables: $\binom{4}{2}=6.$ So add back $6\cdot 4=24.$

Triple intersections

If three variables are each at least $5$, the sum is at least $15>11$, impossible. So no further terms.

Hence $N(11)=364-336+24=52.$

6. Total count

Therefore the required number of matrices is $N(3)+N(5)+N(7)+N(11)=20+52+80+52=204.$

7. Comparison with stored answer

Our derived answer is $\boxed{204}$ which matches the stored correct answer.