1. Interpret the notation

Here $[x^2]$ denotes the greatest integer function (floor), i.e. the largest integer less than or equal to $x^2$.

We need to evaluate

$$I=\int_0^2 [x^2] \, dx.$$

2. Find where $[x^2]$ changes value

As $x$ goes from $0$ to $2$, $x^2$ goes from $0$ to $4$. So $[x^2]$ will take the values $0,1,2,3,4$, changing when $x^2$ crosses integers.

The breakpoints are:

$$x=0,\ 1,\ \sqrt{2},\ \sqrt{3},\ 2.$$

Now determine the floor value on each interval:

• For $0 \le x < 1$, we have $0 \le x^2 < 1$, so $[x^2]=0$.
• For $1 \le x < \sqrt2$, we have $1 \le x^2 < 2$, so $[x^2]=1$.
• For $\sqrt2 \le x < \sqrt3$, we have $2 \le x^2 < 3$, so $[x^2]=2$.
• For $\sqrt3 \le x < 2$, we have $3 \le x^2 < 4$, so $[x^2]=3$.

At $x=2$, $[x^2]=4$, but a single point does not affect the integral.

3. Split the integral interval-wise

Thus,

$$I=\int_0^1 0\,dx+\int_1^{\sqrt2}1\,dx+\int_{\sqrt2}^{\sqrt3}2\,dx+\int_{\sqrt3}^2 3\,dx.$$

4. Evaluate each part

$$\int_0^1 0\,dx=0$$ $$\int_1^{\sqrt2}1\,dx=\sqrt2-1$$ $$\int_{\sqrt2}^{\sqrt3}2\,dx=2(\sqrt3-\sqrt2)$$ $$\int_{\sqrt3}^2 3\,dx=3(2-\sqrt3)=6-3\sqrt3$$

5. Add them

So,

$$I=(\sqrt2-1)+2(\sqrt3-\sqrt2)+6-3\sqrt3.$$

Expand:

$$I=\sqrt2-1+2\sqrt3-2\sqrt2+6-3\sqrt3.$$

Combine like terms:

$$I=5-\sqrt2-\sqrt3.$$

Therefore,

$$\boxed{\int_0^2 [x^2]dx=5-\sqrt2-\sqrt3}.$$

6. Match with the options

Option D is

$$-\sqrt2-\sqrt3+5 = 5-\sqrt2-\sqrt3.$$

So the correct option is D.