1. Let $A=\int_0^n \{x\}\,dx, \qquad B=\int_0^n [x] \,dx$ and the third term be $C=10(n^2-n), \qquad n\in \mathbb N,\ n>1.$

Since these are three consecutive terms of a G.P., we must have $B^2=AC.$

2. Compute $\displaystyle \int_0^n \{x\}\,dx$.

For integer $n$, on each interval $[k,k+1)$, $\{x\}=x-k.$ So $\int_k^{k+1} \{x\}\,dx=\int_k^{k+1}(x-k)\,dx=\int_0^1 t\,dt=\frac12.$ There are $n$ such unit intervals from $0$ to $n$, hence $A=\int_0^n \{x\}\,dx=\frac n2.$

3. Compute $\displaystyle \int_0^n [x] \,dx$.

For $x\in[k,k+1)$, $[x]=k.$ Thus $\int_k^{k+1}[x]\,dx=\int_k^{k+1}k\,dx=k.$ Summing from $k=0$ to $n-1$, $B=\int_0^n [x]\,dx=\sum_{k=0}^{n-1}k=\frac{n(n-1)}2.$

4. Use the G.P. condition: $B^2=AC.$ Substitute: $\left(\frac{n(n-1)}2\right)^2=\left(\frac n2\right)\cdot 10(n^2-n).$ Since $n^2-n=n(n-1)$, $\frac{n^2(n-1)^2}{4}=\frac n2\cdot 10n(n-1)=5n^2(n-1).$ For $n>1$, divide by $n^2(n-1)$: $\frac{n-1}{4}=5.$ So $n-1=20 \implies n=21.$

5. Final answer: $\boxed{21}$

This matches the stored correct answer.