Interpret the given equation

The question states: $10\int_1^x f(t)\,dt = 5x f(x) - x^5 - 9 \qquad (x\ge 1)$

(The printed upper limit appears to be a typo; it must be $x$, otherwise the left side would always be $0$ and the problem would be inconsistent with the options.)

Differentiate both sides

Using the Fundamental Theorem of Calculus, $\frac{d}{dx}\left(10\int_1^x f(t)\,dt\right)=10f(x).$

Differentiate the right-hand side: $\frac{d}{dx}\left(5x f(x)-x^5-9\right)=5f(x)+5x f'(x)-5x^4.$

Therefore, $10f(x)=5f(x)+5x f'(x)-5x^4.$

Divide by $5$: $2f(x)=f(x)+x f'(x)-x^4,$ so $x f'(x)=f(x)+x^4.$

Hence, $f'(x)-\frac1x f(x)=x^3.$

Solve the differential equation

This is a linear differential equation: $f'(x)-\frac1x f(x)=x^3.$

The integrating factor is $\mathrm{IF}=e^{\int -1/x\,dx}=e^{-\ln x}=\frac1x.$

Multiply throughout by $\frac1x$: $\frac1x f'(x)-\frac1{x^2}f(x)=x^2.$

Left side is $\frac{d}{dx}\left(\frac{f(x)}x\right)=x^2.$

Integrate: $\frac{f(x)}x=\int x^2\,dx=\frac{x^3}{3}+C.$

Thus, $f(x)=\frac{x^4}{3}+Cx.$

Use the original equation to find $C$

Put $x=1$ in the given relation: $10\int_1^1 f(t)\,dt = 5\cdot 1\cdot f(1)-1^5-9.$

Since the integral from $1$ to $1$ is $0$, $0=5f(1)-10,$ so $f(1)=2.$

From $f(x)=\frac{x^4}{3}+Cx,$ we get $f(1)=\frac13+C=2.$

Therefore, $C=\frac53.$

Hence, $f(x)=\frac{x^4}{3}+\frac{5x}{3}.$

Compute $f(3)$

$f(3)=\frac{3^4}{3}+\frac{5\cdot 3}{3}=\frac{81}{3}+5=27+5=32.$

Check options

$f(3)=32$ So the correct option is C.