Interpret the setup

Since the system is at rest and the plane is smooth, the inclined plane exerts only a normal reaction on the block of mass $m_1=5\,\text{kg}$.

The figure is the standard arrangement where the $5\,\text{kg}$ block is on a smooth incline and the $3\,\text{kg}$ block hangs vertically.

Find the tension in the string

The hanging mass $m_2=3\,\text{kg}$ is at rest, so its forces balance:

$T = m_2 g = 3 \times 10 = 30\,\text{N}$

For block $m_1$ on the incline

Along the incline, since the block is at rest:

$T = m_1 g \sin\theta$

So,

$30 = 5 \times 10 \sin\theta$ $30 = 50\sin\theta$ $\sin\theta = \frac{3}{5}$

Hence,

$\cos\theta = \frac{4}{5}$

Find the force exerted by the plane on $m_1$

On a smooth incline, the plane exerts only the normal reaction $N$:

$N = m_1 g \cos\theta$

$N = 5 \times 10 \times \frac{4}{5} = 40\,\text{N}$

Check options

• A: $30\,\text{N}$
• B: $40\,\text{N}$
• C: $50\,\text{N}$
• D: $60\,\text{N}$

Therefore, the correct option is B.