1. Given vector

$\vec a = -\hat i + 2\hat j + \hat k = (-1,2,1)$

We are told that $\vec a$ is rotated through a right angle, and in doing so it passes through the y-axis. So the rotation is in the plane containing $\vec a$ and the $y$-axis, through $90^\circ$, such that the final vector $\vec b$ lies on the other side of the $y$-axis.

2. Find the magnitude of $\vec a$

$|\vec a|=\sqrt{(-1)^2+2^2+1^2}=\sqrt{1+4+1}=\sqrt6$

Since rotation preserves magnitude,

$|\vec b|=\sqrt6$

3. Use the y-axis as the intermediate direction

The unit vector along the $y$-axis is $\hat j=(0,1,0)$.

Now,

$\vec a\cdot \hat j = 2$

So the angle $\theta$ between $\vec a$ and the $y$-axis satisfies

$\cos\theta=\frac{2}{\sqrt6}$

Since the vector is rotated through $90^\circ$ and passes through the $y$-axis in the process, the $y$-axis bisects the angle between $\vec a$ and $\vec b$. Thus $\vec b$ is the reflection of $\vec a$ about the $y$-axis in the plane of rotation.

4. Find $\vec b$ by reflection about the y-axis

Resolve $\vec a$ into components parallel and perpendicular to $\hat j$:

• Parallel component: $\vec a_{\parallel} = 2\hat j = (0,2,0)$

• Perpendicular component: $\vec a_{\perp}=\vec a-\vec a_{\parallel}=(-1,0,1)$

Reflection about the $y$-axis keeps the parallel component same and reverses the perpendicular component:

$\vec b=\vec a_{\parallel}-\vec a_{\perp}=(0,2,0)-(-1,0,1)=(1,2,-1)$

So,

$\vec b=\hat i+2\hat j-\hat k$

Check: $\vec a\cdot \vec b = (-1)(1)+2\cdot2+1(-1)= -1+4-1=2$

Also, $|\vec a||\vec b|\cos 90^\circ =0$

This seems inconsistent with a $90^\circ$ angle between $\vec a$ and $\vec b$, so let us interpret the statement more carefully.

5. Correct interpretation of rotation through right angle

The vector is rotated by $90^\circ$ in such a way that during rotation it passes through the $y$-axis. Hence the angle from $\vec a$ to the $y$-axis plus the angle from the $y$-axis to $\vec b$ is $90^\circ$.

Let the angle between $\vec a$ and $\hat j$ be $\theta$, then angle between $\vec b$ and $\hat j$ is $90^\circ-\theta$.

Since rotation is in the plane of $\vec a$ and $\hat j$, let us construct an orthonormal basis in that plane.

Take $\hat e_1=\hat j=(0,1,0)$

The component of $\vec a$ perpendicular to $\hat j$ is $(-1,0,1)$ with magnitude $\sqrt2$, so $\hat e_2=\frac{1}{\sqrt2}(-1,0,1)$

Then $\vec a=2\hat e_1+\sqrt2\hat e_2$

Now rotating by $90^\circ$ in this plane gives $\vec b=-\sqrt2\hat e_1+2\hat e_2$

or the opposite orientation $\vec b=\sqrt2\hat e_1-2\hat e_2$

We choose the one that passes through the positive $y$-axis while rotating from $\vec a$; this is

$\vec b=\sqrt2\hat e_1-2\hat e_2$

So $\vec b=\sqrt2(0,1,0)-2\cdot \frac{1}{\sqrt2}(-1,0,1)$ $\vec b=(\sqrt2,\sqrt2,-\sqrt2)$

This gives $\vec a\cdot \vec b = (-1)(\sqrt2)+2(\sqrt2)+1(-\sqrt2)=0$

So this is correct.

6. Compute $3\vec a+\sqrt2\vec b$

First, $3\vec a=3(-1,2,1)=(-3,6,3)$

Also, $\sqrt2\vec b=\sqrt2(\sqrt2,\sqrt2,-\sqrt2)=(2,2,-2)$

Thus, $3\vec a+\sqrt2\vec b = (-3,6,3)+(2,2,-2)=(-1,8,1)$

7. Projection on $\vec c=5\hat i+4\hat j+3\hat k=(5,4,3)$

Scalar projection of vector $\vec v$ on $\vec c$ is

$\text{proj}_{\vec c}(\vec v)=\frac{\vec v\cdot \vec c}{|\vec c|}$

Here $\vec v=(-1,8,1)$.

Compute dot product:

$\vec v\cdot \vec c = (-1)(5)+8(4)+1(3)=-5+32+3=30$

Now, $|\vec c|=\sqrt{5^2+4^2+3^2}=\sqrt{25+16+9}=\sqrt{50}=5\sqrt2$

Therefore projection is

$\frac{30}{5\sqrt2}=\frac{6}{\sqrt2}=3\sqrt2$

8. Compare with options

$3\sqrt2$

So the correct option is D.