1. Given vectors

$\vec a=\hat i+\hat j+\sqrt2\hat k=(1,1,\sqrt2)$ $\vec b=b_1\hat i+b_2\hat j+\sqrt2\hat k=(b_1,b_2,\sqrt2)$ $\vec c=5\hat i+\hat j+\sqrt2\hat k=(5,1,\sqrt2)$

2. Use the projection condition

The projection vector of $\vec b$ on $\vec a$ is given by

$\operatorname{proj}_{\vec a}\vec b=\frac{\vec b\cdot \vec a}{|\vec a|^2}\vec a$

It is given that this projection vector equals $\vec a$. Hence,

$\frac{\vec b\cdot \vec a}{|\vec a|^2}\vec a=\vec a$

Since $\vec a\neq \vec 0$, we get

$\frac{\vec b\cdot \vec a}{|\vec a|^2}=1$

So,

$\vec b\cdot \vec a=|\vec a|^2$

Now,

$|\vec a|^2=1^2+1^2+(\sqrt2)^2=1+1+2=4$

Also,

$\vec b\cdot \vec a=b_1+b_2+2$

Therefore,

$b_1+b_2+2=4$ $b_1+b_2=2 \qquad (1)$

3. Use the perpendicularity condition

Given that $\vec a+\vec b$ is perpendicular to $\vec c$, so

$(\vec a+\vec b)\cdot \vec c=0$

Now,

$\vec a+\vec b=(1+b_1,1+b_2,2\sqrt2)$

Thus,

$(1+b_1,1+b_2,2\sqrt2)\cdot (5,1,\sqrt2)=0$

$5(1+b_1)+(1+b_2)+2\sqrt2\cdot \sqrt2=0$

$5+5b_1+1+b_2+4=0$

$5b_1+b_2+10=0$ $5b_1+b_2=-10 \qquad (2)$

4. Solve equations (1) and (2)

From (1):

$b_2=2-b_1$

Substitute into (2):

$5b_1+(2-b_1)=-10$ $4b_1+2=-10$ $4b_1=-12$ $b_1=-3$

Then,

$b_2=2-(-3)=5$

So,

$\vec b=(-3,5,\sqrt2)$

5. Find the magnitude of $\vec b$

$|\vec b|=\sqrt{(-3)^2+5^2+(\sqrt2)^2}$ $|\vec b|=\sqrt{9+25+2}$ $|\vec b|=\sqrt{36}=6$

6. Check options

The correct option is:

$\boxed{6}$

So, Option B is correct.