1. Given vectors

$\vec a=\hat i+2\hat j+3\hat k=(1,2,3),\qquad \vec b=3\hat i+\hat j-\hat k=(3,1,-1)$

Since $\vec c$ is coplanar with $\vec a$ and $\vec b$, we can write

$\vec c=x\vec a+y\vec b$

for some scalars $x,y$.

2. Use perpendicular condition

The question states that $\vec c$ is perpendicular to $\vec b$, so

$\vec c\cdot \vec b=0$

Now,

$\vec a\cdot \vec b=1\cdot 3+2\cdot 1+3\cdot(-1)=3+2-3=2$

and

$\vec b\cdot \vec b=3^2+1^2+(-1)^2=9+1+1=11$

Thus,

$\vec c\cdot \vec b=(x\vec a+y\vec b)\cdot \vec b=x(\vec a\cdot \vec b)+y(\vec b\cdot \vec b)=2x+11y=0$

So,

$y=-\frac{2x}{11}$

3. Use the condition $\vec a\cdot \vec c=5$

Also,

$\vec a\cdot \vec a=1^2+2^2+3^2=14$

Hence,

$\vec a\cdot \vec c=\vec a\cdot (x\vec a+y\vec b)=x(\vec a\cdot \vec a)+y(\vec a\cdot \vec b)=14x+2y=5$

Substitute $y=-\frac{2x}{11}$:

$14x+2\left(-\frac{2x}{11}\right)=5$

$14x-\frac{4x}{11}=5$

$\frac{154x-4x}{11}=5$

$\frac{150x}{11}=5$

$x=\frac{11}{30}$

Then

$y=-\frac{2}{11}\cdot \frac{11}{30}=-\frac{1}{15}$

So,

$\vec c=\frac{11}{30}\vec a-\frac{1}{15}\vec b$

4. Find $\vec c$ explicitly

$\frac{11}{30}\vec a=\left(\frac{11}{30},\frac{11}{15},\frac{11}{10}\right)$

$\frac{1}{15}\vec b=\left(\frac{1}{5},\frac{1}{15},-\frac{1}{15}\right)$

Therefore,

$\vec c=\left(\frac{11}{30}-\frac{1}{5},\frac{11}{15}-\frac{1}{15},\frac{11}{10}-\left(-\frac{1}{15}\right)\right)$

$\vec c=\left(\frac{11}{30}-\frac{6}{30},\frac{10}{15},\frac{33}{30}+\frac{2}{30}\right)$

$\vec c=\left(\frac{1}{6},\frac{2}{3},\frac{7}{6}\right)$

5. Compute magnitude

$|\vec c|=\sqrt{\left(\frac{1}{6}\right)^2+\left(\frac{2}{3}\right)^2+\left(\frac{7}{6}\right)^2}$

$=\sqrt{\frac{1}{36}+\frac{4}{9}+\frac{49}{36}}$

$=\sqrt{\frac{1+16+49}{36}}$

$=\sqrt{\frac{66}{36}}$

$=\sqrt{\frac{11}{6}}$

6. Compare with options

Thus,

$|\vec c|=\sqrt{\frac{11}{6}}$

So the correct option is A.