Given family of lines

$x(3\lambda+1)+y(7\lambda+2)=17\lambda+5$

Rearranging in terms of $\lambda$:

$\lambda(3x+7y-17)+(x+2y-5)=0$

Since this represents a family of lines all passing through a fixed point $P$, that point must satisfy this for all $\lambda$.

Find the common point $P$

For the equation to hold for all $\lambda$, both coefficients must vanish:

$3x+7y-17=0$ $x+2y-5=0$

Solve:

From $x+2y=5$, we get $x=5-2y$

Substitute into the first:

$3(5-2y)+7y=17$ $15-6y+7y=17$ $y=2$

Then $x=5-4=1$

So, $P=(1,2)$

Interpret the family geometrically

Since all lines of the family pass through $P=(1,2)$, the family is the set of all lines through $P$.

Among all lines through a fixed point $P$, the line farthest from the origin is the one perpendicular to the line joining the origin to $P$.

The maximum distance from origin equals $OP$.

Equation of the farthest line $L$

Slope of $OP$ is $\frac{2-0}{1-0}=2$

So slope of line perpendicular to $OP$ is $-\frac{1}{2}$

Line through $(1,2)$ with slope $-\frac12$:

$y-2=-\frac12(x-1)$

Multiply by 2:

$2y-4=-x+1$ $x+2y-5=0$

Thus the required line is $L: x+2y-5=0$

(Indeed this corresponds to $\lambda=0$ in the family.)

Distance of $L$ from point $(3,6)$

Distance from point $(x_1,y_1)$ to line $ax+by+c=0$ is

$\frac{|ax_1+by_1+c|}{\sqrt{a^2+b^2}}$

For $L: x+2y-5=0$, we have $a=1,b=2,c=-5$.

So distance from $(3,6)$ is

$d=\frac{|3+2\cdot 6-5|}{\sqrt{1^2+2^2}}$ $d=\frac{|3+12-5|}{\sqrt5}$ $d=\frac{10}{\sqrt5}=2\sqrt5$

Hence,

$d^2=(2\sqrt5)^2=20$

Compare with stored answer

Derived answer is 20, which matches Option B.