1. Rewrite the functional equation

Given $f(x)f\left(\frac1x\right)=f(x)+f\left(\frac1x\right).$

Bring all terms to one side: $f(x)f\left(\frac1x\right)-f(x)-f\left(\frac1x\right)=0.$

Add $1$ on both sides: $\bigl(f(x)-1\bigr)\bigl(f(1/x)-1\bigr)=1.$

So if we define $g(x)=f(x)-1,$ then $g(x)g(1/x)=1.$

Also, since $f:\mathbb R\setminus\{0\}\to(-\infty,1)$, we have $f(x)<1\quad \forall x\ne 0,$ so $g(x)=f(x)-1<0\quad \forall x\ne 0.$

2. Use the fact that $f$ is a quadratic polynomial

Let $f(x)=ax^2+bx+c.$ Then $g(x)=ax^2+bx+(c-1).$

Now $g(1/x)=\frac{a}{x^2}+\frac{b}{x}+(c-1).$

Given $g(x)g(1/x)=1 \quad \forall x\ne 0.$ Multiply both sides by $x^2$: $\bigl(ax^2+bx+(c-1)\bigr)\bigl(a+b x+(c-1)x^2\bigr)=x^2.$

This identity is quite restrictive. A simpler way is to note that since $g(x)$ is a polynomial and $g(x)g(1/x)=1$ for all $x\ne 0$, the only polynomial possibilities are monomials of the form $g(x)=\lambda x^n.$ But because $g(x)$ is a polynomial of degree $2$ and also negative for all $x\ne 0$, the only feasible constant-multiple monomial consistent with degree at most $2$ and the identity is actually a nonzero constant with $\lambda^2=1.$ So $g(x)=\pm 1.$ Since $g(x)<0$ for all $x\ne 0$, we must have $g(x)=-1.$ Therefore $f(x)=g(x)+1=0,$ which is not a quadratic polynomial. So we need to carefully use the range condition and the quadratic form directly.

3. Direct coefficient comparison

Let $f(x)=ax^2+bx+c.$ Then $f(1/x)=\frac{a}{x^2}+\frac{b}{x}+c.$

Substitute into $f(x)f(1/x)=f(x)+f(1/x).$

Expand the left-hand side: $\left(ax^2+bx+c\right)\left(\frac{a}{x^2}+\frac{b}{x}+c\right).$

Multiplying term by term, $=a^2+b^2+c^2+ab\left(x+\frac1x\right)+ac\left(x^2+\frac1{x^2}\right)+bc\left(x+\frac1x\right).$

So $f(x)f(1/x)=a^2+b^2+c^2+(ab+bc)\left(x+\frac1x\right)+ac\left(x^2+\frac1{x^2}\right).$

Now $f(x)+f(1/x)=a\left(x^2+\frac1{x^2}\right)+b\left(x+\frac1x\right)+2c.$

Equating coefficients of the independent expressions $x^2+1/x^2$, $x+1/x$, and constant term:

• Coefficient of $x^2+1/x^2$: $ac=a$ $a(c-1)=0$

• Coefficient of $x+1/x$: $ab+bc=b$ $b(a+c-1)=0$

• Constant term: $a^2+b^2+c^2=2c.$

Since $f$ is degree $2$, we must have $a\ne 0.$ Hence from $a(c-1)=0$, $c=1.$

Then the second condition becomes $b(a+1-1)=ba=0.$ Since $a\ne 0$, we get $b=0.$

Now the constant-term equation gives $a^2+0+1=2$ $a^2=1$ $a=\pm 1.$

Thus possible quadratic polynomials are $f(x)=x^2+1 \quad \text{or} \quad f(x)=1-x^2.$

But the range is given as $f(x)\in(-\infty,1) \quad \forall x\ne 0.$

• For $f(x)=x^2+1$, we have $f(x)>1$ for all $x\ne 0$, impossible.
• For $f(x)=1-x^2$, we have $f(x)<1$ for all $x\ne 0$, valid.

Therefore, $\boxed{f(x)=1-x^2}.$

4. Use the condition $f(K)=-2K$

Substitute $f(x)=1-x^2$: $1-K^2=-2K.$

Rearrange: $K^2-2K-1=0.$

Solve: $K=\frac{2\pm\sqrt{4+4}}{2}=1\pm\sqrt{2}.$

So the possible values are $K_1=1+\sqrt2, \qquad K_2=1-\sqrt2.$

5. Find the sum of squares

We need $K_1^2+K_2^2.$

Using $K_1+K_2=2, \qquad K_1K_2=-1,$ we get $K_1^2+K_2^2=(K_1+K_2)^2-2K_1K_2=2^2-2(-1)=4+2=6.$

Hence the required sum is $\boxed{6}.$

So the correct option is C.