1. Let $A=\sin^{-1}\left(\frac{x+1}{\sqrt{x^2+2x+2}}\right),\qquad B=\sin^{-1}\left(\frac{x}{\sqrt{x^2+1}}\right).$ The given equation is $A-B=\frac{\pi}{4}.$

2. Interpret each inverse sine term as an angle.

Notice that $x^2+2x+2=(x+1)^2+1.$ So if we put $t=x+1$, then $\frac{x+1}{\sqrt{x^2+2x+2}}=\frac{t}{\sqrt{t^2+1}}.$ A standard identity is: $\sin^{-1}\left(\frac{u}{\sqrt{1+u^2}}\right)=\tan^{-1}(u)$ for all real $u$, since $\tan^{-1}(u)\in(-\pi/2,\pi/2)$.

Hence, $A=\tan^{-1}(x+1),\qquad B=\tan^{-1}(x).$ So the equation becomes $\tan^{-1}(x+1)-\tan^{-1}(x)=\frac{\pi}{4}.$

3. Use tangent of difference.

Since both angles lie in $(-\pi/2,\pi/2)$, we can take tangent: $\tan\left(\tan^{-1}(x+1)-\tan^{-1}(x)\right)=\tan\frac{\pi}{4}=1.$ Thus, $\frac{(x+1)-x}{1+x(x+1)}=1$ which gives $\frac{1}{x^2+x+1}=1.$ So, $x^2+x+1=1 \implies x^2+x=0 \implies x(x+1)=0.$ Therefore, $S=\{0,-1\}.$

4. Now evaluate $\sum_{x\in S}\left(\sin\left((x^2+x+5)\frac{\pi}{2}\right)-\cos\left((x^2+x+5)\pi\right)\right).$

For both $x=0$ and $x=-1$, $x^2+x=0,$ so $x^2+x+5=5.$ Hence each term equals $\sin\left(5\cdot\frac{\pi}{2}\right)-\cos(5\pi).$ Now,