1. Find the center and radius of the circle

The circle is $x^2+y^2-6x+8y-103=0.$

Complete the squares: $x^2-6x+y^2+8y=103$ $x^2-6x+9+y^2+8y+16=103+9+16$ $(x-3)^2+(y+4)^2=128.$

So the center is $C(3,-4)$ and the radius is $r=\sqrt{128}=8\sqrt{2}.$

2. Use the condition that the square has sides parallel to the axes

If a square is inscribed in a circle and its sides are parallel to the coordinate axes, then its center is the same as the center of the circle.

Let the vertices of the square be $(3\pm a,",-4\pm a).$

The distance from the center to any vertex equals the radius: $\sqrt{a^2+a^2}=r$ $a\sqrt{2}=8\sqrt{2}$ $a=8.$

Hence the four vertices are: $(3+8,-4+8)=(11,4),$ $(3+8,-4-8)=(11,-12),$ $(3-8,-4+8)=(-5,4),$ $(3-8,-4-8)=(-5,-12).$

3. Find the distance of each vertex from the origin

Distance of $(11,4)$ from origin: $\sqrt{11^2+4^2}=\sqrt{121+16}=\sqrt{137}.$

Distance of $(11,-12)$ from origin: $\sqrt{11^2+(-12)^2}=\sqrt{121+144}=\sqrt{265}.$

Distance of $(-5,4)$ from origin: $\sqrt{(-5)^2+4^2}=\sqrt{25+16}=\sqrt{41}.$

Distance of $(-5,-12)$ from origin: $\sqrt{(-5)^2+(-12)^2}=\sqrt{25+144}=13.$

The smallest of these is $\sqrt{41}.$

4. Conclusion

The vertex nearest to the origin is $(-5,4)$, and its distance from the origin is $\boxed{\sqrt{41}}.$