minimum distance

The curvex^2+y^2=9%2x^2 + 10y^2 + 6xy = 1 is a circle centered at the origin having radius = 3.
The curve 2x^2 + 10y^2 + 6xy = 1, represents an ellipse whose center is at the origin. However, the axes of the ellipse is rotated w.r.t. the x- and y- axes. If we could determine the major-axis of the ellipse then the minimum distance between the circle and the ellipse will be the difference of the circle's radius and the semi-major axes of the ellipse.
In order to obtain the semi-axes, we first find the directions along which the axes of the ellipse lie. The direction of the semi-axes of the central conic
ax^2+2hxy+by^2+c=0
are given by the angle \theta which are obtained by the relation
\tan2\theta = \dfrac{2h}{a-b}
In our case, this angle \theta is obtained by
\tan2\theta = \dfrac{2h}{a-b}=\dfrac{6}{2-10}=-\dfrac{3}{4}
Hence,
\dfrac{2\tan\theta}{1-\tan^2\theta}=-\dfrac{3}{4}
Solving for \tan\theta gives
two values \tan\theta_1=3 and \tan\theta_2=-\dfrac{1}{3}.
These will be the directions along which the semi-axes would lie.
In order to obtain the lengths, we transform the equation to polar coordinates by the transformation
x=r\cos\theta,\quad y=r\sin\theta.
The equation of the ellipse then becomes
2r^2\cos^2\theta + 10r^2\sin^2\theta + 6r^2\sin\theta\cos\theta = 1
That is,
r^2(2\cos^2\theta + 10\sin^2\theta + 6\sin\theta\cos\theta) = \sin^2\theta +\cos^2\theta
i.e.
r^2(2 + 10\tan^2\theta + 6\tan\theta) = 1 +\tan^2\theta
and ultimately
r^2 = \dfrac{1 +\tan^2\theta}{2 + 10\tan^2\theta + 6\tan\theta}
When \tan\theta=\tan\theta_1=3, we get
r_1^2=\dfrac{1+9}{2+10\times 9 +6\times 3}=\dfrac{1}{11}
while when \tan\theta =\tan\theta_2=-\dfrac{1}{3}, we get
r_2^2=\dfrac{1+\frac{1}{9}}{2+10\times\frac{1}{9}-\frac{6}{3}}=1.
Hence, the semi-axes are 1 and \dfrac{1}{\sqrt{11}}
Obviously, the semi-major axes is 1. As such the minimum distance between the circle and the ellipse = 3 - 1 =2