i don't think this is in syllabus
This is a case of rotation of axes without shifting the origin
S=ax + 2hxy + by
Let the axes be rotated through θ
then x=x'cosθ + y'sinθ
and y=x'sinθ + y'cosθ
where (x',y') are new coordinates
so S'=a(x'cosθ + y'sinθ) +2h(x'cosθ + y'sinθ)(x'sinθ + y'cosθ) +b(x'sinθ + y'cosθ)
Now a'=coff. of x'2=a cos2θ+2hsinθcosθ+bsin2θ
h'=(1/2)coff. of x'y' =-a sinθcosθ+h cos2θ-h sin2θ+b sinθcosθ
b'=coff. of y'2 =a sin2θ-2hsinθcosθ+bcos2θ
Now a'+b'=a(cos2θ+sin2θ) +b(cos2θ+sin2θ)
=a+b
Similarly u can prove h'2-a'b'=h2-ab