Try this...

differentiating (given term) wrt a, we get
\int_{0}^{\pi /2}{\frac{-2acos^{2}xdx}{(a^{2}cos^{2}x+b^{2}sin^{2}x)^{2}}} = \frac{-\pi }{2a^{2}b}
\int_{0}^{\pi /2}{\frac{cos^{2}xdx}{(a^{2}cos^{2}x+b^{2}sin^{2}x)^{2}}} = \frac{\pi }{4a^{3}b}.............................(4)
similarly differentiating (3) wrt b we get
\int_{0}^{\pi /2}{\frac{sin^{2}xdx}{(a^{2}cos^{2}x+b^{2}sin^{2}x)^{2}}} = \frac{\pi }{4ab^{3}}.............................(5)
adding (4) and (5) we get
\int_{0}^{\pi /2}{\frac{dx}{(a^{2}cos^{2}x+b^{2}sin^{2}x)^{2}}} = \frac{\pi(a^{2}+b^{2}) }{4a^{3}b^{3}}

hence we can see from the one which we required to find the value of that a = 2 and b = 3

hence value required = 13\pi4 x 8 x 27 = 13\pi864

That's correct.

Just this, actually this qsn came a few yrs back in an ISI MCQ test.

its frm fiitjee GMP (modified one)