Tricky one.. Try this

anyone needs a hint?

Apply Lebnitz formula for differentiation under integration at first...

The result: The given integral equals
\dfrac{\tan^{-1}a}{2}\ \ln(1+a^2)

put x=\tan \theta \Rightarrow dx=\sec ^{2}\theta d\theta

At x=0, \theta=0 and at x=a, \theta=\tan^{-1} a

Substituting the above value, We get,\int_{0}^{\tan^{-1} a}{\frac{\log (1+a\tan \theta )}{\sec ^{2}\theta }}\times \sec ^{2}\theta d\theta

I=\int_{0}^{\tan^{-1} a}{\log (1+a\tan \theta )} d\theta

=\int_{0}^{\tan^{-1} a}{\log (1+a\tan(\tan^{-1} a-) )} d\theta

=\int_{0}^{\tan^{-1} a}{\log \left\{1+a \left( \frac{a-\tan \theta }{1+a\tan \theta \right)}\right\} } d\theta

=\int_{0}^{\tan^{-1} a}{\log (\frac{1+a^{2}}{1+a\tan \theta }) } d\theta

=\int_{0}^{\tan^{-1} a}{\log (1+a^{2})}d\theta-\int_{0}^{\tan^{-1} a}{\log(1+a\tan \theta )}d\theta

\Rightarrow 2I=\int_{0}^{\tan^{-1} a}{\log \left(1+a^{2} \right)}d\theta

\Rightarrow 2I=\tan^{-1} a\times \log (1+a^{2})

\Rightarrow I=\frac{\tan^{-1} a}{2}\times \log (1+a^{2})

Excellent solution.

I had done this problem using Lebnitz formula, but now it seems that I had worked too much on it. Here's mine:

I(a)=\int_{o}^{a}{\frac{ln(1+ax)}{1+x^2}dx}

\frac{\partial I}{\partial x}=\int_{0}^{a}{\frac{x}{(1+x^2)(1+ax)}dx}+\frac{ln(1+a^2)}{1+a^2}

=\int_{0}^{a}{\frac{1}{a^2+1}\left(\frac{x+a}{x^2+1} \right)-\frac{a}{a^2+1}\left(\frac{1}{1+ax} \right)dx} \: +\frac{ln(1+a^2)}{1+a^2}

=\frac{atan^{-1}a}{a^2+1}+\frac{ln(a^2+1)}{2(a^2+1)}

I(a)=\int{\frac{atan^{-1}a}{a^2+1}+\frac{ln(a^2+1)}{2(a^2+1)}da}

Take a=tan\theta

So,

I=\int {ln(sec\theta)+\theta tan\theta d\theta}

=\theta ln(sec\theta)-\int {\theta tan\theta d\theta}+-\int {\theta tan\theta d\theta}+C \\\\=tan^{-1}a.ln(\sqrt{1+a^2})+C \\\\ I(a=0)=0\Rightarrow C=0 \\\\I(a)=tan^{-1}a.ln(\sqrt{1+a^2})