Write it as \left[x^2+(x+1)^2 \right]^n = \sum_{m=0}^n \binom {n}{m} x^{2m} (x+1)^{2(n-m)}
Then it is evident that the coefficient of xr will be
\sum_{m=0}^{\left[\frac{m}{2} \right]} \binom {n}{m} \binom {2n-2m}{r-2m} which is identical to the expression given.