For every ‘left’ bracket you also need to add a ‘right’ bracket, otherwise it gives an error.
For the curly brackets you have to use an extra \ before the bracket, for all other brackets it is not needed.
$$
\left\{ \sum_{n=1}^{\infty} \left( 1 - \frac{x^2}{n^2 \pi^2} \right) + \left[ \int_a^b x dx \right] \right\}
$$