Ich benutzte die Definition
$$f\in \mathcal O(g) \Leftrightarrow \limsup_{n\to \infty}\left|\frac{f(n)}{g(n)}\right|< \infty$$
(a) - falsch
$$f(n) = \frac 1n \Rightarrow \frac{\frac 1n}{\frac 1{n^2}}= n \stackrel{n\to\infty}{\longrightarrow} \infty$$
(b) - richtig
$$\frac{n\log n}{n^{1+\varepsilon}} = \frac{\log n}{n^{\varepsilon}}\stackrel{\text{L'Hospital}}{\sim}\frac{\frac 1n}{\varepsilon n^{\varepsilon-1}}=\frac 1{\varepsilon n^{\varepsilon}}\stackrel{n\to\infty}{\longrightarrow} 0 <\infty$$
(c) - falsch
$$f(n) = 2n,\: g(n) = n \Rightarrow f\in \mathcal O(g)$$
$$\frac{2^{2n}}{2^n} = 2^n \stackrel{n\to\infty}{\longrightarrow} \infty$$