You're receiving notifications only when you open the App because your mailbox doesn't support Exchange ActiveSync. Therefore, your mails are set to fetch periodically.
So what happens is that at a set interval, your phone will query your mailbox to see if there are new messages.
What email provider are you using?
You're recommended to use their Applications instead.