I don't know what this graph of yours looks like, but I'm guessing it would be boron and oxygen. If those elements are the exceptions, the reasoning has to do with their outer electron shells. With a larger radius, it is easier for these elements to lose electrons from their outer shell. However, both boron and oxygen lose an electron from the 2p orbital when an electron is initially taken from them. In each case, the resulting elements from the removal of the first electron make the element more stable than it was previously. Boron empties its 2p shell, so beryllium's full 2s shell keeps it stable. When oxygen loses an electron, it becomes nitrogen, which has three electrons in its 2p shell. These three electrons occupy different orbitals in the shell. Oxygen has that fourth electron sharing an orbital with one of the other electrons and repulsion is greatly increased. As a result, the ionization energy needed to pull an electron off of oxygen (or boron for that matter) is smaller due to these repulsions (they want to get rid of that electron and reach a stable state). I hope this explanation is coherent enough for you and proves to be useful.