It was AFO all along. Even when you thought it wasn't, it was actually AFO. AFO and doctor is always there, watching and waiting for any major disaster to happen. Or if AFO is in a good mood, he decides to fuck with random people too. No one ever did anything bad or has any legitimate reason to actually ever do anything bad, it's just AFO's negative influence on them. AFO probably stole Izuku's quirk too that bastard! And I bet it's AFO's fault that Izukus father is never at home.
No but seriously, from the way his skin is disintegrating, I get the feeling it really was AFO/doctor. Hope not though