Finding that balance between flexible enough and too flexible is difficult. Like you said, everything is never going to be perfect, and there will always be something that goes wrong, and we, as teachers, will have to adjust accordingly.
I think the line needs to be drawn either based on your own comfort level, or when it gets to the point where your teaching or your students' learning is compromised. It is very difficult to think on your feet and change a lesson plan if something goes wrong, but you have to, as a teacher. If worse comes to worse, and a lesson gets so messed up, or something goes very wrong to the point where you don't think you can be that flexible, then just switch that lesson with another one for the another day, which would give you time to go home and re-figure the one that wasn't working. Being able to do that is flexible too.
Flexibility is also just about keeping a level head and staying clam, it's never as bad as it seems.