First things first bro, you can always ditch the cleats and see if you can take it. Secondly, if you are in any unsafe groups, ditch that as well. Thirdly (which i think you are always doing also), just do solo lah.
High chance you can take it without cleats. Without cleats, there is a much higher chance that you'd be able to land safely even if you screwed up on the flat pedals.......it's just normal reflex.
If you dont trust me, try it first on glass and trying starting off with the wrong foot or even trying to switch from sidewalk to road via a normal kerb and you'd definitely be off balance, but see how fast your reflexes can take over (coz tip of feet cannot touch the ground) and you wont really fall. Difficult to describe to you, but you gotta try it. The vast vast vast majority of your other road cyclist friends wont be able to advise much on this to you, simply coz they are not on flats.
Go TB and get those flats and CF pedals which has a lot of pins, mountain bike flat pedals (110mm wide), you wont slip off the pedals even when off-saddle and putting out 400-500W one lah.
No need to ditch cycling, you did not kena a traffic accident with another vehicle, in which this risk can be largely mitigated with routes and timings. You are not using the bike like a motorcycle doing commuting during more normal hours, be it locally or overseas.
For the distances and stuff that you are I are doing, really no use for cleats. Plus the age too and subsequent consequences should something happen. In fact, i know some masters UK duathletes use running shoes, for 20km duathlons....both the first and second places on podium. Overall faster than cleats actually (that's what they claim, i never bother to verify). They are not slow, iirc 42+ avg speed on bike (draft legal event tho, not like T100) and the 2.4k converted run was in 9 mins range...i just checked, 9:18 for 2.4k (the run legs are 2.5k each). These are very very performance oriented fast folks, and if over 20km they can be faster with cleats over flats, for sure they will use it.