I see https://github.com/ApolloAuto/apollo exists which is an autonomous driving software. But I feel that is over kill.
I was trying to imagine either using a sort of electric fence, like a dog collar, and maybe something more analog to be able to detect the perimeter and just have it know to stop or reverse with a quick servo.
https://blog.singleton.io/nnrccar/
https://hackaday.com/2012/01/03/nerual-networks-control-a-to...
Also, I have built a couple toy robots that use "whiskers" to avoid obstacles.
It’s a well documented and easy to implement project.