LLM agent to firewall vulnerable FOSS libs?

As software providers, we use several FOSS libraries. These libraries require periodical security bug fixing, which requires shipping a new release. We are building an LLM agent that can block these security bugs at runtime (like a firewall so to speak) and prevent attackers from exploiting security bugs in vulnerable libraries. Are there any pitfalls that we must avoid? Your feedback is welcome.