how much havoc can a compromised baseband do to a Guardian ROM device?

Anyone knows whether a Nexus 4 baseband processor has r/w access to system memory? The firmware doesn't seem to be loaded at boot, so I presume it's entirely out of reach/ reversing?