What I suggest you try is to change the Power Options of both the VM and host to "High Performance" profile. I think the default is now "Balanced" even for desktops which make the machine (whether physical or virtual) to go to sleep after 30 minutes. It might be the case that the "High Performance" profile might be already be missing.
In the advanced settings, there is also PCI Express - Link State Power Management which is enabled in the "Balanced" but disabled in the "High Performance".
The suspicion that I have is that somehow the host PCI Express goes into some power saving state (possibly messing with the GTX 970), essentially pulling the rug under the VM.