This is really very odd sounding. I wonder if there is some solder problem with the RPi Zero headers. If that was adding some extra resistance (or there were bridged pins) it could maybe have this effect. Would you want to post some photos of the solder for us to inspect?
You could also pull the Pi out of the FPZ and hook up to HDMI. That might tell you if the problem is on the Pi itself.
UPDATE: Are you plugging in any USB devices? Are you connected to a WiFi network?
You could try turning off WiFi. I have seen it affect performance, but not like what it seems you're talking about.
My first test would be to run the Pi by itself (without the FPZ board) connected to HDMI. You might want to connect a keyboard or controller via USB to test it. If it's slow there, I'd inspect the solder and maybe try a different SD card.
You could also pull the Pi out of the FPZ and hook up to HDMI. That might tell you if the problem is on the Pi itself.
UPDATE: Are you plugging in any USB devices? Are you connected to a WiFi network?
You could try turning off WiFi. I have seen it affect performance, but not like what it seems you're talking about.
My first test would be to run the Pi by itself (without the FPZ board) connected to HDMI. You might want to connect a keyboard or controller via USB to test it. If it's slow there, I'd inspect the solder and maybe try a different SD card.
Card Fighters' Clash 2 English Translation ( http://cfc2english.blogspot.com/ )
Neo Geo Pocket Flash Cart and Linker Project ( http://www.flashmasta.com/ )
Avatar art thanks to Trev-Mun ( http://trevmun.deviantart.com/ )
Neo Geo Pocket Flash Cart and Linker Project ( http://www.flashmasta.com/ )
Avatar art thanks to Trev-Mun ( http://trevmun.deviantart.com/ )