I'm noticing a similar behavior, though it seems to be system wide. If I let it sit with no button presses long enough (maybe 10 seconds or so) things will get choppy. Games and the main menu are affected. Things will remain choppy until a second or two after pressing a button, then things smoothen up again.
I'm using the stock SD image with the filled driver option for BoxyPixel.
I'm using the stock SD image with the filled driver option for BoxyPixel.