Possibly both could be the culprit but I'd lean more towards carbs intuitively. The first thing I would try would be to just sip some aminos/whey + waxy maize/dextrose during the workout and see how that goes. That way you don't see any performance decline from removing a potent ergogenic aid, caffeine, and you don't have to revamp your entire diet just yet.