Event camera pixels efficiently encode visual information through triggered events, offering advantages in temporal detail, dynamic range, and data reduction. However, the optimal machine learning method for leveraging these characteristics remains unclear. Existing approaches often convert events into 2D frames, losing crucial time-domain information. A promising alternative is event-graph neural networks, but they suffer from computational intensity and limited temporal dependencies. As a solution, we propose to combine the recently proposed lightweight event-graph neural network HUGNet with gated recurrent units to model temporal dependencies between the features extracted by HUGNet. We benchmark our model against other event-graph and convolutional neural network based approaches on the challenging DVS-Lip dataset (spoken word classification). We find that not only does our method outperform state of the art approaches for similar model sizes, but that, relative to the convolutional models, the number of calculation operations per second was reduced by 81%. Furthermore, we introduce a new event-data augmentation technique that boosts by up to 7.4% the performance of both event-graph and convolutional neural networks on this task.