Long Short-Term Memory (LSTM) is a Neural Network (NN) type that creates temporal connections between its nodes. It models sequence data for applications such as speech recognition, image captioning, DNA sequence analysis, and sentence translation. Applications that are often subject to privacy constraints. This paper thus presents basic building blocks for the homomorphic execution of an LSTM that would respect the privacy of its inputs. By means of TFHE functional bootstrapping, we propose several approaches for homomorphically evaluating discretized flavors of the Sigmoid and Tanh activation functions. Experimental results show that the accuracy of the resulting discretized networks remains comparable to a full precision clear-domain execution. Performance-wise, we are able to homomorphically compute a Sigmoid or Tanh function in 0.3 or 0.15 secs (depending on whether or not multivalue bootstrapping is relied on). This paves the way towards evaluating practical LSTMs over encrypted inputs in around 1 to 3 mins which is competitive with the state of the art.