In areas of activity where the notion of accountability is strong, the adoption of artificial intelligence (AI) is limited by the opacity and lack of understanding of its behavior, all the more so in the embedded domain where neural networks are compressed and executed on microcontrollers. While the NIST introduced in 2021 several principles allowing the AI explainability, this paper introduces a novel scheme, HistoTrust, combining secure hardware and blockchain technology to bring trust in the traceability of AI behavior and allow its explainability. HistoTrust attests in an Ethereum ledger all the relevant data produced by a physical device, especially the heuristics inferred by AI. Thus, the audition of the ledger allows security verifications and AI behavior analysis.