25. Transformers

As good as recurrent networks are, they still face fundamental limitations. Vanishing gradient makes it so RNNs can only look back ~100 data points in a series or ~100 in an LSTM or GRU. Transformers let us learn how all inputs are related to all other inputs through the novel "multi-head attention" mechanism. This video describes that attention mechanism and shows one example, CrabNet, of how they can be applied to materials informatics.

Check out the whole materials informatics series at https://youtube.com/playlist?list=PLL0SWcFqypCl4lrzk1dMWwTUrzQZFt7y0 with workbooks and course notes available at https://github.com/sp8rks/MaterialsInformatics

0:00 limitations of recurrent models
1:30 attention is all you need and word embeddings
3:10 positional encoding to capture context
4:30 transformer architecture components
10:20 attention layer Query Key Value
15:21 CrabNet