Learning Algorithms With Neural Turing Machine

Learning algorithms from pure input-output examples is a core problem in artificial intelligence.Neural Turing Machine employs the external memory to enhance the performance of neural network,which can deal with long term dependencies in learning algorithms.While training Neural Turing Machine to learn addition and multiplication,we find two issues: one is that the model is hard to train,which leads to the difficulty of improving the model's accuracy;the other is that the sigmoid activation function is used frequently,which leads to slow training of the model.In order to improve Neural Turing Machine's performance,the main work is as follows:(1)We use curriculum learning in training Neural Turing Machine to reduce training difficulty.However,there are two defects in conventional curriculum learning: first,it doesn't review the past curriculum while learning new curriculum;second,it doesn't keep balance between the new curriculum and the past one.In order to solve these two issues,we design two new variants of curriculum learning.One is curriculum learning based on fixed curriculum-rate: it generates samples based on a given rate while learning new curriculum,and the other is curriculum learning based on adapted curriculum-rate: it adjusts the proportion of the new curriculum according to its difficulty.(2)We reduce the amount of calculation by introducing Hard sigmoid function.The amount of sigmoid activation function's calculation is large.Hard sigmoid function has the main properties of sigmoid function,and it is also simple in calculation and easy to optimize.Therefore,we use Hard sigmoid function instead of the sigmoid activation function in Neural Turing Machine,and call the new model H-NTM.The experimental results show that these two variants of curriculum learning improve Neural Turing Machine's performance under all experimental circumstances;compared with Neural Turing Machine,H-NTM trains faster;after using curriculum learning,H-NTM achieves higher accuracy and faster training speed than Neural Turing Machine.