Memory is needed for more than just weights