double descent

Large language models like GPT have a fixed memorization capacity of about 3.6 bits per parameter, storing far less raw…