Loss: choose the error that learning will pursue

A loss turns a model prediction and its target into a numerical training signal. The optimizer follows its gradients, so the objective defines which mistakes receive attention.

For regression, squared error punishes large residuals especially strongly. Absolute error gives a different tradeoff. For classification, cross-entropy rewards probability assigned to the observed class, not merely whether the top label was correct.

Same accuracy, different learning signal

Two classifiers can both choose the correct class while assigning it probabilities of 0.51 and 0.99. Their accuracy is identical; their cross-entropy losses are not. The less confident correct prediction receives a stronger incentive to move.

Most neural-network libraries accept raw logits and combine normalization with the loss for numerical stability. In Keras, set from_logits=True when passing scores rather than probabilities. Applying softmax first and then treating the result as logits changes the calculation.

A falling training loss means progress on that objective and data. It does not by itself establish better held-out performance, factual answers or the behaviour an application actually needs.