Where AI Value Lands

Value Functions as Algorithmic Emotion

4 min read

Value Functions as Algorithmic Emotion

One-line

Sutskever’s theoretical proposal (sutskever-age-of-research) that biological emotions are highly robust, low-complexity value functions providing intermediate trajectory feedback during long, sparse-reward reasoning — and that the next breakthrough in AI is engineering an artificial analogue.

The argument

Current RL paradigms train models with terminal rewards (success at the end of a trajectory). For short trajectories (one-shot puzzles, well-defined coding tasks) this works. For long, open-ended reasoning — the kind of behaviour required by continual-learning-paradigm — the credit-assignment problem becomes intractable: by the time a reward arrives, the model cannot reliably attribute it to any specific action taken thousands of steps earlier.

Humans don’t have this problem. They have emotions and intuitions that provide continuous, cheap, robust intermediate feedback during reasoning. A wrong-feeling step gets flagged before the trajectory finishes. Curiosity rewards exploration. Frustration prunes unpromising branches.

Sutskever’s claim: deep ML can construct an analogous structure — a learned value function that scores intermediate trajectory states without needing a verifiable terminal reward.

Why this is hard

The interviewer (citing a DeepMind AlphaCode paper) raises the obvious objection: the state space of complex tasks like software engineering is infinitely wide. Learning a reliable mapping from arbitrary intermediate trajectory → terminal value requires generalising across a combinatorial explosion of paths most of which were never seen in training.

Sutskever’s response — “lack of faith in deep learning” — is an aesthetic argument, not an empirical one. He believes the mathematical breakthrough exists; he refuses to elaborate the mechanism, citing safety and competitive reasons. The proposal is, in the strict sense, unfalsifiable from outside SSI.

Relationship to existing scaling debates

  • Pairs with continual-learning-paradigm. Continual learning is the capability; value functions are the proposed mechanism that makes long-horizon learning tractable. The pair is what Sutskever frames as the “Age of Research” agenda.
  • Cousin of test-time compute. agentic-scaling-law adds compute at inference; value functions add signal density during training. Both attack the long-horizon problem; they are complements, not substitutes.
  • Bridges biological-inspiration debate. Sutskever’s method is explicitly biology as design constraint — emotions as proof-of-concept for cheap robust value signals. This sits in tension with Huang’s “speed-of-light” first-principles-physics methodology (ai-factory-huang): one looks to biology, the other to physical limits. Both are top-down; they bottom out in different reference systems.

Why this might be wrong

  • Emotions may not be value functions. The analogy is poetic; biologically, emotions are entangled with embodiment, hormones, and social signalling. Compressing them to “intermediate RL rewards” may discard most of what makes them useful in humans.
  • The state space objection may be load-bearing. If the AlphaCode critique is correct, no amount of architectural elegance recovers a reliable signal from an unboundedly wide trajectory distribution. The breakthrough Sutskever points to may simply not exist.
  • Reward hacking transfers. If value functions are themselves learned, they are gameable — the same “reward hacking” Sutskever criticises in eval optimisation (eval-real-world-gap) re-emerges one level up. Whose feedback shapes the value function, and how is that signal kept honest?
  • Existing intermediate rewards exist. Process reward models (PRMs), step-wise verifiers, and chain-of-thought tuning are early instances of this idea, already deployed. Sutskever may be claiming a much stronger version than is currently demonstrated.

Why this matters for Where Value Lands

If value functions become a load-bearing component of frontier systems, the competitive moat shifts upstream from data to training-signal design. The asset is no longer “we have the largest corpus” but “we have learned the most robust value function for domain X.” This:

  • Reinforces continual-learning-paradigm’s implication: model-layer moats become harder to commoditise via open-weights releases.
  • Has unknown implications for H2_u-curve-of-value: the bottom of the U could either narrow (only labs with the right algorithmic insight win) or broaden (off-the-shelf value-function APIs commoditise the layer faster than expected).

Graph View

Backlinks