The problem of the charts from OpenAI is that they purport to show linear growth of capability with levels of compute, however, the X-Axis is a log scale. If this is re-plotted using a more normal linear scale, we see massive plateauing of capability.
Each order of magnitude of compute increase leads to only small increases of capability.
Whilst using log scales is useful to show how things behave over scales of 1000 or 10000, it can mislead, as in this case, to think that capability is increasing nice and linearly and we will "get there" quite soon.
These are very similar to the graphs produced to demonstrate emergence on that old "emergent properties" paper of about 2022 in concept which also demonstrated plateauing when re-plotted to linear scales.
I always wonder when LLM researchers use this form of graph, whether they are intentionally misleading their readers (and themselves), or whether they are incompetent and do not understand the implications of a logarithmic X-axis and that straight line.
The problem of the charts from OpenAI is that they purport to show linear growth of capability with levels of compute, however, the X-Axis is a log scale. If this is re-plotted using a more normal linear scale, we see massive plateauing of capability.
Each order of magnitude of compute increase leads to only small increases of capability.
Whilst using log scales is useful to show how things behave over scales of 1000 or 10000, it can mislead, as in this case, to think that capability is increasing nice and linearly and we will "get there" quite soon.
These are very similar to the graphs produced to demonstrate emergence on that old "emergent properties" paper of about 2022 in concept which also demonstrated plateauing when re-plotted to linear scales.
I always wonder when LLM researchers use this form of graph, whether they are intentionally misleading their readers (and themselves), or whether they are incompetent and do not understand the implications of a logarithmic X-axis and that straight line.