**Links to**: [[03 Semantic noise]], [[Semantics]], [[Syntax]], [[Linguistics]], [[NLU]], [[Kripkenstein]], [[Modulations]], [[Prompt engineering]]. ### Main ideas on NLP are presented in: [[03 Semantic noise]]. The distributional semantics hypothesis (that linguistic objects with comparable distributions must have similar meanings) upon which much of NLP rests, can be creatively questioned by the challenges made apparent by the polysemantic nature of some neurons (where we observe, in NNs, neurons that activate with combinations of apparently “unrelated” features). Essentially, we can also frame this conceptual-engineering challenge as a question of relative [[Interest]]: all features are possible, all concepts are possible, the challenges lie in understanding how we pool our collective attention towards them. Issues of scaling, in LLMs, are of particular importance here. Do we want to scale for the sake of scaling, hoping for possible discoveries, or do we want to tune, based on what we know, to refine what we “already” know/expect? Either way it’s inference all the way down/up. See also: [[ARC]]. %% https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation https://en.wikipedia.org/wiki/Distributional_semantics _linguistic items with similar distributions have similar meanings._ From BIGBench: ## Motivation Recent language models have achieved dramatic gains by significantly increasing their scale, in terms of parameter count and dataset size. Notable examples of models that make use of this scalability include [RoBERTa](https://arxiv.org/abs/1907.11692), which attained dramatically better performance by training for longer; [T5-11B](https://arxiv.org/abs/1910.10683), which achieved near-human performance on the challenging [SuperGLUE](https://super.gluebenchmark.com/) benchmark by scaling to 11 billion parameters; [GShard](https://arxiv.org/abs/2006.16668), which produced a 600-billion parameter machine translation model that supports over 100 languages with state-of-the-art accuracy; and [GPT-3](https://arxiv.org/abs/2005.14165), which showed that scaling language models beyond 100 billion parameters could achieve strong performance on many tasks without any further task-specific training. Indeed, the "[scaling laws](https://arxiv.org/abs/2001.08361)" of these models demonstrate approximately log-linear improvements in performance over > 6 orders of magnitude in parameter count. _Extrapolation of these trends suggests that a model with an additional 3-5 orders of magnitude of parameters would saturate performance on most current benchmarks._ This naive prediction of broadly superhuman performance from larger models makes it important to better understand what we can expect from these models as they continue to scale. Will scaling lead to models that can perform any text-based task? Are there fundamental capabilities that large language models lack? BIG-Bench is aimed at making progress on these and similar questions. ## [](https://github.com/google/BIG-bench/blob/main/docs/doc.md#call-for-benchmark-tasks) ## Call for benchmark tasks In order to answer these questions about the near-future capabilities of language models, and to prepare for the likely saturation of performance on many current benchmarks, we invite researchers to contribute diverse tasks to the BIG Benchmark, that challenge or exceed the capabilities of current language models. Tasks which measure almost any aspect of large language model capabilities are in-scope for this benchmark, including tasks which quantify social bias in language models. By soliciting benchmark tasks from the community we hope to provide a diverse and large scale benchmark. We encourage the contribution of tasks by experts in fields such as linguistics, cognitive science, philosophy, logic, and neuroscience. We also encourage contributions from language model skeptics: Our goal is to test model capabilities quantitatively, either establishing or disproving possible limitations of these models. Tasks will be allowed to interrogate a language model by asking it to generate text continuations or estimate log-probabilities of text strings. The task must then return a score capturing the model’s performance. The format and goals of tasks will be otherwise unrestricted. As some examples, tasks might involve: deriving mathematical proofs; engaging in common sense reasoning; answering theory of mind questions about a story; providing captions for ASCII-art images; discriminating synthetic from human text; performing arithmetic; solving ciphers; answering multiple-choice historical questions; solving crossword puzzles. We are particularly interested in tasks that are difficult for large language models, but that also are not over-tuned to the limits or idiosyncrasies of existing large language models such as GPT-3. Aspirationally, this benchmark will be a target for large language model research over the next decade, and therefore tasks that are extremely difficult for existing large language models are likely to be included. Contributors will propose tasks by submitting a GitHub pull request implementing their proposed language model task using the supplied API. In addition to code, each submitted task will contain a `README.md` file describing the task and how it is intended to probe the capabilities of language models. Tasks will further return meta-information describing their properties and their measurement targets, to better enable downstream analysis (for example, tasks will be categorized according to a provided list of [keywords](https://github.com/google/BIG-bench/blob/main/keywords.md))."