Skip to contents

Token efficiency

Source: vignettes/token-efficiency.Rmd
token-efficiency.Rmd

gptr logo

Keep data and computation in R, and send models enough context to choose useful actions. For inline work, an object around 4 GB can remain in the same R session across turns. The model sees a compact description while its R tool computes on the object. No repeated export and reload cycle is required; R copies, computation time and request latency still depend on the workflow. Workers receive serialized context rather than sharing the parent workspace. Four practices help:

  • Refer to live objects with a compact description rather than printing large tables into a prompt. The model can compute on those objects with the r tool.
  • Compose several deterministic operations in one R call and return a bounded result.
  • Reuse a session’s stable prompt and growing conversation when the provider supports prompt caching. Changing models, tools, compaction or rewind can change that prefix.
  • Inspect usage and set budgets before starting a costly workflow.

The executed examples use fake providers. Their request counts and estimates demonstrate the interfaces; they are not measurements of a real model’s speed, quality or bill.

Inspecting the prompt 

library(gptr)
view = gptr_prompt(preset = "minimal")
attr(view, "tool_names")
#> [1] "read"  "r"     "edit"  "write"
view$sections
#>       name tier tokens
#> 1 preamble   T0     42
#> 2    tools   T0     86
#> 3    rules   T0    348
#> 4    modes   T0     84
#> 5  context   T0    152
view$total_tokens
#> [1] 1875

gptr_prompt() inspects the system sections and tool definitions with estimated token counts; printing it shows their text. It does not query a provider. The standard preset describes most capabilities as functions in peter$, with the core model-visible r, read, edit and write tools and ask when applicable. view$total_tokens already counts the tool array and a preview of the first message; your prompt and the objects you attach add to it.

Object descriptions and result size 

writeLines(gptr_describe(iris))
#> <data.frame> 150 x 5, 7.1 KB
#>   $ Sepal.Length <dbl> min 4.3, median 5.8, max 7.9; no NA
#>   $ Sepal.Width <dbl> min 2, median 3, max 4.4; no NA
#>   $ Petal.Length <dbl> min 1, median 4.35, max 6.9; no NA
#>   $ Petal.Width <dbl> min 0.1, median 1.3, max 2.5; no NA
#>   $ Species <fct> 3 levels, top: setosa (50), versicolor (50), virginica (50); no NA
writeLines(gptr_describe(iris, budget = 60))
#> <data.frame> 150 x 5, 7.1 KB
#>   Sepal.Length:dbl Sepal.Width:dbl Petal.Length:dbl Petal.Width:dbl Species:fct

gptr_describe() summarizes class, shape and values within a budget (150 tokens by default). Packages can register methods for their own classes. Generative-model context normally uses these descriptions and names; System 1 questions send their input states, so their egress and size rules differ. See System 1 decisions.

work = new.env()

session = peter("Compare mpg and weight by cylinder count.", mtcars,
                model = fake, envir = work, mode = auto)

work$slopes
#>         4         6         8
#> -5.647025 -2.780106 -2.192438

The calculation runs in R once the model requests it. Return a small table, statistic or plot rather than a full intermediate object. Long r output keeps a bounded head and tail; the full spilled text can be read through peter$out(id) when an output ID was supplied.

Reported usage and estimated components 

gptr_usage(session, by = "model")
#>   group        requests  input  output  cache_read  cache_write  cost
#> 1 fake/fake-1  2         3290   48      0           0            0
#>
#> # 1 row
#> 2 requests, 3.3k tokens in, $0.0000

ledger = gptr_usage(session, detail = TRUE)
aggregate(tokens ~ component, data = ledger, FUN = sum)
#>       component tokens
#> 1      attached    130
#> 2   environment    210
#> 3        images      0
#> 4         other    162
#> 5       project      0
#> 6            t0   2684
#> 7            t1    562
#> 8  tool_results     53
#> 9         tools   1926
#> 10   transcript     70
#> 11    workspace     58

The usage summary contains provider-reported input, output, cache reads/writes and cost. by = "session", "agent", "model" or "route" groups it; child requests are included once. gptr_usage() without a session also includes the process-level System 1 log.

The detailed ledger estimates components such as tools, workspace, attached objects, transcript and tool results. Its sum can differ from reported usage because tokenizers, provider accounting, images and fake-provider behavior differ. Unknown reported usage or cost stays NA, not zero. A local model’s known zero metered charge does not imply zero hardware cost or latency.

Budgets and partial results 

budget_error = tryCatch(
  peter("Summarize mpg.", model = small, budget = list(tokens = 500)),
  gptr_error_budget = function(e) e
)
class(budget_error)[1]
#> [1] "gptr_error_budget_tokens"
budget_error$session$status
#> [1] "budget"

The estimated input for the first request already exceeds 500 tokens, so this call stops before that request. A top-level run’s defaults are 2 million tokens and 5 US dollars; budget = list(tokens = ..., cost = ..., turns = ...) supplies limits for that call. Change settings deliberately when different defaults are needed:

gptr_config(budget = list(tokens = 20000, cost = 1, turns = 10), .scope = "session")
session = peter("Review this analysis.", budget = list(tokens = 12000, turns = 6))

Checks happen before requests using estimated next-input tokens and usage already recorded. The next output or final provider charge can exceed a remaining allowance; unknown usage cannot establish a hard monetary ceiling. A budget is a stopping policy, not a billing reservation. At the console the UI can ask to extend it; without a person present the call signals a budget condition carrying the partial session as e$session. Inspect that session’s status, transcript, workspace and usage before deciding whether to continue.

Teams and fan-outs share the root call’s budget; launching more children does not give each one a fresh root allowance. With gptr_parallel(), member budget =, tools = and per-run .opts are not applied. Set the intended limits through the governing settings or enclosing run, cap concurrency and inspect the returned children’s statuses. See Teams and background work.

Compaction

At the chat prompt, /compact replaces older model context with a summary. Add a focus when particular details must survive, for example /compact retain the model comparison and next steps. Automatic compaction follows the configured context and cold-cache thresholds; see Configuration.

Compaction can make a model request and add usage. It keeps the R objects already in your session, while the model no longer sees every older turn verbatim. Preserve important decisions in project files and named R objects. A changed request prefix can also affect provider cache reuse.

Three different caches

Mechanism What is reused Inspect
Provider prompt cache A repeated request prefix gptr_usage(session)
System 1 answer cache Matching input, question and model Metadata and gptr_cache()
Replay / System 2 cache Recorded code and saved answer gptr_blocks(path), gptr_cache()

A stable prefix makes provider cache reuse possible. Providers decide whether a cache hit occurs, subject to their support, minimum size, expiry and pricing; gptr does not guarantee a hit after the first request. Compaction, image elision, changed resources or rewind can break the prefix. Document replay makes no model request for a usable recorded call but still runs its R code. Clearing an answer cache changes what can be reused.

Explicit ways to reduce context 

peter("Summarize the model fit.", fit, .opts = list(context = "names"))
peter("Find the mean of x. Return one number.", .opts = list(context = "none"))
peter("Compute the mean.", x, .opts = list(preset = "minimal"))

context = "names" lists workspace names and uses headers for explicitly attached objects. "none" omits both built-in workspace and attached-object descriptions; the R bindings still exist and can be used by tool code, as x is in the second example. Prompt text, tool results, images and other context blocks can still reach the model. System 1 input states use a separate encoding route. Execution model explains these boundaries. preset = "minimal" reduces the tool/prompt surface; it does not authorize actions or remove required safety checks.

{identifier} in a literal prompt can insert a short atomic value when that is sufficient. For a long workflow, continue one session, keep intermediate data in R and request concise reports between stages. Use a new session or an explicit fork when a separate analysis is intended.