Repository navigation
Add shiny-r-optimize skill for Shiny app performance - #91
Merged
Merged
Conversation
gadenbuie
marked this pull request as ready for review
October 6, 2026 23:40
gadenbuie
changed the title
Collaborator
Author
|
@cpsievert @schloerke if you have a minute to give this a quick scan -- have I missed anything? |
cpsievert
approved these changes
Oct 7, 2026
schloerke
approved these changes
Oct 7, 2026
Contributor
|
Thank you! |
Comprehensive performance-optimization skill for Shiny for R apps: SKILL.md diagnostic workflow + 7 reference files distilled from the Posit Conf 2024 Level-Up Shiny workshops, the official shiny.posit.co performance articles, and broader ecosystem research (Mastering Shiny, promises/mirai, Appsilon, shinyloadtest). Intentionally oversized (~17k tokens) as a first draft; a review pass will trim to a leaner, more prescriptive version. - Register in shiny plugin - Category README entry - Eval workspace kept uncommitted (shiny/shiny-optimize-workspace/)
Review pass over the first draft (1,430 -> 1,097 lines, ~17.2k -> ~12.2k tokens, -29%): - Restructure workflow: apply safe behavior-preserving fixes from the smell scan immediately; measurement confirms rather than gates (addresses analysis-paralysis seen in eval iteration 1) - Instruct reading only the reference the diagnosis points to - Merge scaling-users.md into diagnosis.md (7 -> 6 reference files) - De-duplicate flush-cycle explanation; cut prose padding, version trivia, and inline citations in favor of short further-reading lists - Preserve verbatim: ExtendedTask pattern, bindCache key rules, bindCache|>bindEvent ordering, debounce/throttle semantics, suspendWhenHidden behavior, data-format benchmarks - Fix stale scaling-users.md mention in category README
Eval findings (with-skill 100% vs 95.8% baseline, 2.4x faster, 22% fewer tokens) and grader observations drove five surgical additions: - Report integrity: cited measurements must be saved under outputs/measurements/ and claims checked against them; unverifiable claims are flagged, not asserted - Preserve-behavior rule now flags timing-semantics changes (debounce latency, cross-session cache sharing) - Parity check on one representative output before finishing - caching.md: bindCache vs memoise vs hoist decision guide - async-tasks.md: clarify shared $result() consumer semantics Skill remains lean: 1,122 lines / ~12.5k tokens
Following the skill-creator convention, keep the eval workspace as a sibling of the skill directory. Commit selectively: - evals/evals.json: eval prompts and assertions for the three test cases - apps/: deliberately-slow fixture apps (sales-dashboard, model-explorer) with READMEs documenting the planted performance problems - .gitignore: exclude iteration results (regenerable run outputs) and the generated sales.csv fixture (deterministic, regenerate via Rscript generate-data.R)
The fixture apps are meant to be diagnosed blind. Remove comments that flag the app as a deliberate exercise or name specific planted problems: - '*before* fixture' header notes in both apps - '(once per session is fine for now, right?)' next to the per-session read - 'the market research API is slow' / 'slow external service' next to Sys.sleep - 'so you can tell whether the app is responsive' on the live clock
The app directories are handed wholesale to the agent under eval, so the READMEs listing every planted problem were effectively an answer key. Move them to evals/GRADING.md (graders only) and reduce each app README to regeneration instructions and a pointer to the grader notes.
Without context, a bare Sys.sleep() reads as deletable dead code, so an eval agent could 'optimize' it away by simply deleting the line. Restore the comments framing the sleeps as calls to a slow external service so the delay is plausibly load-bearing.
From the eval-suite review: - Rewrite fixture READMEs as in-universe docs; they no longer announce 'fixture for evals' or point to evals/GRADING.md (which shipped inside the directory given to the agent under test) - Delete the workspace-level work/ scratch tree, which contained a complete reference solution - Add evals/RUNNING.md documenting the isolation protocol: run each eval from a clean git-archive export in a temp dir (no .git reachable), write eval_metadata.json only after runs complete, never run from the workspace root - Drop the measurement-artifacts-saved assertions (eval-0/1/2); artifact hygiene is a skill concern, not a graded criterion - Rescope req-gating-added to outcome-based outputs-gate-invalid-inputs; it false-failed two fully working, boot-verified apps - De-steer all three eval prompts: remove the 'reactive graph' hint, the freeze/clock symptom detail, and the explicit 'profile it / top 3 / before-after numbers' asks; add a plausible red-herring user theory. eval-2's ranked-top3 assertion is reworded to prioritized diagnosis to match
Not part of the skill-creator schema for this file, unconsumed by the aggregation/viewer scripts, and pointed at iteration directories that are deliberately not committed.
SKILL.md: - Recommend debrief over raw profvis: pv_* text summaries are made for agent consumption (overview, focus, hot lines, suggestions, compare) - Reorganize the measurement block by who runs it: profvis/debrief and bench::mark are agent-runnable; reactlog is a user activity (agent can reconstruct the reactive graph from code) and shinyloadtest is an end-of-process check of a DEPLOYED app against target load - Rework 'one change at a time': batch obvious, predictably-effective fixes and re-measure once; isolate changes only when attribution is genuinely uncertain - Drop the Shiny for Python paragraph (separate skill planned) async-tasks.md: - Call out that ExtendedTask keeps BOTH the waiting session and other sessions responsive, unlike older promise strategies that only unblocked other users - Point to the r-mirai skill (posit-dev/skills r-lib/r-mirai) for detailed mirai guidance data-loading.md: - Feature parquet alongside feather in the format table - Name dbplyr as the lazy-SQL backend; add guidance on where to collect() (realize once into a shared reactive vs. stay lazy and collect at the end) - Note DuckDB+httpfs can query parquet directly from S3 - Scope pool to real database servers; show a server example and note SQLite files / in-memory DuckDB need no pool diagnosis.md: - shinyapps.io is being sunset; cover Posit Connect Cloud (connect.posit.cloud) with the equivalent instance-sizing + run-choice knobs - Fold debrief into the profvis reading guide; reframe reactlog as primarily a user-driven tool
Frees the shiny-optimize name for a future Shiny for Python skill. Renames the skill directory, its eval workspace, and updates the frontmatter name, marketplace.json registration, category README, and eval suite references. Historical iteration records on disk keep the old name deliberately.
The 1.1 bump from the review-notes commit is premature — the skill is still unreleased in this PR.
Expand the Resources list for the shiny-r-optimize skill with the sources used in its development: the async and bindCache articles, Mastering Shiny, profvis/debrief, reactlog, shinyloadtest/shinycannon, mirai/crew, DuckDB/pool, and Connect scheduler docs.
- async-tasks.md / SKILL.md / README.md / diagnosis.md: lead with mirai everywhere; ExtendedTask example rewritten with mirai + daemons(); backends section swapped (mirai first); hard rule #1 feeds executed values into mirai() named args per schloerke's suggestion - SKILL.md: explain the outputs/measurements/ folder layout (schloerke) - data-loading.md: drop standalone vroom() row, note readr is vroom-powered (schloerke); add saveRDS()/readRDS() for non-tidy artifacts (cpsievert) - reactive-graph.md: nuance debounce/throttle intro (Shiny's internal debouncing, bursts when compute outlasts the window), example is now event_data('plotly_brushing'), shinyreact note (cpsievert) - rendering-ui.md: ExtendedTask quick reference (cpsievert) - cpsievert's caching.md Connect persistent data dir comment: pending a reference link from Carson
reactive-graph.md: new subsection after the debounce/throttle guidance — when smoothing isn't enough, gate the expensive update behind a trigger. bslib::input_task_button() is a direct actionButton() drop-in: automatic busy state from the flush alone, no server-side changes, no ExtendedTask. submitButton() noted as the blunt legacy option. SKILL.md reactive ladder and rendering-ui.md gate-speculative-outputs updated to match.
gadenbuie
force-pushed
the
skill/shiny-optimize
branch
from
October 14, 2026 20:26
f1bd14b to
f7aed29
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
shiny/shiny-r-optimize— a skill that guides an app developer through finding and fixing performance bottlenecks in an existing Shiny for R app: the app that worked fine during prototyping but is now too slow, too fragile, or unable to support the number of users who actually need it.Purpose
The skill walks an agent through a diagnosis-then-treatment workflow: understand the complaint → scan the source for classic performance smells and apply the safe, behavior-preserving fixes immediately → measure to confirm (profvis, reactlog, bench, shinyloadtest) → classify the bottleneck → apply the cheapest effective fix → verify with a parity check and saved measurements. Its core principle is "do less work before doing work faster": eliminate, reuse, accelerate, and only then relocate work off the main process. All fixes preserve app logic and behavior.
Use cases
Structure
SKILL.md(workflow, smell table, quick-wins scan, symptom→fix table, fix ladders) plus six reference files loaded on demand:diagnosis.md(profiling/load-testing, incl. the multi-process model and Connect/shinyapps.io knobs),reactive-graph.md,caching.md,async-tasks.md,data-loading.md,rendering-ui.md.Dependencies
profvis,reactlog,bench,shinyloadtest,memoise,cachem,future/promises/mirai,duckdb,pool/dbplyr,DT,bslib). The skill instructs the agent to ask before adding new dependencies to an app.ExtendedTask(≥ 1.6.0 forbindCache/bindEvent); version constraints are noted inline.Documentation
shinyplugin in.claude-plugin/marketplace.json; documented inshiny/README.md.Verification
shiny-r-optimize (1,122 lines, ~12,542 tokens)
Skill description: ~99 tokens
Evals
Evaluated over two iterations with three realistic scenarios (broad-slowness dashboard, multi-user freezing, profile-first triage) against deliberately-broken fixture apps, each run graded against 6–10 objective assertions by an independent grader:
Iteration 2 also used ~22% fewer tokens than unassisted runs. Eval fixtures, prompts, assertions, per-run grades, and benchmark aggregates are preserved in
shiny/shiny-r-optimize-workspace/(not included in this PR).