Skip to content

Latest commit

 

History

History
189 lines (147 loc) · 8.04 KB

CONTRIBUTING.md

File metadata and controls

189 lines (147 loc) · 8.04 KB

contributing

build commands

# UI deps, playwright browsers, and rust crates (run after a fresh clone)
make deps

# rust backend and CLI
make build

# UI (react + typescript + vite)
make build-ui

the toolchain versions are pinned in .mise.toml. pyo3 caps out at python 3.13, so the build breaks against a newer system interpreter unless mise is providing the pinned one.

testing

every test here is end-to-end: it drives the built binaries or the HTTP API from outside, so there are no rust unit tests to keep in step.

  • make precommit: runs linting and all tests. recommended before pushing.
  • make test: runs all e2e tests (API, CLI, UI, workflows, release).
  • make test-api-e2e: runs HTTP API tests using tests/run_e2e.sh and hurl.
  • make test-ui-e2e: runs UI tests using playwright (ui/tests/).
  • make test-cli-e2e: runs CLI tests.
  • make test-e2e: runs the shipped workflows in workflows/ end to end (tests/e2e/, python).
  • make test-release: checks the tag guard described under cutting a release.

workflow e2e

make test-e2e executes the real workflows against fake-openai, a programmable openai-compatible mock, so the openai-backed nodes are covered without a live provider or an api key. clone it beside this repo (or point FAKE_OPENAI_DIR at it); the target skips with a message when it is absent.

the tests are python -- a different language from the app under test, so they stay honestly black-box -- and drive flow-cli and flow-server as subprocesses. each run gets a scratch working directory holding only symlinks to user_nodes/ and docs/, so the repo's .env, cache, and generated_assets/ are never touched and inherited OPENAI_* variables cannot redirect a test at a real server.

a test programs the mock's response queue, runs a workflow, then asserts on both the workflow's output and the requests the mock captured. --llamaswap enables the mock's non-standard gateway surface (/upstream/<model>/v1/* and model aliases), which is off by default so the same suite can prove the nodes work against a plain openai server.

three openai-backed workflows are not covered, because a mock cannot stand in for what they need: feedcast and fetch-summarize reach the open internet, and mesh-quip needs radio hardware.

the server embeds the built UI using rust-embed, so run npm run build in ui/ before cargo build to include UI changes.

cutting a release

.github/workflows/release.yml publishes on a v* tag. the tag is the trigger, but scripts/bundle-release.sh names tarballs from Cargo.toml, so the two have to agree:

# 1. bump Cargo.toml, commit, and check the guard agrees before tagging
scripts/release-version.sh tag v0.6.0

# 2. tag and push
git tag v0.6.0 && git push origin v0.6.0

a tag that disagrees with Cargo.toml fails the prepare job in about twenty seconds, before any build starts. make test-release covers that guard.

the run then goes:

  • prepare resolves the version and opens a draft release, whose notes are a setup preamble plus github's generated changelog.
  • build-macos / build-linux build the twelve tarballs in parallel and attach each one to the draft as it finishes.
  • publish adds SHA256SUMS and flips the draft public.

so the release only becomes visible once every tarball is on it. if a build fails the draft stays a draft: fix it and re-run, or publish it by hand from the releases page. a tag with a suffix (v0.6.0-rc1) is marked a prerelease and does not become latest.

workflow_dispatch on a branch runs the same builds and uploads them as run artifacts without touching releases, which is the way to smoke-test a change to the workflow without burning a tag.

architecture

this is a visual node-based workflow execution engine with a rust backend and react frontend.

core engine (rust)

  • src/engine.rs: workflow execution engine with topological sort, result caching (SHA256 hash of inputs), and dependency resolution. the NodeRegistry manages node type registration.
  • src/node.rs: the Node trait defines the interface all nodes implement: name(), inputs(), outputs(), and async execute(). nodes declare typed I/O via the DataType enum.
  • src/graph.rs: Workflow and NodeInstance structs define the workflow graph format. node connections use {"$node": "id", "$output": "field"} syntax.
  • src/value.rs: dynamic Value enum (Null, Boolean, Integer, Float, String, Array, Object) used throughout the system.
  • src/nodes/: node implementations -- add new nodes here and register in mod.rs via register_all().

binaries

  • flow-server (src/bin/flow-server.rs): axum HTTP server with endpoints:
    • GET /api/nodes -- list available node types
    • POST /api/execute -- execute a workflow
    • GET/POST /api/workflows/{name} -- load/save workflows (stored in the data dir's workflows/ as JSON)
  • flow-cli (src/bin/flow-cli.rs): CLI runner for saved workflow files.

UI (react + reactflow)

  • ui/src/App.tsx: main canvas component with drag-drop node creation, edge connections, and workflow execution.
  • ui/src/components/Nodes.tsx: visual node components for each node type.
  • workflows saved from the UI are JSON with nodes and edges arrays including visual layout.

adding new node types

  1. create a node struct in src/nodes/ implementing the Node trait.
  2. register it in src/nodes/mod.rs via registry.register("NodeName", || Box::new(node::NodeType)).
  3. add a matching react component in ui/src/components/Nodes.tsx with input/output handles.
  4. register the component in the nodeTypes map in App.tsx.

scripting and user nodes

the engine supports dynamic node creation via files placed in the user_nodes/ directory. supported formats (all enabled by default):

  • rhai (.rhai) -- sandboxed scripting baked into the rust binary. fast and safe; the bundled openai_* nodes are written in rhai.
  • python (.py) -- full cpython via pyo3.
  • typescript (.ts) -- executed via the embedded boa engine.
  • declarative (.json) -- compose existing nodes into a reusable unit without writing any code.

files in user_nodes/ are automatically loaded on server startup.

a python node that needs a third-party package (XmlSelect needs lxml, MeshtasticSend needs meshtastic) must import it inside execute() rather than at module scope. a failed top-level import fails the whole load, so the node drops out of the registry with only a startup WARN to show for it, and a declarative node composing it still registers and then breaks when run.

those packages live in .venv, listed in requirements-nodes.txt and installed by make deps (make deps-python alone re-installs them). mise creates and activates the venv; without mise, uv creates it. an embedded interpreter reads no pyvenv.cfg, so flow adds the venv to sys.path itself, taking it from FLOW_PYTHON_VENV, else VIRTUAL_ENV, else ./.venv.

the embedded interpreter

pyo3 links libpython at build time but CPython resolves its sys.prefix at RUN time, from the first python3 on PATH. the two have to be the same install: pair one's stdlib with the other's library and imports fail in ways that name the wrong thing -- a mise python-build-standalone install compiles ssl INTO libpython instead of shipping _ssl.so, so crossing it with the system libpython leaves no reachable _ssl and every https url in a python node fails as unknown url type: https.

make builds against the .mise.toml pin (PYO3_PYTHON), and build.rs adds an rpath to that interpreter's libdir so the loader cannot bind a different libpython than the one on PATH. interpreters already in a loader directory (/usr/..., /lib...) get no rpath: nothing to disambiguate, so the release workflow's python:* containers build exactly as before. make test-e2e covers this -- a probe node reports whether ssl imports and whether urllib offers an https handler.