From 38535f9b99b6a9c774d30ae3d491e398944c8d9d Mon Sep 17 00:00:00 2001 From: JohnnyT Date: Sat, 24 Oct 2026 16:06:39 -0600 Subject: [PATCH] Compares exported positions and drives the diff cases The conformance runner refused the statifier suite's ten diff cases by naming their host keys, and refused the four cases that state an exported position. The reference's harness drives a diff case like any host case and compares none of to_source, expect_diff, mapping and expect_compatible_at, so the runner now accepts the four keys and compares nothing of them. A step's expect_position is now compared, once its configuration agrees, with the state's position exported and rendered as the reference renders it: the id sets and history lists sorted, each live invocation as its state and index, the running flag, and the datamodel without _event, _ioprocessors and _sessionid, and without _name unless the scxml element carries a name attribute, each value in its JSON form. A date, datetime or duration value fails the step naming the variable, and a case stating a position on a document with a state that has no id fails before it is driven. A case stating a position sends its steps' event data, as the reference's harness does. Every statifier case passes, and the ratchet records the fourteen new passes: the registry holds 309 entries, 36 of them statifier's. Refs: srb-1kv --- README.md | 7 +- conformance/README.md | 38 ++-- conformance/registry.json | 14 ++ test/conformance/drive_test.rb | 13 +- test/conformance/gap_list_test.rb | 2 +- test/conformance/position_expectation.rb | 177 ++++++++++++++++++ test/conformance/position_expectation_test.rb | 175 +++++++++++++++++ test/conformance/runner.rb | 63 ++++--- test/conformance/statifier_suite_test.rb | 59 +++--- 9 files changed, 470 insertions(+), 78 deletions(-) create mode 100644 test/conformance/position_expectation.rb create mode 100644 test/conformance/position_expectation_test.rb diff --git a/README.md b/README.md index b043c51..34a6112 100644 --- a/README.md +++ b/README.md @@ -345,11 +345,11 @@ claims to pass, and the gate fails if one of them does not. | Claim | Cases claimed | Of | |---|---|---| | `scion` | 119 | 119 cases in the scion suite | -| `statifier` | 22 | 36 cases in the statifier suite | +| `statifier` | 36 | 36 cases in the statifier suite | | `w3c-mandatory` | 152 | 154 mandatory cases in the w3c suite | | `w3c-optional` | 2 | 14 optional cases in the w3c suite | -The registry holds 295 entries, which is the sum of the four rows. A claim is +The registry holds 309 entries, which is the sum of the four rows. A claim is the set of its entries, not a statement that every case of its suite passes. What is not claimed yet: @@ -358,9 +358,6 @@ What is not claimed yet: the reference's registry does not claim them either. - Twelve optional w3c cases whose host needs the Basic HTTP Event I/O Processor, which this package does not provide. -- Fourteen cases of the `statifier` suite, the reference's own: the ten diff - cases, whose host keys the conformance runner does not read yet; and the four - cases that state an exported position, which it does not compare yet. Running the gate prints the gap list: every unclaimed case with the features it needs and whether it passes or why it fails. diff --git a/conformance/README.md b/conformance/README.md index 9c67a0c..b26e164 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -103,27 +103,43 @@ once every configuration agrees, the sends handed over the whole drive, in order and each with its outcome, must be exactly `expect_sends`. A `` that reaches the host marks the delayed send it names `"outcome": "cancelled"` when the expected item at that send's position -asks for it, and a step of a case with a host object sends its event's -`data`, when it gives one. A send whose expected item asks the host to +asks for it, and a step of a case with a host object, or of a case stating +an exported position, sends its event's `data`, when it gives one. A send +whose expected item asks the host to report it failed (`"outcome": "fail"`) is marked so and reported through `Statifier.report_send_failed` once the driver call that handed it returns, before the configuration is read, as the reference's harness reports it before it reads the next configuration; a report the driver refuses because -the chart has stopped is dropped. Any other host key (the accepts check's -`declared_events` and `expect_accepts`, the diff pair's `to_source`, -`expect_diff`, `mapping` and `expect_compatible_at`), and a step stating an -exported position each fail the case naming what it asks for. This +the chart has stopped is dropped. A case's `host.declared_events` and +`host.expect_accepts` are read as the reference's harness reads them: +`Statifier.check_accepts` answers for the compiled chart before it starts, +and its two lists must be exactly `expect_accepts`. A host object's diff +pair, `to_source` and `expect_diff` with an optional `mapping` and +`expect_compatible_at`, is compared by nothing here, as the reference's +harness compares none of the four: a case carrying it is driven like any +other host case, and agrees on its configurations and its sends. A step's +`expect_position` is compared, once that step's configuration agrees, with +the state's position exported through `Statifier.export_position` and +rendered as the reference's harness renders it +(`Conformance::PositionExpectation` in +`test/conformance/position_expectation.rb`): the configuration, the entered +states and the states to invoke, each sorted; each history state's recorded +states, sorted; each active invocation as its state and index, sorted; the +running flag; and the datamodel without `_event`, `_ioprocessors` and +`_sessionid`, and without `_name` unless the `` element carries a +`name` attribute, each value in its JSON form. A date, a datetime or a +duration has no JSON form and fails the step naming its variable, and a +case stating a position on a document with a state that has no id fails +before it is driven. Any other host key fails the case naming it. This registry claims the scion suite, every one of its cases, the w3c cases the -run passes and the statifier cases the run passes: the `w3c-mandatory` +run passes and every case of the statifier suite: the `w3c-mandatory` claim covers its mandatory entries, the `w3c-optional` claim its optional ones and the `statifier` claim its statifier entries, since a claim is the set of its entries (the copy's `RATCHET.md`, "Claims"), not a statement that every case of the suite passes. The w3c cases left out are the two mandatory cases the run fails and the optional cases whose host object -names the BasicHTTP processor, which the drive does not provide. The -statifier cases left out are the ones that fail for what the drive does -not do yet (the gap list names each with its reason); each joins the claim -when the ratchet sees it pass. Beside the drive, the position round-trip +names the BasicHTTP processor, which the drive does not provide. Beside the +drive, the position round-trip property (`test/conformance/position_property.rb`), a test of the gate, takes a point after the start and after every step of each scion case, exports the position there, carries it through JSON, imports it into a fresh diff --git a/conformance/registry.json b/conformance/registry.json index 34e3391..ea7ff5c 100644 --- a/conformance/registry.json +++ b/conformance/registry.json @@ -123,6 +123,16 @@ {"case_id":"scion/targetless-transition/test2","suite":"scion"}, {"case_id":"scion/targetless-transition/test3","suite":"scion"}, {"case_id":"statifier/accepts/loan_declares_an_unreachable_event","suite":"statifier"}, + {"case_id":"statifier/diff/hold_queue_data_added","suite":"statifier"}, + {"case_id":"statifier/diff/hold_queue_identical","suite":"statifier"}, + {"case_id":"statifier/diff/hold_queue_renamed_with_mapping","suite":"statifier"}, + {"case_id":"statifier/diff/hold_queue_renamed_without_mapping","suite":"statifier"}, + {"case_id":"statifier/diff/hold_waiting_at_an_edited_state","suite":"statifier"}, + {"case_id":"statifier/diff/hold_waiting_through_an_idle_edit","suite":"statifier"}, + {"case_id":"statifier/diff/hold_waiting_under_an_edited_ancestor","suite":"statifier"}, + {"case_id":"statifier/diff/loan_waiting_across_a_damaged_arm","suite":"statifier"}, + {"case_id":"statifier/diff/loan_waiting_across_a_mapped_rename","suite":"statifier"}, + {"case_id":"statifier/diff/loan_waiting_across_a_regrouping","suite":"statifier"}, {"case_id":"statifier/library/hold_queue_available_with_no_holds","suite":"statifier"}, {"case_id":"statifier/library/hold_queue_collected_cancels_timer","suite":"statifier"}, {"case_id":"statifier/library/hold_queue_expiry_moves_to_next","suite":"statifier"}, @@ -130,6 +140,9 @@ {"case_id":"statifier/library/loan_dispute_restores_late_shallowly","suite":"statifier"}, {"case_id":"statifier/library/loan_dispute_returns_to_history","suite":"statifier"}, {"case_id":"statifier/library/loan_due_soon_then_overdue","suite":"statifier"}, + {"case_id":"statifier/library/loan_lost_after_timer","suite":"statifier"}, + {"case_id":"statifier/library/loan_position_counts_renewals","suite":"statifier"}, + {"case_id":"statifier/library/loan_position_records_history","suite":"statifier"}, {"case_id":"statifier/library/loan_renew_refused_at_max","suite":"statifier"}, {"case_id":"statifier/library/loan_renew_refused_by_hold","suite":"statifier"}, {"case_id":"statifier/library/loan_renew_refused_by_standing","suite":"statifier"}, @@ -138,6 +151,7 @@ {"case_id":"statifier/library/patron_checkout_allowed","suite":"statifier"}, {"case_id":"statifier/library/patron_checkout_refused_while_owed","suite":"statifier"}, {"case_id":"statifier/library/patron_initial_both_regions","suite":"statifier"}, + {"case_id":"statifier/library/patron_position_in_every_region","suite":"statifier"}, {"case_id":"statifier/library/patron_reinstated_after_block","suite":"statifier"}, {"case_id":"statifier/send/registered_delayed_cancel","suite":"statifier"}, {"case_id":"statifier/send/registered_immediate","suite":"statifier"}, diff --git a/test/conformance/drive_test.rb b/test/conformance/drive_test.rb index 7c13ece..f3d3faa 100644 --- a/test/conformance/drive_test.rb +++ b/test/conformance/drive_test.rb @@ -46,10 +46,10 @@ def test_the_drive_starts_the_chart_and_compares_the_active_leaves_after_each_st # sabotage: answer nil for a case with a host key (drop the `other` line in # Drive.unhonoured); the first assert_equal on the reason fails. def test_a_case_asking_what_the_runner_does_not_honour_fails_naming_it - item = chart_case(%(), %w[a]).merge("host" => { "send_types" => [], "expect_diff" => {}, + item = chart_case(%(), %w[a]).merge("host" => { "send_types" => [], "x_other" => {}, "x_unknown" => 1 }) - assert_equal "the case's host object (expect_diff, x_unknown) is not honoured here", + assert_equal "the case's host object (x_other, x_unknown) is not honoured here", Conformance::Drive.call(item) assert_match(/does not compile/, drive(" -> { {} } }) end - # sabotage: add `to_source` to HONOURED_HOST_KEYS; the assert_equal on the - # refusal fails (the key is then read and passes unnamed). + # sabotage: answer nil from Drive.unhonoured whenever a processor is + # provided (`return if provided.any?` before the `other` line); the + # assert_equal on the refusal fails (the case is then driven). def test_a_host_key_the_drive_does_not_read_is_refused_beside_a_provided_processor item = host_case([PROCESSOR]) - item["host"]["to_source"] = "" + item["host"]["x_unknown"] = "" - assert_equal "the case's host object (to_source) is not honoured here", + assert_equal "the case's host object (x_unknown) is not honoured here", Conformance::Drive.call(item, provided: { PROCESSOR => -> { {} } }) end diff --git a/test/conformance/gap_list_test.rb b/test/conformance/gap_list_test.rb index ddd7c88..f39af0a 100644 --- a/test/conformance/gap_list_test.rb +++ b/test/conformance/gap_list_test.rb @@ -23,7 +23,7 @@ def test_names_every_unclaimed_case_in_corpus_order_and_no_claimed_one ids = lines.map { |line| line.delete_prefix("conformance: gap: ").split.first } assert_equal UNCLAIMED.map { |item| item["id"] }, ids - assert_equal %w[w3c statifier], ids.map { |id| id.split("/").first }.uniq + assert_equal %w[w3c], ids.map { |id| id.split("/").first }.uniq end # sabotage: leave the features out of the line in gap_lines; the diff --git a/test/conformance/position_expectation.rb b/test/conformance/position_expectation.rb new file mode 100644 index 0000000..bceaff2 --- /dev/null +++ b/test/conformance/position_expectation.rb @@ -0,0 +1,177 @@ +# frozen_string_literal: true + +require "json" +require "predicator/tagged" +require_relative "../../lib/statifier" + +module Conformance + # The position a statifier case's step states in its `expect_position`, + # compared as the reference's harness compares it + # (`Mix.Statifier.Corpus.PositionExpectation` in + # lib/mix/statifier/corpus/position_expectation.ex at the tag the copy + # names): the state's position exported with {Statifier.export_position} + # and rendered as the JSON value a case writes, with exactly these seven + # members: + # + # - `configuration`, `entered_states` and `states_to_invoke`: each set of + # state ids as an array sorted ascending; + # - `history_values`: each history state's id to its recorded state ids, + # an array sorted ascending; + # - `active_invocations`: one object per live invocation, `state` (the id + # of the state that owns the ``) and `index` (its ordinal among + # that state's own), sorted by state and then by index, the invocation's + # id left out; + # - `running`: the position's boolean as it is; + # - `datamodel`: every variable but `_event`, `_ioprocessors` and + # `_sessionid`, and but `_name` too unless the document's `` + # element carries a `name` attribute, each value in its JSON form: a + # string, a number, a boolean and nil as they are, the undefined value + # as nil, a list as an array and a map as an object, member by member. + # A date, a datetime or a duration, or a list or map holding one, has no + # JSON form, and the step fails naming the variable. + # + # The position's other members (the three sequences, `status`, the + # counters, `trace` and the round budget) are not rendered, as the + # reference renders none of them. Values compare as JSON values: two + # numbers agree when they are equal in value, arrays element by element in + # order, objects member by member. + module PositionExpectation + # The system variables the rendering leaves out whatever the document. + LEFT_OUT_VARIABLES = %w[_event _ioprocessors _sessionid].freeze + + module_function + + # Whether any of item's steps carries an `expect_position`. + def expected?(item) = item.fetch("steps").any? { _1.key?("expect_position") } + + # nil when every state of machine but the root carries an id, else the + # reason naming the count of states without one: a case that expects a + # position needs every state named, since the export refuses a position + # holding a state it cannot name. + def named(machine) + unnamed = machine.states.drop(1).count { _1.id.nil? } + return if unnamed.zero? + + "a case that expects a position needs every state to carry an id, " \ + "and #{unnamed} state(s) of this document have none" + end + + # The rendering of state's exported position: `[rendering, nil]`, or + # `[nil, reason]` for the export's refusal or a datamodel value with no + # JSON form. + def render(state) + exported = Statifier.export_position(state) + return [nil, "the export refused the position: #{exported.error.reason} #{exported.error.detail}"] \ + unless exported.ok? + + position = exported.value + datamodel, reason = datamodel(position["datamodel"], state.machine_state.machine.name) + return [nil, reason] if reason + + [members(position, datamodel), nil] + end + + # The seven members, the datamodel already rendered. + def members(position, datamodel) + { "configuration" => position["configuration"].sort, + "entered_states" => position["entered_states"].sort, + "states_to_invoke" => position["states_to_invoke"].sort, + "history_values" => position["history_values"].transform_values(&:sort), + "active_invocations" => invocations(position["active_invocations"]), + "running" => position["running"], + "datamodel" => datamodel } + end + + # Each live invocation as its state and index, sorted. + def invocations(active) + active.map { { "state" => _1["state"], "index" => _1["invoke_index"] } }.sort_by { [_1["state"], _1["index"]] } + end + + # The datamodel's texts decoded and rendered: `[datamodel, nil]`, or + # `[nil, reason]` naming the first variable, by name, with no JSON form. + # `document_name` is the `` element's `name` attribute, nil + # without one. + def datamodel(texts, document_name) + left_out = document_name.is_a?(String) ? LEFT_OUT_VARIABLES : [*LEFT_OUT_VARIABLES, "_name"] + rendered = {} + (texts.keys - left_out).sort.each do |name| + value = Predicator::Tagged.decode!(texts[name]) + json = json_form(value) + return [nil, "the datamodel's #{name} holds a #{kind(value)} value, which has no JSON form here"] \ + if json.equal?(NONE) + + rendered[name] = json + end + [rendered, nil] + end + + # What json_form answers for a value with no JSON form. + NONE = Object.new.freeze + + # value's JSON form, or NONE. + def json_form(value) + case value + when Predicator::UNDEFINED, nil then nil + when String, Integer, Float, true, false then value + when Array then collect(value.map { json_form(_1) }) + when Hash then object(value) + else NONE + end + end + + # A map's JSON form, member by member in key order, or NONE. + def object(map) + keys = map.keys.sort + values = collect(keys.map { json_form(map[_1]) }) + values.equal?(NONE) ? NONE : keys.zip(values).to_h + end + + # items, or NONE when one of them is NONE. + def collect(items) = items.any? { _1.equal?(NONE) } ? NONE : items + + # The kind of a value with no JSON form, as a person reads it. + def kind(value) + case value + when Predicator::Date then "date" + when Predicator::DateTime then "datetime" + when Predicator::Duration then "duration" + when Array then "list" + when Hash then "map" + else value.class.name + end + end + + # nil when `step`, the one-based step `number`, states no position or + # state's agrees with it, else the reason. + def at_step(state, step, number) + return unless step.key?("expect_position") + + compare(step["expect_position"], state, number, step.fetch("event").fetch("name")) + end + + # nil when the rendering of state's position is exactly `expected`, a + # step's `expect_position`, else the reason: the step by its one-based + # `number` and its event's name, then each member that differs with both + # values as JSON, or why the position cannot be rendered. + def compare(expected, state, number, event) + label = "after step #{number} (#{event}), expect_position" + actual, reason = render(state) + return "#{label} cannot be compared: #{reason}" if reason + + differing = differing(expected.is_a?(Hash) ? expected : {}, actual) + "#{label} differs: #{differing}" unless differing.nil? + end + + # Each member of the two that differs, with both values, in member + # order; nil when none does. + def differing(wanted, actual) + keys = (wanted.keys | actual.keys).sort.reject { wanted[_1] == actual[_1] } + return if keys.empty? + + keys.map { "#{_1}: expected #{encode(wanted[_1])}, but got #{encode(actual[_1])}" }.join("; ") + end + + # A member's value as the reason writes it: nothing for an absent one. + def encode(value) = value.nil? ? "nothing" : JSON.generate(value) + end +end diff --git a/test/conformance/position_expectation_test.rb b/test/conformance/position_expectation_test.rb new file mode 100644 index 0000000..e45e1de --- /dev/null +++ b/test/conformance/position_expectation_test.rb @@ -0,0 +1,175 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "runner" + +# A step's expect_position (Conformance::PositionExpectation in +# test/conformance/position_expectation.rb): the exported position rendered +# as the reference renders it, compared once the step's configuration +# agrees, and the cases the drive refuses before it starts. +class PositionExpectationTest < Minitest::Test + ROOT = %(xmlns="http://www.w3.org/2005/07/scxml" version="1.0" datamodel="predicator") + + # A chart that records history on leaving a, takes the event's data into + # v on e.set, and holds a nested value in m. + HISTORIED = <<~XML + + + + + + + + XML + + def compiled(body, attributes = "") = Statifier.compile(%(#{body})).value + + def moved(result) = result.value.state + + def started(body, attributes = "") = moved(Statifier.start(compiled(body, attributes), session_id: "s1")) + + def render(state) = Conformance::PositionExpectation.render(state) + + # sabotage: render `_sessionid` (drop it from LEFT_OUT_VARIABLES); the + # assert_equal on the rendering fails. + # sabotage: render each history list as `[]` in + # PositionExpectation.members; the same assert_equal fails. + def test_the_rendering_has_exactly_the_seven_members_with_every_id_set_sorted + state = moved(Statifier.step(started(HISTORIED), "e.go")) + rendering, reason = render(state) + + assert_nil reason + assert_equal({ "configuration" => %w[b], "entered_states" => %w[a a2 b], "states_to_invoke" => [], + "history_values" => { "h" => %w[a2] }, "active_invocations" => [], "running" => true, + "datamodel" => { "m" => { "a" => "x", "z" => [1, nil] }, "v" => 0 } }, rendering) + end + + # A chart whose two parallel regions each hold an invocation, b's + # written first, and b holding two. + INVOKING = <<~XML.freeze + + + + + + + + XML + + # sabotage: render the invoke id as the index in + # PositionExpectation.invocations (`"index" => _1["invoke_id"]`); the + # assert_equal fails. + def test_each_active_invocation_is_its_state_and_index_sorted_without_its_id + rendering, = render(started(INVOKING)) + + assert_equal [{ "state" => "a", "index" => 0 }, { "state" => "b", "index" => 0 }, { "state" => "b", "index" => 1 }], + rendering["active_invocations"] + end + + # sabotage: always leave `_name` out (drop the `document_name.is_a?` + # branch in PositionExpectation.datamodel); the second assert_equal fails. + # sabotage: never leave `_name` out; the first assert_equal fails. + def test_name_is_rendered_only_when_the_scxml_element_carries_a_name_attribute + unnamed, = render(started(%())) + named, = render(started(%(), %( name="chart.one"))) + + assert_equal({}, unnamed["datamodel"]) + assert_equal({ "_name" => "chart.one" }, named["datamodel"]) + end + + # The state HISTORIED leaves with `value` in v. + def holding(value) = moved(Statifier.step(moved(Statifier.step(started(HISTORIED), "e.go")), "e.set", data: value)) + + NO_FORM = { "date" => Predicator::Date.new(year: 2026, month: 1, day: 2), + "datetime" => Predicator::DateTime.new(epoch_seconds: 0, microsecond: 0), + "duration" => Predicator::Duration.new(years: 0, months: 0, weeks: 0, days: 0, hours: 1, minutes: 0, + seconds: 0, milliseconds: 0) }.freeze + + # sabotage: answer a date as its tagged map in PositionExpectation.json_form + # (`when Predicator::Date then value.to_h`); the first assert_equal fails. + def test_a_date_datetime_or_duration_value_fails_naming_the_variable + NO_FORM.each do |kind, value| + reason = "the datamodel's v holds a #{kind} value, which has no JSON form here" + + assert_equal [nil, reason], render(holding(value)), kind + end + + assert_equal "the datamodel's v holds a list value, which has no JSON form here", + render(holding([NO_FORM["date"]]))[1] + end + + # sabotage: answer nil from PositionExpectation.compare whatever differs + # (`return if true`); the assert_equal on the reason fails. + def test_a_differing_member_is_named_with_both_values + state = started(%()) + expected = { "configuration" => %w[b], "entered_states" => %w[a], "states_to_invoke" => [], + "history_values" => {}, "active_invocations" => [], "running" => true, "datamodel" => {} } + + assert_equal 'after step 2 (e.go), expect_position differs: configuration: expected ["b"], but got ["a"]', + Conformance::PositionExpectation.compare(expected, state, 2, "e.go") + assert_nil Conformance::PositionExpectation.compare(expected.merge("configuration" => %w[a]), state, 2, "e.go") + assert_match(/states_to_invoke: expected nothing, but got \[\]/, + Conformance::PositionExpectation.compare(expected.except("states_to_invoke"), state, 1, "e")) + end +end + +# The drive's side of a step's expect_position (Conformance::Drive in +# test/conformance/runner.rb): the step's data sent, the position compared, +# a document with an unnamed state refused before the drive, and the diff +# pair accepted and compared by nothing. +class PositionExpectationDriveTest < Minitest::Test + ROOT = PositionExpectationTest::ROOT + + # A statifier case around body, its one step event e.go with `data`, the + # step stating `position`. + def position_case(body, position, data: nil) + event = { "name" => "e.go" } + event["data"] = data unless data.nil? + { "id" => "statifier/x/y", "source" => %(#{body}), "initial_configuration" => %w[a], + "steps" => [{ "event" => event, "configuration" => %w[b], "expect_position" => position }] } + end + + CARRY = <<~XML + + + + XML + + def carried(value) + { "configuration" => %w[b], "entered_states" => %w[a b], "states_to_invoke" => [], "history_values" => {}, + "active_invocations" => [], "running" => true, "datamodel" => { "v" => value } } + end + + # sabotage: send the name alone for a case with no host object (`data = + # item.key?("host")` in Drive.drive); the first assert_equal fails. + # sabotage: drop the position comparison from Drive.drive_step (`[state, + # failed]`); the second assert_equal fails. + def test_a_case_with_no_host_stating_a_position_sends_the_step_data_and_compares_the_position + assert_equal :pass, Conformance::Drive.call(position_case(CARRY, carried(7), data: 7)) + assert_equal "after step 1 (e.go), expect_position differs: datamodel: expected {\"v\":8}, but got {\"v\":7}", + Conformance::Drive.call(position_case(CARRY, carried(8), data: 7)) + end + + # sabotage: answer nil from PositionExpectation.named; the assert_equal + # on the reason fails (the case is then driven). + def test_a_case_stating_a_position_on_a_document_with_an_unnamed_state_fails_before_it_is_driven + body = %() + + assert_equal "a case that expects a position needs every state to carry an id, " \ + "and 1 state(s) of this document have none", + Conformance::Drive.call(position_case(body, carried(0))) + end + + # sabotage: drop `to_source` from UNCOMPARED_HOST_KEYS; the assert_equal + # fails (the case is then refused naming the key). + def test_a_case_carrying_a_diff_pair_is_driven_and_the_pair_is_compared_by_nothing + item = position_case(CARRY, carried(0)).except("steps").merge( + "steps" => [{ "event" => { "name" => "e.go" }, "configuration" => %w[b] }], + "host" => { "send_types" => [], "expect_sends" => [], "to_source" => "not a document", + "expect_diff" => { "anything" => true }, "mapping" => { "x" => "y" }, + "expect_compatible_at" => ["nowhere"] } + ) + + assert_equal :pass, Conformance::Drive.call(item) + end +end diff --git a/test/conformance/runner.rb b/test/conformance/runner.rb index 5a8bf4b..f175d1a 100644 --- a/test/conformance/runner.rb +++ b/test/conformance/runner.rb @@ -3,6 +3,7 @@ require_relative "corpus" require_relative "host" require_relative "accepts" +require_relative "position_expectation" require_relative "../../lib/statifier" module Conformance @@ -85,8 +86,9 @@ def after(result) # above, from the start of the chart; once every configuration agrees, # the sends handed to it over the whole drive, in order and each with its # outcome, must be exactly `expect_sends`. A step of a case with a host - # object sends its event's `data`, when it gives one, as the harness - # injects it. A send whose expected item asks the host to report it + # object, or of a case stating an exported position, sends its event's + # `data`, when it gives one, as the harness injects it (the harness runs + # both kinds of case through `run/2`). A send whose expected item asks the host to report it # failed (`"outcome": "fail"`) is reported through # {Statifier.report_send_failed} once each driver call that handed it # returns, before the configuration is read, as the harness reports it @@ -102,11 +104,20 @@ def after(result) # included. Either key without the other fails the case, naming the one # present. # - # Any other host key (the statifier suite's `to_source`, `expect_diff`, - # `mapping` and `expect_compatible_at`, and any key a later corpus adds) - # is not honoured here and fails the case naming the key. A case stating - # an exported position asks for what this runner does not compare yet, - # and fails with that named. + # A host object's diff pair, `to_source` and `expect_diff` with an + # optional `mapping` and `expect_compatible_at`, is accepted and compared + # by nothing here: a case carrying it is driven like any other host case, + # as the reference's harness drives it (its moduledoc in host_case.ex: + # "this runner compares none of them"). Any other host key (any key a + # later corpus adds) is not honoured here and fails the case naming the + # key. + # + # A step's `expect_position` is compared, once that step's configuration + # agrees, with the rendering of the state's exported position + # (Conformance::PositionExpectation, test/conformance/position_expectation.rb), + # as the harness compares it (`position/3` in host_case.ex). A case that + # states one on any step needs every state of its document to carry an id, + # and fails before it is driven when one has none (`named/2` there). module Drive # How long, in virtual ms, the clock may run before an event to drain # short timers. @@ -124,8 +135,12 @@ module Drive # its entry here and nowhere else. PROVIDED = {}.freeze - # The host keys the drive reads; any other is not honoured. - HONOURED_HOST_KEYS = %w[event_io_processors send_types expect_sends declared_events expect_accepts].freeze + # The diff pair's host keys: accepted, and compared by nothing here. + UNCOMPARED_HOST_KEYS = %w[to_source expect_diff mapping expect_compatible_at].freeze + + # The host keys the drive reads or accepts; any other is not honoured. + HONOURED_HOST_KEYS = (%w[event_io_processors send_types expect_sends declared_events expect_accepts] + + UNCOMPARED_HOST_KEYS).freeze # The answer for item: :pass, or the reason it failed. provided is the # table of processors the drive can register (PROVIDED, unless a test @@ -137,20 +152,23 @@ def self.call(item, provided: PROVIDED) machine, failed = Accepts.prepared(item) return failed if failed + unnamed = PositionExpectation.expected?(item) && PositionExpectation.named(machine) + return unnamed if unnamed + host = Host.for(item) driven = drive(item, machine, host.processors.merge(registered(item, provided)), host) driven == :pass ? host.verdict : driven end # What the case asks of its host that this runner does not honour, or - # nil: a host key it does not read, a processor `provided` holds no - # entry for, or an exported position. + # nil: a host key it neither reads nor accepts, or a processor + # `provided` holds no entry for. def self.unhonoured(item, provided = PROVIDED) host = item.fetch("host", {}) other = host.keys - HONOURED_HOST_KEYS return "the case's host object (#{other.join(", ")}) is not honoured here" unless other.empty? - not_provided(host, provided) || position(item) + not_provided(host, provided) end # The processors the host object names that `provided` holds no entry @@ -163,13 +181,6 @@ def self.not_provided(host, provided) "which this package does not provide" end - # The reason for a case stating an exported position, or nil. - def self.position(item) - return unless item.fetch("steps").any? { _1.key?("expect_position") } - - "a step states an exported position, which this runner does not compare yet" - end - # The URIs a host object's `event_io_processors` names, in order; a value # that is not a list is named whole, as it came, so it is refused. def self.named(host) @@ -192,17 +203,20 @@ def self.drive(item, machine, processors = {}, host = nil) failed = compare(state, item.fetch("initial_configuration"), "the initial configuration") return failed if failed + data = item.key?("host") || PositionExpectation.expected?(item) item.fetch("steps").each_with_index do |step, index| - state, failed = drive_step(state, step, index, calls, host: item.key?("host")) + state, failed = drive_step(state, step, index, calls, host: data) return failed if failed end :pass end # One step: the clock settled, the event sent, the expectation awaited - # and compared. `[state, reason]`, the reason nil when it agrees. A step - # of a case with a host object (`host`) sends its event's `data` when it - # gives one; any other sends the name alone. + # and compared, and then the step's exported position when it states + # one. `[state, reason]`, the reason nil when it agrees. A step of a + # case the harness runs as a host case (`host`: one with a host object + # or stating a position) sends its event's `data` when it gives one; + # any other sends the name alone. def self.drive_step(state, step, index, calls = Calls.plain, host: false) state = settle(state, calls) since = state.now_ms @@ -210,7 +224,8 @@ def self.drive_step(state, step, index, calls = Calls.plain, host: false) name = event.fetch("name") data = host ? event.fetch("data", Predicator::UNDEFINED) : Predicator::UNDEFINED state = await(calls.step(state, name, data), step.fetch("configuration"), since, calls) - [state, compare(state, step.fetch("configuration"), "step #{index + 1} (event #{name.inspect})")] + failed = compare(state, step.fetch("configuration"), "step #{index + 1} (event #{name.inspect})") + [state, failed || PositionExpectation.at_step(state, step, index + 1)] end # The state a call answered; a refused call is a bug here, since every diff --git a/test/conformance/statifier_suite_test.rb b/test/conformance/statifier_suite_test.rb index 2b5644d..a8f41b5 100644 --- a/test/conformance/statifier_suite_test.rb +++ b/test/conformance/statifier_suite_test.rb @@ -4,49 +4,46 @@ require_relative "runner" # The statifier suite, the reference's own cases, as the drive answers it: -# every case of its accepts, library, send and system_variables groups -# passes but the ones that ask for what the runner does not do yet, and -# each of those, and every case of the other groups, fails with that -# named. +# every case of every group passes, the diff cases driven with their diff +# pair compared by nothing and the cases stating an exported position +# compared on it. class StatifierSuiteTest < Minitest::Test CORPUS = Conformance::Corpus.load CASES = CORPUS.cases_of("statifier") RESULTS = CASES.to_h { |item| [item["id"], Conformance::Drive.call(item)] } - POSITION = "a step states an exported position, which this runner does not compare yet" - - def group(item) = item["id"].split("/")[1] - def position?(item) = item["steps"].any? { _1.key?("expect_position") } - # sabotage: compare the handed sends in reverse in Host#verdict; the - # assert_equal on the passing ids fails (a case handed two sends or more - # then disagrees). - # sabotage: drop the exported-position refusal from Drive.unhonoured; the - # same assert_equal fails (the position cases are then driven and pass, - # their positions uncompared). + # assert_equal fails (a case handed two sends or more then disagrees). # sabotage: answer the accepts check with its two lists swapped in - # Accepts.compare; the same assert_equal fails (the accepts case then + # Accepts.compare; the assert_equal fails (the accepts case then # disagrees). - def test_every_accepts_library_send_and_system_variables_case_passes_but_what_the_runner_does_not_do_yet - groups = CASES.select { %w[accepts library send system_variables].include?(group(_1)) } - passing = groups.reject { position?(_1) }.map { _1["id"] } - - assert_equal passing, RESULTS.select { _2 == :pass }.keys - refute_empty passing + # sabotage: render `_name` whatever the document (drop it from the + # left-out list in PositionExpectation.datamodel); the assert_equal fails + # (the four cases stating a position then disagree on their datamodel). + def test_every_case_of_the_suite_passes + assert_equal CASES.to_h { [_1["id"], :pass] }, RESULTS + assert_equal 36, CASES.size end - # sabotage: reword the reason Drive.position answers; the assert_equal on - # the reason of a case stating an exported position fails. - def test_each_case_left_fails_naming_what_it_asks_for - CASES.reject { RESULTS[_1["id"]] == :pass }.each do |item| - assert_equal expected_reason(item), RESULTS[item["id"]], item["id"] - end + # The first case stating a position. + POSITIONED = CASES.find { _1["steps"].any? { |step| step.key?("expect_position") } } + + # That case, its first such step's `running` flipped: `[case, step + # number, the flag the case states now]`. + def flipped + altered = Marshal.load(Marshal.dump(POSITIONED)) + index = altered["steps"].index { _1.key?("expect_position") } + position = altered["steps"][index]["expect_position"] + position["running"] = !position["running"] + [altered, index + 1, position["running"]] end - def expected_reason(item) - other = item.fetch("host", {}).keys - Conformance::Drive::HONOURED_HOST_KEYS - return "the case's host object (#{other.join(", ")}) is not honoured here" unless other.empty? + # sabotage: answer nil from PositionExpectation.compare (`return if + # true`); the assert_match on the reason fails. + def test_a_case_stating_a_position_the_chart_does_not_hold_fails_naming_the_member + altered, number, running = flipped + differs = "expect_position differs: running: expected #{running}, but got #{!running}" - POSITION + assert_match(/\Aafter step #{number} \(.*\), #{differs}\z/, Conformance::Drive.call(altered)) end end