diff --git a/README.md b/README.md index b043c51..34a6112 100644 --- a/README.md +++ b/README.md @@ -345,11 +345,11 @@ claims to pass, and the gate fails if one of them does not. | Claim | Cases claimed | Of | |---|---|---| | `scion` | 119 | 119 cases in the scion suite | -| `statifier` | 22 | 36 cases in the statifier suite | +| `statifier` | 36 | 36 cases in the statifier suite | | `w3c-mandatory` | 152 | 154 mandatory cases in the w3c suite | | `w3c-optional` | 2 | 14 optional cases in the w3c suite | -The registry holds 295 entries, which is the sum of the four rows. A claim is +The registry holds 309 entries, which is the sum of the four rows. A claim is the set of its entries, not a statement that every case of its suite passes. What is not claimed yet: @@ -358,9 +358,6 @@ What is not claimed yet: the reference's registry does not claim them either. - Twelve optional w3c cases whose host needs the Basic HTTP Event I/O Processor, which this package does not provide. -- Fourteen cases of the `statifier` suite, the reference's own: the ten diff - cases, whose host keys the conformance runner does not read yet; and the four - cases that state an exported position, which it does not compare yet. Running the gate prints the gap list: every unclaimed case with the features it needs and whether it passes or why it fails. diff --git a/conformance/README.md b/conformance/README.md index 9c67a0c..b26e164 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -103,27 +103,43 @@ once every configuration agrees, the sends handed over the whole drive, in order and each with its outcome, must be exactly `expect_sends`. A `` that reaches the host marks the delayed send it names `"outcome": "cancelled"` when the expected item at that send's position -asks for it, and a step of a case with a host object sends its event's -`data`, when it gives one. A send whose expected item asks the host to +asks for it, and a step of a case with a host object, or of a case stating +an exported position, sends its event's `data`, when it gives one. A send +whose expected item asks the host to report it failed (`"outcome": "fail"`) is marked so and reported through `Statifier.report_send_failed` once the driver call that handed it returns, before the configuration is read, as the reference's harness reports it before it reads the next configuration; a report the driver refuses because -the chart has stopped is dropped. Any other host key (the accepts check's -`declared_events` and `expect_accepts`, the diff pair's `to_source`, -`expect_diff`, `mapping` and `expect_compatible_at`), and a step stating an -exported position each fail the case naming what it asks for. This +the chart has stopped is dropped. A case's `host.declared_events` and +`host.expect_accepts` are read as the reference's harness reads them: +`Statifier.check_accepts` answers for the compiled chart before it starts, +and its two lists must be exactly `expect_accepts`. A host object's diff +pair, `to_source` and `expect_diff` with an optional `mapping` and +`expect_compatible_at`, is compared by nothing here, as the reference's +harness compares none of the four: a case carrying it is driven like any +other host case, and agrees on its configurations and its sends. A step's +`expect_position` is compared, once that step's configuration agrees, with +the state's position exported through `Statifier.export_position` and +rendered as the reference's harness renders it +(`Conformance::PositionExpectation` in +`test/conformance/position_expectation.rb`): the configuration, the entered +states and the states to invoke, each sorted; each history state's recorded +states, sorted; each active invocation as its state and index, sorted; the +running flag; and the datamodel without `_event`, `_ioprocessors` and +`_sessionid`, and without `_name` unless the `` element carries a +`name` attribute, each value in its JSON form. A date, a datetime or a +duration has no JSON form and fails the step naming its variable, and a +case stating a position on a document with a state that has no id fails +before it is driven. Any other host key fails the case naming it. This registry claims the scion suite, every one of its cases, the w3c cases the -run passes and the statifier cases the run passes: the `w3c-mandatory` +run passes and every case of the statifier suite: the `w3c-mandatory` claim covers its mandatory entries, the `w3c-optional` claim its optional ones and the `statifier` claim its statifier entries, since a claim is the set of its entries (the copy's `RATCHET.md`, "Claims"), not a statement that every case of the suite passes. The w3c cases left out are the two mandatory cases the run fails and the optional cases whose host object -names the BasicHTTP processor, which the drive does not provide. The -statifier cases left out are the ones that fail for what the drive does -not do yet (the gap list names each with its reason); each joins the claim -when the ratchet sees it pass. Beside the drive, the position round-trip +names the BasicHTTP processor, which the drive does not provide. Beside the +drive, the position round-trip property (`test/conformance/position_property.rb`), a test of the gate, takes a point after the start and after every step of each scion case, exports the position there, carries it through JSON, imports it into a fresh diff --git a/conformance/registry.json b/conformance/registry.json index 34e3391..ea7ff5c 100644 --- a/conformance/registry.json +++ b/conformance/registry.json @@ -123,6 +123,16 @@ {"case_id":"scion/targetless-transition/test2","suite":"scion"}, {"case_id":"scion/targetless-transition/test3","suite":"scion"}, {"case_id":"statifier/accepts/loan_declares_an_unreachable_event","suite":"statifier"}, + {"case_id":"statifier/diff/hold_queue_data_added","suite":"statifier"}, + {"case_id":"statifier/diff/hold_queue_identical","suite":"statifier"}, + {"case_id":"statifier/diff/hold_queue_renamed_with_mapping","suite":"statifier"}, + {"case_id":"statifier/diff/hold_queue_renamed_without_mapping","suite":"statifier"}, + {"case_id":"statifier/diff/hold_waiting_at_an_edited_state","suite":"statifier"}, + {"case_id":"statifier/diff/hold_waiting_through_an_idle_edit","suite":"statifier"}, + {"case_id":"statifier/diff/hold_waiting_under_an_edited_ancestor","suite":"statifier"}, + {"case_id":"statifier/diff/loan_waiting_across_a_damaged_arm","suite":"statifier"}, + {"case_id":"statifier/diff/loan_waiting_across_a_mapped_rename","suite":"statifier"}, + {"case_id":"statifier/diff/loan_waiting_across_a_regrouping","suite":"statifier"}, {"case_id":"statifier/library/hold_queue_available_with_no_holds","suite":"statifier"}, {"case_id":"statifier/library/hold_queue_collected_cancels_timer","suite":"statifier"}, {"case_id":"statifier/library/hold_queue_expiry_moves_to_next","suite":"statifier"}, @@ -130,6 +140,9 @@ {"case_id":"statifier/library/loan_dispute_restores_late_shallowly","suite":"statifier"}, {"case_id":"statifier/library/loan_dispute_returns_to_history","suite":"statifier"}, {"case_id":"statifier/library/loan_due_soon_then_overdue","suite":"statifier"}, + {"case_id":"statifier/library/loan_lost_after_timer","suite":"statifier"}, + {"case_id":"statifier/library/loan_position_counts_renewals","suite":"statifier"}, + {"case_id":"statifier/library/loan_position_records_history","suite":"statifier"}, {"case_id":"statifier/library/loan_renew_refused_at_max","suite":"statifier"}, {"case_id":"statifier/library/loan_renew_refused_by_hold","suite":"statifier"}, {"case_id":"statifier/library/loan_renew_refused_by_standing","suite":"statifier"}, @@ -138,6 +151,7 @@ {"case_id":"statifier/library/patron_checkout_allowed","suite":"statifier"}, {"case_id":"statifier/library/patron_checkout_refused_while_owed","suite":"statifier"}, {"case_id":"statifier/library/patron_initial_both_regions","suite":"statifier"}, + {"case_id":"statifier/library/patron_position_in_every_region","suite":"statifier"}, {"case_id":"statifier/library/patron_reinstated_after_block","suite":"statifier"}, {"case_id":"statifier/send/registered_delayed_cancel","suite":"statifier"}, {"case_id":"statifier/send/registered_immediate","suite":"statifier"}, diff --git a/test/conformance/drive_test.rb b/test/conformance/drive_test.rb index 7c13ece..f3d3faa 100644 --- a/test/conformance/drive_test.rb +++ b/test/conformance/drive_test.rb @@ -46,10 +46,10 @@ def test_the_drive_starts_the_chart_and_compares_the_active_leaves_after_each_st # sabotage: answer nil for a case with a host key (drop the `other` line in # Drive.unhonoured); the first assert_equal on the reason fails. def test_a_case_asking_what_the_runner_does_not_honour_fails_naming_it - item = chart_case(%(), %w[a]).merge("host" => { "send_types" => [], "expect_diff" => {}, + item = chart_case(%(), %w[a]).merge("host" => { "send_types" => [], "x_other" => {}, "x_unknown" => 1 }) - assert_equal "the case's host object (expect_diff, x_unknown) is not honoured here", + assert_equal "the case's host object (x_other, x_unknown) is not honoured here", Conformance::Drive.call(item) assert_match(/does not compile/, drive(" -> { {} } }) end - # sabotage: add `to_source` to HONOURED_HOST_KEYS; the assert_equal on the - # refusal fails (the key is then read and passes unnamed). + # sabotage: answer nil from Drive.unhonoured whenever a processor is + # provided (`return if provided.any?` before the `other` line); the + # assert_equal on the refusal fails (the case is then driven). def test_a_host_key_the_drive_does_not_read_is_refused_beside_a_provided_processor item = host_case([PROCESSOR]) - item["host"]["to_source"] = "" + item["host"]["x_unknown"] = "" - assert_equal "the case's host object (to_source) is not honoured here", + assert_equal "the case's host object (x_unknown) is not honoured here", Conformance::Drive.call(item, provided: { PROCESSOR => -> { {} } }) end diff --git a/test/conformance/gap_list_test.rb b/test/conformance/gap_list_test.rb index ddd7c88..f39af0a 100644 --- a/test/conformance/gap_list_test.rb +++ b/test/conformance/gap_list_test.rb @@ -23,7 +23,7 @@ def test_names_every_unclaimed_case_in_corpus_order_and_no_claimed_one ids = lines.map { |line| line.delete_prefix("conformance: gap: ").split.first } assert_equal UNCLAIMED.map { |item| item["id"] }, ids - assert_equal %w[w3c statifier], ids.map { |id| id.split("/").first }.uniq + assert_equal %w[w3c], ids.map { |id| id.split("/").first }.uniq end # sabotage: leave the features out of the line in gap_lines; the diff --git a/test/conformance/position_expectation.rb b/test/conformance/position_expectation.rb new file mode 100644 index 0000000..bceaff2 --- /dev/null +++ b/test/conformance/position_expectation.rb @@ -0,0 +1,177 @@ +# frozen_string_literal: true + +require "json" +require "predicator/tagged" +require_relative "../../lib/statifier" + +module Conformance + # The position a statifier case's step states in its `expect_position`, + # compared as the reference's harness compares it + # (`Mix.Statifier.Corpus.PositionExpectation` in + # lib/mix/statifier/corpus/position_expectation.ex at the tag the copy + # names): the state's position exported with {Statifier.export_position} + # and rendered as the JSON value a case writes, with exactly these seven + # members: + # + # - `configuration`, `entered_states` and `states_to_invoke`: each set of + # state ids as an array sorted ascending; + # - `history_values`: each history state's id to its recorded state ids, + # an array sorted ascending; + # - `active_invocations`: one object per live invocation, `state` (the id + # of the state that owns the ``) and `index` (its ordinal among + # that state's own), sorted by state and then by index, the invocation's + # id left out; + # - `running`: the position's boolean as it is; + # - `datamodel`: every variable but `_event`, `_ioprocessors` and + # `_sessionid`, and but `_name` too unless the document's `` + # element carries a `name` attribute, each value in its JSON form: a + # string, a number, a boolean and nil as they are, the undefined value + # as nil, a list as an array and a map as an object, member by member. + # A date, a datetime or a duration, or a list or map holding one, has no + # JSON form, and the step fails naming the variable. + # + # The position's other members (the three sequences, `status`, the + # counters, `trace` and the round budget) are not rendered, as the + # reference renders none of them. Values compare as JSON values: two + # numbers agree when they are equal in value, arrays element by element in + # order, objects member by member. + module PositionExpectation + # The system variables the rendering leaves out whatever the document. + LEFT_OUT_VARIABLES = %w[_event _ioprocessors _sessionid].freeze + + module_function + + # Whether any of item's steps carries an `expect_position`. + def expected?(item) = item.fetch("steps").any? { _1.key?("expect_position") } + + # nil when every state of machine but the root carries an id, else the + # reason naming the count of states without one: a case that expects a + # position needs every state named, since the export refuses a position + # holding a state it cannot name. + def named(machine) + unnamed = machine.states.drop(1).count { _1.id.nil? } + return if unnamed.zero? + + "a case that expects a position needs every state to carry an id, " \ + "and #{unnamed} state(s) of this document have none" + end + + # The rendering of state's exported position: `[rendering, nil]`, or + # `[nil, reason]` for the export's refusal or a datamodel value with no + # JSON form. + def render(state) + exported = Statifier.export_position(state) + return [nil, "the export refused the position: #{exported.error.reason} #{exported.error.detail}"] \ + unless exported.ok? + + position = exported.value + datamodel, reason = datamodel(position["datamodel"], state.machine_state.machine.name) + return [nil, reason] if reason + + [members(position, datamodel), nil] + end + + # The seven members, the datamodel already rendered. + def members(position, datamodel) + { "configuration" => position["configuration"].sort, + "entered_states" => position["entered_states"].sort, + "states_to_invoke" => position["states_to_invoke"].sort, + "history_values" => position["history_values"].transform_values(&:sort), + "active_invocations" => invocations(position["active_invocations"]), + "running" => position["running"], + "datamodel" => datamodel } + end + + # Each live invocation as its state and index, sorted. + def invocations(active) + active.map { { "state" => _1["state"], "index" => _1["invoke_index"] } }.sort_by { [_1["state"], _1["index"]] } + end + + # The datamodel's texts decoded and rendered: `[datamodel, nil]`, or + # `[nil, reason]` naming the first variable, by name, with no JSON form. + # `document_name` is the `` element's `name` attribute, nil + # without one. + def datamodel(texts, document_name) + left_out = document_name.is_a?(String) ? LEFT_OUT_VARIABLES : [*LEFT_OUT_VARIABLES, "_name"] + rendered = {} + (texts.keys - left_out).sort.each do |name| + value = Predicator::Tagged.decode!(texts[name]) + json = json_form(value) + return [nil, "the datamodel's #{name} holds a #{kind(value)} value, which has no JSON form here"] \ + if json.equal?(NONE) + + rendered[name] = json + end + [rendered, nil] + end + + # What json_form answers for a value with no JSON form. + NONE = Object.new.freeze + + # value's JSON form, or NONE. + def json_form(value) + case value + when Predicator::UNDEFINED, nil then nil + when String, Integer, Float, true, false then value + when Array then collect(value.map { json_form(_1) }) + when Hash then object(value) + else NONE + end + end + + # A map's JSON form, member by member in key order, or NONE. + def object(map) + keys = map.keys.sort + values = collect(keys.map { json_form(map[_1]) }) + values.equal?(NONE) ? NONE : keys.zip(values).to_h + end + + # items, or NONE when one of them is NONE. + def collect(items) = items.any? { _1.equal?(NONE) } ? NONE : items + + # The kind of a value with no JSON form, as a person reads it. + def kind(value) + case value + when Predicator::Date then "date" + when Predicator::DateTime then "datetime" + when Predicator::Duration then "duration" + when Array then "list" + when Hash then "map" + else value.class.name + end + end + + # nil when `step`, the one-based step `number`, states no position or + # state's agrees with it, else the reason. + def at_step(state, step, number) + return unless step.key?("expect_position") + + compare(step["expect_position"], state, number, step.fetch("event").fetch("name")) + end + + # nil when the rendering of state's position is exactly `expected`, a + # step's `expect_position`, else the reason: the step by its one-based + # `number` and its event's name, then each member that differs with both + # values as JSON, or why the position cannot be rendered. + def compare(expected, state, number, event) + label = "after step #{number} (#{event}), expect_position" + actual, reason = render(state) + return "#{label} cannot be compared: #{reason}" if reason + + differing = differing(expected.is_a?(Hash) ? expected : {}, actual) + "#{label} differs: #{differing}" unless differing.nil? + end + + # Each member of the two that differs, with both values, in member + # order; nil when none does. + def differing(wanted, actual) + keys = (wanted.keys | actual.keys).sort.reject { wanted[_1] == actual[_1] } + return if keys.empty? + + keys.map { "#{_1}: expected #{encode(wanted[_1])}, but got #{encode(actual[_1])}" }.join("; ") + end + + # A member's value as the reason writes it: nothing for an absent one. + def encode(value) = value.nil? ? "nothing" : JSON.generate(value) + end +end diff --git a/test/conformance/position_expectation_test.rb b/test/conformance/position_expectation_test.rb new file mode 100644 index 0000000..e45e1de --- /dev/null +++ b/test/conformance/position_expectation_test.rb @@ -0,0 +1,175 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "runner" + +# A step's expect_position (Conformance::PositionExpectation in +# test/conformance/position_expectation.rb): the exported position rendered +# as the reference renders it, compared once the step's configuration +# agrees, and the cases the drive refuses before it starts. +class PositionExpectationTest < Minitest::Test + ROOT = %(xmlns="http://www.w3.org/2005/07/scxml" version="1.0" datamodel="predicator") + + # A chart that records history on leaving a, takes the event's data into + # v on e.set, and holds a nested value in m. + HISTORIED = <<~XML + + + + + + + + XML + + def compiled(body, attributes = "") = Statifier.compile(%(#{body})).value + + def moved(result) = result.value.state + + def started(body, attributes = "") = moved(Statifier.start(compiled(body, attributes), session_id: "s1")) + + def render(state) = Conformance::PositionExpectation.render(state) + + # sabotage: render `_sessionid` (drop it from LEFT_OUT_VARIABLES); the + # assert_equal on the rendering fails. + # sabotage: render each history list as `[]` in + # PositionExpectation.members; the same assert_equal fails. + def test_the_rendering_has_exactly_the_seven_members_with_every_id_set_sorted + state = moved(Statifier.step(started(HISTORIED), "e.go")) + rendering, reason = render(state) + + assert_nil reason + assert_equal({ "configuration" => %w[b], "entered_states" => %w[a a2 b], "states_to_invoke" => [], + "history_values" => { "h" => %w[a2] }, "active_invocations" => [], "running" => true, + "datamodel" => { "m" => { "a" => "x", "z" => [1, nil] }, "v" => 0 } }, rendering) + end + + # A chart whose two parallel regions each hold an invocation, b's + # written first, and b holding two. + INVOKING = <<~XML.freeze + + + + + + + + XML + + # sabotage: render the invoke id as the index in + # PositionExpectation.invocations (`"index" => _1["invoke_id"]`); the + # assert_equal fails. + def test_each_active_invocation_is_its_state_and_index_sorted_without_its_id + rendering, = render(started(INVOKING)) + + assert_equal [{ "state" => "a", "index" => 0 }, { "state" => "b", "index" => 0 }, { "state" => "b", "index" => 1 }], + rendering["active_invocations"] + end + + # sabotage: always leave `_name` out (drop the `document_name.is_a?` + # branch in PositionExpectation.datamodel); the second assert_equal fails. + # sabotage: never leave `_name` out; the first assert_equal fails. + def test_name_is_rendered_only_when_the_scxml_element_carries_a_name_attribute + unnamed, = render(started(%())) + named, = render(started(%(), %( name="chart.one"))) + + assert_equal({}, unnamed["datamodel"]) + assert_equal({ "_name" => "chart.one" }, named["datamodel"]) + end + + # The state HISTORIED leaves with `value` in v. + def holding(value) = moved(Statifier.step(moved(Statifier.step(started(HISTORIED), "e.go")), "e.set", data: value)) + + NO_FORM = { "date" => Predicator::Date.new(year: 2026, month: 1, day: 2), + "datetime" => Predicator::DateTime.new(epoch_seconds: 0, microsecond: 0), + "duration" => Predicator::Duration.new(years: 0, months: 0, weeks: 0, days: 0, hours: 1, minutes: 0, + seconds: 0, milliseconds: 0) }.freeze + + # sabotage: answer a date as its tagged map in PositionExpectation.json_form + # (`when Predicator::Date then value.to_h`); the first assert_equal fails. + def test_a_date_datetime_or_duration_value_fails_naming_the_variable + NO_FORM.each do |kind, value| + reason = "the datamodel's v holds a #{kind} value, which has no JSON form here" + + assert_equal [nil, reason], render(holding(value)), kind + end + + assert_equal "the datamodel's v holds a list value, which has no JSON form here", + render(holding([NO_FORM["date"]]))[1] + end + + # sabotage: answer nil from PositionExpectation.compare whatever differs + # (`return if true`); the assert_equal on the reason fails. + def test_a_differing_member_is_named_with_both_values + state = started(%()) + expected = { "configuration" => %w[b], "entered_states" => %w[a], "states_to_invoke" => [], + "history_values" => {}, "active_invocations" => [], "running" => true, "datamodel" => {} } + + assert_equal 'after step 2 (e.go), expect_position differs: configuration: expected ["b"], but got ["a"]', + Conformance::PositionExpectation.compare(expected, state, 2, "e.go") + assert_nil Conformance::PositionExpectation.compare(expected.merge("configuration" => %w[a]), state, 2, "e.go") + assert_match(/states_to_invoke: expected nothing, but got \[\]/, + Conformance::PositionExpectation.compare(expected.except("states_to_invoke"), state, 1, "e")) + end +end + +# The drive's side of a step's expect_position (Conformance::Drive in +# test/conformance/runner.rb): the step's data sent, the position compared, +# a document with an unnamed state refused before the drive, and the diff +# pair accepted and compared by nothing. +class PositionExpectationDriveTest < Minitest::Test + ROOT = PositionExpectationTest::ROOT + + # A statifier case around body, its one step event e.go with `data`, the + # step stating `position`. + def position_case(body, position, data: nil) + event = { "name" => "e.go" } + event["data"] = data unless data.nil? + { "id" => "statifier/x/y", "source" => %(#{body}), "initial_configuration" => %w[a], + "steps" => [{ "event" => event, "configuration" => %w[b], "expect_position" => position }] } + end + + CARRY = <<~XML + + + + XML + + def carried(value) + { "configuration" => %w[b], "entered_states" => %w[a b], "states_to_invoke" => [], "history_values" => {}, + "active_invocations" => [], "running" => true, "datamodel" => { "v" => value } } + end + + # sabotage: send the name alone for a case with no host object (`data = + # item.key?("host")` in Drive.drive); the first assert_equal fails. + # sabotage: drop the position comparison from Drive.drive_step (`[state, + # failed]`); the second assert_equal fails. + def test_a_case_with_no_host_stating_a_position_sends_the_step_data_and_compares_the_position + assert_equal :pass, Conformance::Drive.call(position_case(CARRY, carried(7), data: 7)) + assert_equal "after step 1 (e.go), expect_position differs: datamodel: expected {\"v\":8}, but got {\"v\":7}", + Conformance::Drive.call(position_case(CARRY, carried(8), data: 7)) + end + + # sabotage: answer nil from PositionExpectation.named; the assert_equal + # on the reason fails (the case is then driven). + def test_a_case_stating_a_position_on_a_document_with_an_unnamed_state_fails_before_it_is_driven + body = %() + + assert_equal "a case that expects a position needs every state to carry an id, " \ + "and 1 state(s) of this document have none", + Conformance::Drive.call(position_case(body, carried(0))) + end + + # sabotage: drop `to_source` from UNCOMPARED_HOST_KEYS; the assert_equal + # fails (the case is then refused naming the key). + def test_a_case_carrying_a_diff_pair_is_driven_and_the_pair_is_compared_by_nothing + item = position_case(CARRY, carried(0)).except("steps").merge( + "steps" => [{ "event" => { "name" => "e.go" }, "configuration" => %w[b] }], + "host" => { "send_types" => [], "expect_sends" => [], "to_source" => "not a document", + "expect_diff" => { "anything" => true }, "mapping" => { "x" => "y" }, + "expect_compatible_at" => ["nowhere"] } + ) + + assert_equal :pass, Conformance::Drive.call(item) + end +end diff --git a/test/conformance/runner.rb b/test/conformance/runner.rb index 5a8bf4b..f175d1a 100644 --- a/test/conformance/runner.rb +++ b/test/conformance/runner.rb @@ -3,6 +3,7 @@ require_relative "corpus" require_relative "host" require_relative "accepts" +require_relative "position_expectation" require_relative "../../lib/statifier" module Conformance @@ -85,8 +86,9 @@ def after(result) # above, from the start of the chart; once every configuration agrees, # the sends handed to it over the whole drive, in order and each with its # outcome, must be exactly `expect_sends`. A step of a case with a host - # object sends its event's `data`, when it gives one, as the harness - # injects it. A send whose expected item asks the host to report it + # object, or of a case stating an exported position, sends its event's + # `data`, when it gives one, as the harness injects it (the harness runs + # both kinds of case through `run/2`). A send whose expected item asks the host to report it # failed (`"outcome": "fail"`) is reported through # {Statifier.report_send_failed} once each driver call that handed it # returns, before the configuration is read, as the harness reports it @@ -102,11 +104,20 @@ def after(result) # included. Either key without the other fails the case, naming the one # present. # - # Any other host key (the statifier suite's `to_source`, `expect_diff`, - # `mapping` and `expect_compatible_at`, and any key a later corpus adds) - # is not honoured here and fails the case naming the key. A case stating - # an exported position asks for what this runner does not compare yet, - # and fails with that named. + # A host object's diff pair, `to_source` and `expect_diff` with an + # optional `mapping` and `expect_compatible_at`, is accepted and compared + # by nothing here: a case carrying it is driven like any other host case, + # as the reference's harness drives it (its moduledoc in host_case.ex: + # "this runner compares none of them"). Any other host key (any key a + # later corpus adds) is not honoured here and fails the case naming the + # key. + # + # A step's `expect_position` is compared, once that step's configuration + # agrees, with the rendering of the state's exported position + # (Conformance::PositionExpectation, test/conformance/position_expectation.rb), + # as the harness compares it (`position/3` in host_case.ex). A case that + # states one on any step needs every state of its document to carry an id, + # and fails before it is driven when one has none (`named/2` there). module Drive # How long, in virtual ms, the clock may run before an event to drain # short timers. @@ -124,8 +135,12 @@ module Drive # its entry here and nowhere else. PROVIDED = {}.freeze - # The host keys the drive reads; any other is not honoured. - HONOURED_HOST_KEYS = %w[event_io_processors send_types expect_sends declared_events expect_accepts].freeze + # The diff pair's host keys: accepted, and compared by nothing here. + UNCOMPARED_HOST_KEYS = %w[to_source expect_diff mapping expect_compatible_at].freeze + + # The host keys the drive reads or accepts; any other is not honoured. + HONOURED_HOST_KEYS = (%w[event_io_processors send_types expect_sends declared_events expect_accepts] + + UNCOMPARED_HOST_KEYS).freeze # The answer for item: :pass, or the reason it failed. provided is the # table of processors the drive can register (PROVIDED, unless a test @@ -137,20 +152,23 @@ def self.call(item, provided: PROVIDED) machine, failed = Accepts.prepared(item) return failed if failed + unnamed = PositionExpectation.expected?(item) && PositionExpectation.named(machine) + return unnamed if unnamed + host = Host.for(item) driven = drive(item, machine, host.processors.merge(registered(item, provided)), host) driven == :pass ? host.verdict : driven end # What the case asks of its host that this runner does not honour, or - # nil: a host key it does not read, a processor `provided` holds no - # entry for, or an exported position. + # nil: a host key it neither reads nor accepts, or a processor + # `provided` holds no entry for. def self.unhonoured(item, provided = PROVIDED) host = item.fetch("host", {}) other = host.keys - HONOURED_HOST_KEYS return "the case's host object (#{other.join(", ")}) is not honoured here" unless other.empty? - not_provided(host, provided) || position(item) + not_provided(host, provided) end # The processors the host object names that `provided` holds no entry @@ -163,13 +181,6 @@ def self.not_provided(host, provided) "which this package does not provide" end - # The reason for a case stating an exported position, or nil. - def self.position(item) - return unless item.fetch("steps").any? { _1.key?("expect_position") } - - "a step states an exported position, which this runner does not compare yet" - end - # The URIs a host object's `event_io_processors` names, in order; a value # that is not a list is named whole, as it came, so it is refused. def self.named(host) @@ -192,17 +203,20 @@ def self.drive(item, machine, processors = {}, host = nil) failed = compare(state, item.fetch("initial_configuration"), "the initial configuration") return failed if failed + data = item.key?("host") || PositionExpectation.expected?(item) item.fetch("steps").each_with_index do |step, index| - state, failed = drive_step(state, step, index, calls, host: item.key?("host")) + state, failed = drive_step(state, step, index, calls, host: data) return failed if failed end :pass end # One step: the clock settled, the event sent, the expectation awaited - # and compared. `[state, reason]`, the reason nil when it agrees. A step - # of a case with a host object (`host`) sends its event's `data` when it - # gives one; any other sends the name alone. + # and compared, and then the step's exported position when it states + # one. `[state, reason]`, the reason nil when it agrees. A step of a + # case the harness runs as a host case (`host`: one with a host object + # or stating a position) sends its event's `data` when it gives one; + # any other sends the name alone. def self.drive_step(state, step, index, calls = Calls.plain, host: false) state = settle(state, calls) since = state.now_ms @@ -210,7 +224,8 @@ def self.drive_step(state, step, index, calls = Calls.plain, host: false) name = event.fetch("name") data = host ? event.fetch("data", Predicator::UNDEFINED) : Predicator::UNDEFINED state = await(calls.step(state, name, data), step.fetch("configuration"), since, calls) - [state, compare(state, step.fetch("configuration"), "step #{index + 1} (event #{name.inspect})")] + failed = compare(state, step.fetch("configuration"), "step #{index + 1} (event #{name.inspect})") + [state, failed || PositionExpectation.at_step(state, step, index + 1)] end # The state a call answered; a refused call is a bug here, since every diff --git a/test/conformance/statifier_suite_test.rb b/test/conformance/statifier_suite_test.rb index 2b5644d..a8f41b5 100644 --- a/test/conformance/statifier_suite_test.rb +++ b/test/conformance/statifier_suite_test.rb @@ -4,49 +4,46 @@ require_relative "runner" # The statifier suite, the reference's own cases, as the drive answers it: -# every case of its accepts, library, send and system_variables groups -# passes but the ones that ask for what the runner does not do yet, and -# each of those, and every case of the other groups, fails with that -# named. +# every case of every group passes, the diff cases driven with their diff +# pair compared by nothing and the cases stating an exported position +# compared on it. class StatifierSuiteTest < Minitest::Test CORPUS = Conformance::Corpus.load CASES = CORPUS.cases_of("statifier") RESULTS = CASES.to_h { |item| [item["id"], Conformance::Drive.call(item)] } - POSITION = "a step states an exported position, which this runner does not compare yet" - - def group(item) = item["id"].split("/")[1] - def position?(item) = item["steps"].any? { _1.key?("expect_position") } - # sabotage: compare the handed sends in reverse in Host#verdict; the - # assert_equal on the passing ids fails (a case handed two sends or more - # then disagrees). - # sabotage: drop the exported-position refusal from Drive.unhonoured; the - # same assert_equal fails (the position cases are then driven and pass, - # their positions uncompared). + # assert_equal fails (a case handed two sends or more then disagrees). # sabotage: answer the accepts check with its two lists swapped in - # Accepts.compare; the same assert_equal fails (the accepts case then + # Accepts.compare; the assert_equal fails (the accepts case then # disagrees). - def test_every_accepts_library_send_and_system_variables_case_passes_but_what_the_runner_does_not_do_yet - groups = CASES.select { %w[accepts library send system_variables].include?(group(_1)) } - passing = groups.reject { position?(_1) }.map { _1["id"] } - - assert_equal passing, RESULTS.select { _2 == :pass }.keys - refute_empty passing + # sabotage: render `_name` whatever the document (drop it from the + # left-out list in PositionExpectation.datamodel); the assert_equal fails + # (the four cases stating a position then disagree on their datamodel). + def test_every_case_of_the_suite_passes + assert_equal CASES.to_h { [_1["id"], :pass] }, RESULTS + assert_equal 36, CASES.size end - # sabotage: reword the reason Drive.position answers; the assert_equal on - # the reason of a case stating an exported position fails. - def test_each_case_left_fails_naming_what_it_asks_for - CASES.reject { RESULTS[_1["id"]] == :pass }.each do |item| - assert_equal expected_reason(item), RESULTS[item["id"]], item["id"] - end + # The first case stating a position. + POSITIONED = CASES.find { _1["steps"].any? { |step| step.key?("expect_position") } } + + # That case, its first such step's `running` flipped: `[case, step + # number, the flag the case states now]`. + def flipped + altered = Marshal.load(Marshal.dump(POSITIONED)) + index = altered["steps"].index { _1.key?("expect_position") } + position = altered["steps"][index]["expect_position"] + position["running"] = !position["running"] + [altered, index + 1, position["running"]] end - def expected_reason(item) - other = item.fetch("host", {}).keys - Conformance::Drive::HONOURED_HOST_KEYS - return "the case's host object (#{other.join(", ")}) is not honoured here" unless other.empty? + # sabotage: answer nil from PositionExpectation.compare (`return if + # true`); the assert_match on the reason fails. + def test_a_case_stating_a_position_the_chart_does_not_hold_fails_naming_the_member + altered, number, running = flipped + differs = "expect_position differs: running: expected #{running}, but got #{!running}" - POSITION + assert_match(/\Aafter step #{number} \(.*\), #{differs}\z/, Conformance::Drive.call(altered)) end end