feat(sdk): fetch only the named fields with only

[saltas888]

Why

Developers who write integrations, scripts and generators with the SDK cannot ask for only some fields of a node. Every get(), all() and filters() query fetches the default set: every attribute and most relationships. include adds to that set and exclude removes from it, so a query fetches more than the code reads and grows each time someone adds a field to the schema.

Goal: a developer names the fields they need, gets those plus the fields the SDK needs, and the query stays the same when the schema grows. Adds only in the API.

Jira: INFP-532

-> Reviewed locally using claudes' /code-review high

Notes to reviewer:

DO NOT REVIEW THE CODE

• Read public contract.md
• Read data-model.md
• Note1: There's some code duplication explained in here feat(sdk): fetch only the named fields with only #1423 (comment)
• Note2: - hfid in only follows how node.hfid already works: the SDK calculates it from the node's fields and never reads the server's hfid. So naming hfid fetches the fields it needs. Nodes not from only are unchanged. Details: spec.md (FR-005).
• Note3:is_loaded lets code check whether a field was fetched before reading it.** On a node from only, reading an unfetched field raises FieldNotLoadedError. Before this PR, a field missing from the response and a field set to null both read as None. is_loaded works on every node, including after exclude, and never raises.

Existing bugs found during end-to-end testing

These came up while testing this pull request against Infrahub 1.11.5. None of them is caused by this pull request.

• bug: RelationshipManager.fetch() raises ValueError for ancestors and descendants that were not loaded #1425: ancestors.fetch() and descendants.fetch() raise ValueError when the relationship was not loaded.
• bug: GraphQL query fails with "Cannot return null for non-nullable field" when node_metadata is requested on an empty parent infrahub#11100: a GraphQL query that requests node_metadata on an empty parent fails on the server.
• bug: assigning to a cardinality-many relationship is silently accepted and fails at save() with a misleading error #1152 (already open): assigning a list to a cardinality-many relationship replaces the relationship with a list, and save() then fails.

[codecov]

Codecov Report

❌ Patch coverage is 97.60956% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
infrahub_sdk/node/relationship.py	90.12%	5 Missing and 3 partials ⚠️
infrahub_sdk/node/node.py	98.82%	0 Missing and 2 partials ⚠️
infrahub_sdk/protocols_base.py	50.00%	0 Missing and 2 partials ⚠️

@@                 Coverage Diff                  @@
##           infrahub-develop    #1423      +/-   ##
====================================================
+ Coverage             86.86%   87.36%   +0.50%     
====================================================
  Files                   152      153       +1     
  Lines                 14813    15221     +408     
  Branches               1982     2056      +74     
====================================================
+ Hits                  12867    13298     +431     
+ Misses                 1382     1367      -15     
+ Partials                564      556       -8     

Flag	Coverage Δ
integration-tests	42.62% <58.56%> (+0.24%)	⬆️
python-3.10	61.73% <74.70%> (+0.81%)	⬆️
python-3.11	61.71% <74.70%> (+0.79%)	⬆️
python-3.12	61.71% <74.70%> (+0.79%)	⬆️
python-3.13	61.71% <74.70%> (+0.79%)	⬆️
python-3.14	61.71% <74.70%> (+0.79%)	⬆️
python-filler-3.12	23.07% <22.90%> (+0.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
infrahub_sdk/client.py	85.90% <100.00%> (+0.59%)	⬆️
infrahub_sdk/exceptions/__init__.py	100.00% <ø> (ø)
infrahub_sdk/exceptions/base.py	95.39% <100.00%> (+0.68%)	⬆️
infrahub_sdk/node/_only.py	100.00% <100.00%> (ø)
infrahub_sdk/node/attribute.py	100.00% <100.00%> (ø)
infrahub_sdk/node/related_node.py	95.13% <100.00%> (+4.03%)	⬆️
infrahub_sdk/node/node.py	89.49% <98.82%> (+1.71%)	⬆️
infrahub_sdk/protocols_base.py	77.44% <50.00%> (-0.86%)	⬇️
infrahub_sdk/node/relationship.py	89.77% <90.12%> (+7.90%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cloudflare-workers-and-pages]

Deploying infrahub-sdk-python with    Cloudflare Pages

Latest commit:	6d00aaa
Status:	✅  Deploy successful!
Preview URL:	https://0c02088c.infrahub-sdk-python.pages.dev
Branch Preview URL:	https://ds-infp-532-only-field-selec.infrahub-sdk-python.pages.dev

View logs

[saltas888]

Follow-up refactoring: build the only query with the default builder (generate_query_data / generate_query_data_node) instead of this separate builder and its sync copy.

What it needs first:

• A private way for the default builder to request a relationship without fetching its peer's fields. Today include does both. No new public parameter.
• node.hfid from the server's hfid for nodes from only, since the default builder can't request "a reference plus one peer attribute".
• Fixes for three gaps in the default builder:
  • _process_hierarchical_fields passes the node's exclude to the hierarchy peers;
  • hierarchy fields are not processed on hierarchical generics;
  • the fragment loop drops fields that an implementing kind inherits from another generic.

A local probe built the query both ways for every case in tests/unit/sdk/test_node_only_query.py: 20 of 32 matched; the rest differ for the reasons above.

[ogenstad]

The main reason to have this only parameter is for the sake of performance to avoid over fetching and only collect what we need. The benefits of this is a) less data on the wire to transfer b) less work for the infrahub server.

It seems like this choice could be sensible for schemas that only have attribute local components that compose the hfid of a node. However if the hfid has a part tied to a remote node through a relationship that lookup is a lot heavier for the backend than an extra attribute lookup (as the hfid is).

Say we have a sample schema:

  - name: Character
    namespace: Rpg
    label: "Character"
    icon: "mdi:shield-person"
    default_filter: name__value
    generate_template: true
    order_by:
      - name__value
    display_label: "{{ name__value }}"
    human_friendly_id: [name__value, profession__name__value]
    #uniqueness_constraints: [[name__value]]
    #display_labels: null
    # display_label: "name__value"
    inherit_from:
      - LineageOwner
      - LineageSource
      - CoreArtifactTarget
      - RpgEntity
    attributes:
      - name: name
        description: The name of the character
        kind: Text
        unique: true
      - name: description
        description: Brief description
        kind: Text
        optional: true
    relationships:
      - name: profession
        peer: RpgClass
        optional: false
        cardinality: one
        kind: Attribute

This is just a snippet but the relevant part is the hfid that contains a related peer.

If we start the SDK with INFRAHUB_ECHO_GRAPHQL_QUERIES=true we get something like this:

>>> client.all(kind="RpgCharacter", only=["hfid"])

query All_RpgCharacter ($offset: Int!, $limit: Int!) {
    RpgCharacter(offset: $offset, limit: $limit) {
        count
        edges {
            node {
                id
                display_label
                __typename
                name {
                    value
                }
                profession {
                    node {
                        id
                        hfid
                        display_label
                        __typename
                        name {
                            value
                        }
                    }
                }
            }
        }
    }
}

It could be that this decision was to avoid querying for an extra attribute, but the cost is instead that we need to include a relationship to the mix, then by default we also search for the hfid and display_label on the remote peer.

Typically you'd see this on Interface nodes where the hfid would combine the local attribute "name" with a relationship to a Device that also has an attribute called "Name". One of the reasons we introduced the hfid "attribute" was to avoid a dynamic lookup where the device would be evaluated as part of the query. As such I think this has to change.

[saltas888]

Infrahub's CI passed against this branch at commit 907694f, in opsmill/infrahub#11108: a draft on develop that only moved the python_sdk submodule. 39 checks passed and none failed. 16 were skipped because a submodule change does not match their path filters.

The jobs that ran include the backend unit tests (Python 3.12, 3.13 and 3.14), the component, functional, integration and Docker integration tests, the end-to-end tests and the benchmarks.

The three commits pushed after 907694f (4a2ddfb, cf442ef and 9b5f70d) change infrahub_sdk/client.py and infrahub_sdk/node/relationship.py and were not part of that run.