Skip to content

Document the intermediate data storage utilities in the SYCL backend - #2847

Merged
akukanov merged 6 commits into
mainfrom
rfc/sycl-backend-data-storage
Nov 20, 2026
Merged

akukanov merged 6 commits into
mainfrom
rfc/sycl-backend-data-storage

Conversation

@akukanov

@akukanov akukanov commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

This RFC describes how to use the intermediate data storage in the SYCL backend patterns and their callers. It covers both the existing storage types (__{device|result|combined}_storage) and the new __storage_holder class template and related utilities introduced in #2819.

Comment on lines +81 to +84
On construction it first attempts to allocate host USM, which allows the result
to be read back to the host without an explicit copy. If host USM is not
available, it falls back to the device USM or `sycl::buffer` strategy of
`__device_storage`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Host USM is used only on Level Zero GPUs, by policy rather than by availability: __allocate_usm<_T, sycl::usm::alloc::host> returns nullptr on every other device, including OpenCL CPU and GPU devices that support host USM (

template <typename _T, sycl::usm::alloc __alloc_t>
_T*
__allocate_usm(const sycl::queue& __q, std::size_t __elements)
{
static_assert(__alloc_t == sycl::usm::alloc::host || __alloc_t == sycl::usm::alloc::device);
_T* __result = nullptr;
if constexpr (__alloc_t == sycl::usm::alloc::host)
{
#if _ONEDPL_SYCL_L0_EXT_PRESENT
// Only use host USM on L0 GPUs. Other devices should use device USM instead to avoid notable slowdown.
sycl::device __device = __q.get_device();
if (__device.is_gpu() && __device.has(sycl::aspect::usm_host_allocations) &&
__device.get_backend() == __dpl_sycl::__level_zero_backend)
{
__result = sycl::malloc<_T>(__elements, __q, __alloc_t);
}
#endif
}
else
{
if (__q.get_device().has(sycl::aspect::usm_device_allocations))
__result = sycl::malloc<_T>(__elements, __q, __alloc_t);
}
return __result;
}
: "Only use host USM on L0 GPUs. Other devices should use device USM instead to avoid notable slowdown."). The same wording appears at L112 ("When host USM is available") and L124 ("if host USM is not available"). A possible rewording for this paragraph, which also drops the trailing space on L84:

Suggested change
On construction it first attempts to allocate host USM, which allows the result
to be read back to the host without an explicit copy. If host USM is not
available, it falls back to the device USM or `sycl::buffer` strategy of
`__device_storage`.
On construction it first attempts to allocate host USM, which allows the result
to be read back to the host without an explicit copy. Host USM is only used on
Level Zero GPUs; on other devices, including those that support host USM, it
falls back to the device USM or `sycl::buffer` strategy of `__device_storage`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a clarification.

Comment on lines +357 to +361
`__extract()` consumes the holder and converts its content into a tuple of
`__copyable_storage_state` objects with shared ownership of the underlying
allocations. The `__future` holds these states for its own lifetime, ensuring
the memory remains valid until the asynchronous result is consumed. The holder
itself is left in a state where its destructor has nothing to free.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two details could be added here.

  • __future has no destructor that waits (
    class __future : private std::tuple<_Args...>
    ), so the memory stays valid as long as the __future or a copy of it exists, not until the result is consumed. If it is destroyed before the event completes, the USM is freed while the kernels may still run.
  • The third mode is not described: an algorithm called in __deferrable_mode uses __checked_deferrable_wait(), which waits whenever the future carries a payload, so that the temporary storage is not destroyed before the kernels finish (
    void
    __checked_deferrable_wait()
    {
    #if !ONEDPL_ALLOW_DEFERRED_WAITING
    wait();
    #else
    if constexpr (sizeof...(_Args) > 0)
    {
    // We should have this wait() call to ensure that the temporary data is not destroyed before the kernel code finished
    wait();
    }
    #endif
    }
    ). A short "Deferrable Algorithm" subsection would cover it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 - rephrased to avoid misinterpretation.
2 - not yet sure of importance.

Comment on lines +132 to +134
- `__get_accessor(access_mode_tag, storage, cgh, property_list)` —
acquires an accessor to the data storage region, or to the scratch-only
region for `__combined_storage`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a __combined_storage this is not a scratch-only region: __get_accessor binds to the __device_storage base (

__get_accessor(_ModeTagT, __device_storage<_T>& __st, sycl::handler& __cgh, const sycl::property_list& __prop_list = {})
,
__get_accessor(sycl::handler& __cgh, const sycl::property_list& __prop_list = {})
{
return __combi_accessor<_T, _AccessMode>(__cgh, __sycl_buf, __usm_buf.get(), __prop_list);
}
) and covers its whole allocation. With the single-allocation layout (
this->__initialize(__q, __sz + __result_sz); // a combined buffer, starting with scratch
) the accessor therefore also spans the result region. With USM this is only a convention, but with a sycl::buffer the accessor range, and with it no_init and the dependency tracking, includes the result. Maybe: "acquires an accessor to the whole data storage; for a __combined_storage that shares one allocation, this includes the result region".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accessing the result area is neither guaranteed nor intended. The intended usage of __get_accessor for __combined_storage is only for the scratch allocation.

Comment on lines +173 to +175
- `_NScratch` — the number of scratch storage slots. This must equal the
maximum number of `__device_storage` and `__combined_storage` objects
simultaneously used by any code path through the backend pattern.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_NScratch only needs to be at least that number. It may also help to say that exceeding it is caught only by an assert in __store_scratch and __store (

assert(__scratch_count < _NScratch);
,
assert(__scratch_count < _NScratch);
), i.e. it is undefined behaviour in a release build:

Suggested change
- `_NScratch` — the number of scratch storage slots. This must equal the
maximum number of `__device_storage` and `__combined_storage` objects
simultaneously used by any code path through the backend pattern.
- `_NScratch` — the number of scratch storage slots. This must be at least the
maximum number of `__device_storage` and `__combined_storage` objects
simultaneously used by any code path through the backend pattern;
exceeding it is only caught by an assertion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed.

Comment on lines +294 to +298
Care must be taken when a kernel is in flight and holds a reference to storage
memory: destroying a storage object or the holder before the kernel completes
will cause premature deallocation. The backend pattern is responsible for
ensuring that all in-flight kernels have completed or that ownership has been
transferred to the holder before any storage object goes out of scope.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be worth stating three details here:

  • the destructor of __storage_holder, like those of the storages, does not wait for kernels;
  • a storage backed by sycl::buffer does not free early: the buffer destructor blocks until the kernels using it finish, so destroying it early silently synchronizes instead;
  • if an exception is thrown after a kernel has been submitted but before the wait (for example, by a later submit in the same pattern), the USM is freed while that kernel may still be running. That is a known limitation rather than something a pattern can always prevent.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rephrased to cover these points.

__parallel_pattern_impl(sycl::queue& __q,
__parallel_pattern_holder<ValueType>& __holder, ...)
{
__combined_storage<<ValueType> __scratch_and_result(__q, __scratch_n, 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo:

Suggested change
__combined_storage<<ValueType> __scratch_and_result(__q, __scratch_n, 1);
__combined_storage<ValueType> __scratch_and_result(__q, __scratch_n, 1);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Comment on lines +192 to +196
template <bool _Bounded, typename _ValueType, typename _StopPosType>
using __transform_scan_storage_holder =
std::conditional_t<_Bounded,
__storage_holder<2, _ValueType, _StopPosType>,
__storage_holder<2, _ValueType>>;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: in #2828 and #2830 the scan holders use __storage_holder<1, ...>; 2 comes with #2839. Fine if this document lands after #2839; otherwise 1 matches the code.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the long run, 2 is correct.

```mermaid
flowchart TB
subgraph CallerStart["Algorithm"]
S1["Construct holder\n__parallel_pattern_holder&lt;N, Ts...&gt; __holder(__q)"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the alias hides the number of scratch slots (L184-L186), so the caller writes __parallel_pattern_holder<Ts...>, as in the pseudocode below (__parallel_pattern_holder<ValueType>):

Suggested change
S1["Construct holder\n__parallel_pattern_holder&lt;N, Ts...&gt; __holder(__q)"]
S1["Construct holder\n__parallel_pattern_holder&lt;Ts...&gt; __holder(__q)"]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

sycl::event __event = __parallel_pattern_async(__q, __holder, ...);
__event.wait_and_throw();

ValueType value;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: value-initialized, as at L88 and as __get_results does for its destination:

Suggested change
ValueType value;
ValueType value{};

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not important.

}
```

Note that `_NScratch` is set to 1; the small-size path stores no scratch, while

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trailing whitespace.

Suggested change
Note that `_NScratch` is set to 1; the small-size path stores no scratch, while
Note that `_NScratch` is set to 1; the small-size path stores no scratch, while

@akukanov akukanov added this to the 2022.15.0 milestone Nov 10, 2026
@akukanov

Copy link
Copy Markdown
Contributor Author

Added to the next milestone and moved to supported, as the corresponding functionality is already merged.

danhoeflinger
danhoeflinger previously approved these changes Nov 20, 2026

@danhoeflinger danhoeflinger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple minor things which I think would help, not required.
Otherwise LGTM

@danhoeflinger danhoeflinger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@akukanov
akukanov merged commit 2308aa8 into main Nov 20, 2026
2 checks passed
@akukanov
akukanov deleted the rfc/sycl-backend-data-storage branch November 20, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants