Skip to content

Refactoring of internal storage types (continued) - #2819

Merged
akukanov merged 28 commits into
mainfrom
dev/storage-refactoring-continued-akukanov
Oct 13, 2026
Merged

akukanov merged 28 commits into
mainfrom
dev/storage-refactoring-continued-akukanov

Conversation

@akukanov

@akukanov akukanov commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

This PR continues improving the internal SYCL backend infrastructure to deal with temporary buffers and device-computed results. It complements the storage types (__device/result/combined/_storage) with a "storage holder" to keep the allocated memory alive and pass it from a backend pattern to its caller.

Changes made:

  • create a new utility header specially for storage types and related helpers, move respective types and functions there.
    • __result_and_scratch_storage and __future are not moved and will mostly remain intact until fully replaced.
  • add a new __storage_holder class tempate to specify a set of storage buffers that a backend pattern may deal with.
    • scratch buffers are kept type-erased, only to be released at a later point
    • result buffers can be of several types that are defined by a parameter pack.
  • add an implementation-detail test for the new template.
  • use this new infrastructure to replace __result_and_scratch_storage in merge sort (__parallel_sort_impl)
  • (unrelated but rather small) generalize utilities for optional result storage used in reduce-then-scan

How it is supposed to work:

  • a backend pattern defines an alias to __storage_holder to specialize its return types and limit the number of scratch buffers.
  • callers use that alias to create an instance of storage holder and pass it to the pattern by reference.
  • the pattern creates and uses storage types as needed, and at the end deposits them to the holder.
  • a sycl::event that signals kernel completion remains (a part of) the returned value.
  • the caller is responsible to ensure sufficient lifetime for the holder and its content.
  • the returned values can be extracted from the holder via its __copy_result method.
  • the content of the holder can be extracted into a tuple for wrapping into __future.

@akukanov akukanov added this to the 2022.15.0 milestone Sep 3, 2026
@akukanov
akukanov force-pushed the dev/storage-refactoring-continued-akukanov branch 2 times, most recently from cf23f8d to 7ed97ca Compare September 4, 2026 09:52
@akukanov
akukanov changed the base branch from main to dev/code-cleanup-2026-09 September 4, 2026 11:42
@akukanov
akukanov force-pushed the dev/storage-refactoring-continued-akukanov branch 15 times, most recently from 5d61050 to 69d0f26 Compare September 9, 2026 19:04
Base automatically changed from dev/code-cleanup-2026-09 to main September 9, 2026 19:07
@akukanov
akukanov force-pushed the dev/storage-refactoring-continued-akukanov branch 3 times, most recently from fd4584f to 7be2138 Compare September 9, 2026 19:34
@akukanov
akukanov force-pushed the dev/storage-refactoring-continued-akukanov branch 2 times, most recently from 8ef73cf to 641a89a Compare September 18, 2026 19:48
@akukanov
akukanov force-pushed the dev/storage-refactoring-continued-akukanov branch from 641a89a to f67f551 Compare September 18, 2026 19:57
Co-authored-by: Sergey Kopienko <sergey.kopienko@intel.com>
@akukanov
akukanov force-pushed the dev/storage-refactoring-continued-akukanov branch from e263656 to e124db1 Compare September 29, 2026 14:07
auto __temp_acc = __get_accessor(sycl::write_only, __scratch_storage, __cgh, __dpl_sycl::__no_init{});
auto __stop_pos_acc =
__internal::__get_stop_pos_accessor_opt<_Bounded>(sycl::write_only, __cgh, __stop_pos_storage);
auto __stop_pos_acc = __get_accessor(sycl::write_only, __stop_pos_storage, __cgh, __dpl_sycl::__no_init{});

@danhoeflinger danhoeflinger Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a new (benign) bug, just a propagated one, but we are touching it so lets fix it.
__no_init is wrong here, when talking about any block past block 0. However, instead of trying to have it dependent on block, lets just remove it. When subsequent blocks get an accessor that is write with no_init, the originally block 0 written value can be overwritten by uninitialized data on the way out.

When USM is available this is USM data anyway so it isn't used in most cases.

If we wanted to get fancy, we don't even need this accessor after block zero in this kernel, but again with USM it is not costly so lets just remove no_init.

Suggested change
auto __stop_pos_acc = __get_accessor(sycl::write_only, __stop_pos_storage, __cgh, __dpl_sycl::__no_init{});
auto __stop_pos_acc = __get_accessor(sycl::write_only, __stop_pos_storage, __cgh);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, so my initial variant was right :) I added no_init in response to Sergey's feedback; but I should have looked thoroughly.

@akukanov akukanov Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we wanted to get fancy, we don't even need this accessor after block zero in this kernel, but again with USM it is not costly so lets just remove no_init.

Actually, I have got an idea that if a constructor of __result_storage takes an initial value, the reduce submitter would not need __stop_pos_storage and __stop_pos_initial_state at all. And the initialization from host should not add much extra overhead, except maybe for device USM. Maybe it is worth exploring.

@danhoeflinger danhoeflinger Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I suppose if we expect the norm to be host_usm for __stop_pos_storage, then yes we could do this for a small optimization. However, it would be at a detriment for the other two options, device USM and sycl::buffer.

It would save a little bit of time in the most optimized case, and simplify the code, but risk I think a lot more in the other cases (extra kernel launch for a fill or extra host->device transfer). I don't have strong opinions here against this, but I'm not sure its worth a lot of effort either.

@akukanov akukanov Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will it be really a detriment for sycl::buffer, once we removed no_init?
Also do we have any hardware and SYCL platform in mind with device USM but no host USM?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe so, because creating a sycl::buffer from a range without initial value likely allows the runtime to avoid any initial transfer from host to device. If it is initialized with a value or backed with data on the host, then it would need to have a host device transfer on the first kernel launch. I think the way it is created now with a range may skip that first host device transfer (even after removing no_init), or at least enables that option for the runtime (whether or not they take it is up to them I suppose).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the cases which support USM but not host USM are rare and not something we really care much about. No modern GPUs are in this category as far as I understand.

danhoeflinger
danhoeflinger previously approved these changes Oct 7, 2026

@danhoeflinger danhoeflinger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I believe this is in good shape now. (needs a rebase with main)

@SergeyKopienko SergeyKopienko left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some additional comments.

@SergeyKopienko
SergeyKopienko self-requested a review October 12, 2026 15:57
SergeyKopienko
SergeyKopienko previously approved these changes Oct 12, 2026

@SergeyKopienko SergeyKopienko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Left some optional comments

@akukanov
akukanov dismissed stale reviews from SergeyKopienko and danhoeflinger via 5f3a7d4 October 13, 2026 10:15

@SergeyKopienko SergeyKopienko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@akukanov
akukanov merged commit bcbdba7 into main Oct 13, 2026
24 of 25 checks passed
@akukanov
akukanov deleted the dev/storage-refactoring-continued-akukanov branch October 13, 2026 16:11
SergeyKopienko added a commit that referenced this pull request Oct 15, 2026
* move storage types and related utilities to a separate header
* generalize optional result storage from reduce-then-scan
* add __storage_holder to manage storage lifetime for asynchronous patterns
* use the holder with merge sort
* add unit test for __storage_holder
---------

Co-authored-by: Sergey Kopienko <sergey.kopienko@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants