[Schamper]

Depends on #178.

Adds write support to the compiler. Also add a bunch more tests.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 251 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (e200c0b) to head (6ddc32c).

Files with missing lines	Patch %	Lines
dissect/cstruct/compiler.py	0.00%	244 Missing ⚠️
dissect/cstruct/types/base.py	0.00%	7 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##            main    #179    +/-   ##
======================================
  Coverage   0.00%   0.00%            
======================================
  Files         22      22            
  Lines       3294    3533   +239     
======================================
- Misses      3294    3533   +239     

Flag	Coverage Δ
unittests	0.00% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed]

Merging this PR will improve performance by 55.98%

1 warning

⚠️ 6 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚡ 6 improved benchmarks
✅ 39 untouched benchmarks
⏩ 47 skipped benchmarks¹

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	test_benchmark_union_array[compiled-write]	340.7 µs	139.8 µs	×2.4
⚡	test_benchmark_enum_flag_bits[compiled-write]	151.9 µs	84.8 µs	+79.06%
⚡	test_benchmark_union[compiled-write]	104.5 µs	70.9 µs	+47.27%
⚡	test_benchmark_union_nested[compiled-write]	115.9 µs	80.3 µs	+44.27%
⚡	test_benchmark_basic[compiled-write]	108.7 µs	80.2 µs	+35.49%
⚡	test_benchmark_nested_dynamic[compiled-write]	162.6 µs	141.8 µs	+14.67%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing compiler-writes (6ddc32c) with main (e200c0b)}

Footnotes

1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[yunzheng]

ah, write support is basically just writing the bytes of the structure to a file handle?

[yunzheng]

There are some conflicts, even after merging #178. Also what is the difference in a compiled write and just writing the dumps() or bytes() version directly?

[Schamper]

Also what is the difference in a compiled write and just writing the dumps() or bytes() version directly?

It's the same, compiled writes only change how dumps() and bytes() pack the bytes underwater.

Basically, instead of running this slow loop in Structure._write:

dissect.cstruct/dissect/cstruct/types/structure.py

Lines 299 to 355 in e200c0b

	def _write(cls, stream: BinaryIO, data: Structure, *, endian: Endianness) -> int:
	bit_buffer = None
	struct_start = stream.tell()
	for field in cls.__fields__:
	field_type = field.type
	if bit_buffer is not None and bit_buffer._type is not None:
	bit_field_type = (
	(field_type.__type__ if isinstance(field_type, EnumMetaType) else field_type)
	if field.bits
	else None
	)
	# Current field is not a bit field, but previous was
	# Or, moved to a bit field of another type, e.g. uint16 f1 : 8, uint32 f2 : 8;
	if bit_buffer._type != bit_field_type:
	# Flush the current bit buffer so we can process alignment properly
	bit_buffer.flush()
	offset = stream.tell()
	if field.offset is not None and offset < struct_start + field.offset:
	# Field is at a specific offset, either aligned or added that way
	stream.write(b"\x00" * (struct_start + field.offset - offset))
	offset = struct_start + field.offset
	if cls.__align__ and field.offset is None:
	bit_type = bit_buffer._type if bit_buffer is not None else None
	if bit_type is None or bit_buffer._remaining == 0 or bit_type != field_type:
	# Previous field was dynamically sized and we need to align
	align_pad = -offset & (field.alignment - 1)
	stream.write(b"\x00" * align_pad)
	offset += align_pad
	value = getattr(data, field._name)
	if value is None:
	value = field_type.__default__()
	if field.bits:
	if bit_buffer is None:
	bit_buffer = BitBuffer(stream, endian=endian)
	if isinstance(field_type, EnumMetaType):
	bit_buffer.write(field_type.__type__, value.value, field.bits)
	else:
	bit_buffer.write(field_type, value, field.bits)
	else:
	field_type._write(stream, value, endian=endian)
	if bit_buffer is not None and bit_buffer._type is not None:
	bit_buffer.flush()
	if cls.__align__:
	# Align the stream
	stream.write(b"\x00" * (-stream.tell() & (cls.__alignment__ - 1)))
	return stream.tell() - struct_start

The compiler will now generate a _write() that's optimized for that specific structure:

In [2]: from dissect.cstruct import cstruct

In [3]: cs = cstruct("""
   ...: struct test {
   ...:     uint8 a;
   ...:     uint8 b;
   ...: };
   ...: """)

In [4]: cs.test(a=1, b=2).dumps()
Out[4]: b'\x01\x02'

In [5]: print(cs.test._write.__source__)
def _write(cls, stream, data, *, endian):
    if cls is not _cls0: return _struct_write(cls, stream, data, endian=endian)

    try:
        buf = _pack0[endian](data.a, data.b)
    except _struct_error:
        buf = _pack0[endian](_or_default(data.a, _type0), _or_default(data.b, _type0))
    stream.write(buf)

    return 2

We've been doing the same for Structure._read for a number of years already, this PR just adds it for Structure._write too. The benchmarks show the performance improvement from doing this.