[disaykin]

Fixed an issue where binary data ([]byte) was always serialized as a base64 string in MongoDB. This occurred because the output plugin was converting the structure to a JSON string and then re-parsing it using the extJSON parser, which loses the native binary type.

Replaced extJSONFromMap with structuredFromMap leveraging msg.AsStructured(). This prevents immediate string serialization and preserves native Go slices, enabling the MongoDB driver to properly handle binary payloads.

Backward Compatibility

To maintain full backward compatibility and prevent breaking downstream client applications, the old serialization behavior (saving binary fields as base64 strings) is kept by default.

A new configuration option, preserve_binary (defaulting to false), has been introduced for both MongoDB output and processor components. Users can set this flag to true to opt-in to the correct native BSON BinData representation.

Some integration and unit tests have been added to verify both the default and new execution paths.

[CLAassistant]

All committers have signed the CLA.

[disaykin]

Hi team! I've added a dedicated unit and integration tests (output_test.go) inside the internal mongodb package to verify this change using testcontainers. The test successfully passes and confirms that Go-native []byte fields are now correctly preserved and stored as native bson.Binary (BinData) inside MongoDB instead of being forced into base64 strings. Regarding backward compatibility: since this alters the data type stored in the database for raw bytes, please let me know if we should safely gate this behind a new configuration flag (like preserve_binary: false by default), or if you prefer to accept this as the correct default behavior for the connector. Ready for your review!

[disaykin]

Update: I went ahead and implemented the preserve_binary option (defaulting to false) for both the output and processor components to guarantee absolute backward compatibility out of the box. I've also added integration and unit tests covering both default and new behaviors.

[disaykin]

Hi @mmatczuk, could you please take a look when you have a moment?

[disaykin]

Hi team! I've just rebased the branch with the latest main to ensure everything is up to date and all tests/CI pass smoothly. Since this fixes a silent data type mutation issue in MongoDB (saving raw bytes as base64), it would be great to get this reviewed when you have a moment. Thanks!

CC @mmatczuk @squiidz @Leward

[Leward]

You can already store native BinData with document_map, without a new option.

The output reads the result of document_map as MongoDB extended JSON. In extended JSON, binary data uses $binary.

Example 1: the message is raw bytes

Each Kafka record is a file. We store the full record as BinData:

input:
  redpanda:
    seed_brokers: [ localhost:9092 ]
    topics: [ files ]
    consumer_group: mongo_writer

output:
  mongodb:
    url: mongodb://localhost:27017
    database: example
    collection: files
    operation: insert-one
    document_map: |
      root.name = metadata("kafka_key")
      root.content = {
        "$binary": {
          "base64": content().encode("base64"),
          "subType": "00"
        }
      }

Example 2: the message is JSON with a base64 field

The field is already base64, so we do not encode it again:

input:
  generate:
    count: 1
    mapping: |
      root.id = "img-1"
      root.thumbnail = "iVBORw0KGgo="

output:
  mongodb:
    url: mongodb://localhost:27017
    database: example
    collection: images
    operation: insert-one
    document_map: |
      root._id = this.id
      root.thumbnail = { "$binary": { "base64": this.thumbnail, "subType": "00" } }

In both examples, MongoDB stores the field as BinData (subtype 0). If you write raw bytes instead (for example root.content = content()), you get a base64 string. This is how JSON writes bytes, and it is the documented behavior.

---

I also found some problems with preserve_binary: true:

• It stops extended JSON for the full document, not only for binary fields. For example, with this mapping:

  document_map: |
    root.created_at = { "$date": { "$numberLong": "1700000000000" } }

  MongoDB stores created_at as an object with a $date key, not as a date. $oid and $numberDecimal have the same problem.

• The field order of the document can change, because the document becomes a Go map.

• filter_map and hint_map still use extended JSON. So the same config can mean two different things.

Because of this, I prefer not to add the option.

But a $binary example in the document_map docs would help other users who have this problem. Would you be open to change the PR to add that example (or open a new one)? The docs come from the field description in internal/impl/mongodb/common.go.

Sidenote: the docs/modules/ files are generated, so you can remove your changes to them.

[disaykin]

Hi @Leward,

Thank you for the detailed feedback and for catching the regressions with $date and $oid. You are completely right—the current patch broke the existing Extended JSON path by replacing the original serialization logic entirely.

Previously, the connector relied on extJSONFromMap, which handled Extended JSON operators correctly but had a silent limitation: native []byte fields (e.g., coming from Protobuf or Avro messages) were forced into standard JSON base64 strings instead of becoming native BSON BinData.

While your suggested manual workaround using {"$binary": ...} works for raw JSON, it is unfortunately not applicable when the pipeline ingests Protobuf/Avro payloads where binary fields are already decoded as native Go []byte. Forcing users to manually extract, encode, and wrap these fields in the configuration defeats the purpose of an automated connector.

To solve this properly, I propose to refine the PR as follows:

1. Use the original behavior by default: We will keep the default execution path using the original logic so that Extended JSON operators work exactly as they did before, guaranteeing 100% backward compatibility and preventing data type mutations for existing deployments.
2. Keep and fix the preserve_binary option: The flag is named correctly for this specific scope. When opted-in (true), the plugin will ensure that Go-native []byte fields are properly preserved and mapped to BSON primitive.Binary (BinData) alongside the correctly evaluated Extended JSON types.
3. Ensure consistency across all maps: I will fix the gaps you mentioned and implement this logic consistently across all components, including filter_map, hint_map, and the processor.

This eliminates the need for manual configuration boilerplates for binary payloads while keeping the connector completely stable and backward-compatible.

What do you think about this adjusted direction? If it sounds good, I will refactor the code to fix the regressions and cover all mapping modes.

[disaykin]

Hi @Leward,

Thanks for the feedback! I have fully refactored the code to address your concerns. The regressions should be fixed now.

Regarding the documentation: I also tried to update the document_map description with a note about the $binary operator. However, I’m having trouble adding the full YAML examples you provided.

When I add them to documentMapExamples(), they lack context (titles/descriptions). When I try to find the right place for service example spec at the component level, I get a bit lost in the internal structure of common.go and how it ties to the final ConfigSpec.

Could you please point me to the specific line/function where I should inject these structured component-level examples? Or, if it's easier, feel free to push that documentation piece directly to this branch.

Thanks again for your help!