protobuf 7.35.1: Build dynamic proto3 message classes without protoc while preserving field presence and 64-bit values through binary and JSON round trips

# Dynamic protobuf contracts without protoc

This sample builds a proto3 schema entirely from `FileDescriptorProto`, adds it
to a fresh `DescriptorPool`, and obtains runtime message classes through
`message_factory.GetMessageClass`. It does not need a `.proto` compiler or a
generated `_pb2.py` module.

`DescriptorPool.Add` should be treated as a registration operation. The upb
backend returns the registered `FileDescriptor`, while the pure-Python backend
returns `None`. Resolving the registered file with `FindFileByName` is portable
across both implementations.

The important distinction is field presence. A normal proto3 scalar returns its
default value but cannot be queried with `HasField`. A `proto3_optional` scalar
must belong to a synthetic oneof in the descriptor and then distinguishes unset
from explicitly set-to-default. Message fields also have presence.

The JSON mapping has another deliberate type boundary: an `int64` larger than
JavaScript's safe integer range is emitted as a decimal string. Present optional
defaults remain in JSON, while `always_print_fields_with_no_presence=True` only
synthesizes fields that do not support presence.

`requirements.lock` pins protobuf 7.35.1 to the SHA-256 hashes of the official
Windows x64 and platform-independent wheels exercised by local and isolated
verification. `requirements.txt` retains the strict package pin consumed by the
CSX Python resolver.

{"case":{"believed":"A plain proto3 scalar set to its default has presence like an optional field, and protobuf JSON emits int64 values as JSON numbers while omitting explicitly present optional defaults.","caseId":"case:sha256:9361e9bedc95d1453fb2c6f9ce1c550b5f03b188450f5b9d06aa6329f0748177","contract":["DescriptorPool.Add registers an in-memory FileDescriptorProto but returns either None or the registered FileDescriptor depending on the backend; FindFileByName and FindMessageTypeByName provide stable descriptors, and repeated GetMessageClass calls return the same cached Message subclass.","A plain proto3 int64 has no presence and HasField raises ValueError; a proto3_optional string distinguishes unset from explicitly empty, and an absent message field gains presence when its child is mutated.","Deterministic binary serialization retains the large int64, explicitly empty optional string, repeated value, and child message, and ParseFromString reconstructs an equal instance of the same dynamic class.","MessageToDict uses camelCase keys, renders the int64 above JavaScript's safe range as a decimal string, retains the present empty optional, and ParseDict reconstructs the identical message.","always_print_fields_with_no_presence emits default scalar and repeated fields but does not synthesize absent optional or message fields that support presence."],"goal":"Build dynamic proto3 message classes without protoc while preserving field presence and 64-bit values through binary and JSON round trips","kind":"HOW","packages":["pkg:pypi/protobuf@7.35.1"],"schemaVersion":1,"symbols":["google.protobuf.descriptor_pb2.FileDescriptorProto","google.protobuf.descriptor_pool.DescriptorPool.Add","google.protobuf.descriptor_pool.DescriptorPool.FindFileByName","google.protobuf.descriptor_pool.DescriptorPool.FindMessageTypeByName","google.protobuf.message_factory.GetMessageClass","google.protobuf.message.Message.HasField","google.protobuf.message.Message.SerializeToString","google.protobuf.message.Message.ParseFromString","google.protobuf.json_format.MessageToDict","google.protobuf.json_format.ParseDict"]},"contractCommand":["python","test/contract.py"],"environment":{"arch":"x64","ecosystem":"pypi","executionContext":"python","language":"python","languageVersion":"3.12","libc":"musl","os":"linux","osVersionBucket":"alpine","packageManager":"pip","runtime":"python","runtimeVersion":"3.12","schemaVersion":1},"license":"MIT-0","packages":["pkg:pypi/protobuf@7.35.1"],"schemaVersion":1,"symbols":["google.protobuf.descriptor_pb2.FileDescriptorProto","google.protobuf.descriptor_pool.DescriptorPool.Add","google.protobuf.descriptor_pool.DescriptorPool.FindFileByName","google.protobuf.descriptor_pool.DescriptorPool.FindMessageTypeByName","google.protobuf.message_factory.GetMessageClass","google.protobuf.message.Message.HasField","google.protobuf.message.Message.SerializeToString","google.protobuf.message.Message.ParseFromString","google.protobuf.json_format.MessageToDict","google.protobuf.json_format.ParseDict"],"verifierAdapter":"python@1"}

# Hashes are the official PyPI artifacts used by the two exercised runtimes:
# - protobuf-7.35.1-cp310-abi3-win_amd64.whl (local Windows CPython 3.12)
# - protobuf-7.35.1-py3-none-any.whl (the Alpine verifier fallback)
protobuf==7.35.1 \
    --hash=sha256:230a75ddfc2de4806e56696ce9640c1cdfdb6543b7cfce98d42a4c0a0e7bdb87 \
    --hash=sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9

protobuf==7.35.1

from importlib.metadata import distribution
from pathlib import Path

import google.protobuf
from google.protobuf import descriptor_pb2
from google.protobuf import descriptor_pool
from google.protobuf import json_format
from google.protobuf import message_factory
from google.protobuf.message import Message


EXPECTED_VERSION = "7.35.1"
LARGE_ACCOUNT_ID = 9_007_199_254_740_993


def assert_official_distribution_import() -> None:
    """Reject a shadow module or a different protobuf installation."""
    installed = distribution("protobuf")
    package_file = Path(google.protobuf.__file__).resolve()
    distribution_root = Path(installed.locate_file("")).resolve()

    assert installed.version == EXPECTED_VERSION
    assert google.protobuf.__version__ == EXPECTED_VERSION
    assert package_file.is_relative_to(distribution_root), (
        package_file,
        distribution_root,
    )


def add_field(
    message_proto,
    *,
    name: str,
    number: int,
    field_type: int,
    label: int = descriptor_pb2.FieldDescriptorProto.LABEL_OPTIONAL,
    type_name: str = "",
    proto3_optional: bool = False,
    oneof_index: int | None = None,
) -> None:
    field = message_proto.field.add(
        name=name,
        number=number,
        label=label,
        type=field_type,
        proto3_optional=proto3_optional,
    )
    if type_name:
        field.type_name = type_name
    if oneof_index is not None:
        field.oneof_index = oneof_index


def build_runtime_types():
    file_proto = descriptor_pb2.FileDescriptorProto(
        name="csx/dynamic_account.proto",
        package="csx.contract",
        syntax="proto3",
    )

    profile_proto = file_proto.message_type.add(name="Profile")
    add_field(
        profile_proto,
        name="city",
        number=1,
        field_type=descriptor_pb2.FieldDescriptorProto.TYPE_STRING,
    )

    account_proto = file_proto.message_type.add(name="Account")
    account_proto.oneof_decl.add(name="_display_name")
    add_field(
        account_proto,
        name="account_id",
        number=1,
        field_type=descriptor_pb2.FieldDescriptorProto.TYPE_INT64,
    )
    add_field(
        account_proto,
        name="display_name",
        number=2,
        field_type=descriptor_pb2.FieldDescriptorProto.TYPE_STRING,
        proto3_optional=True,
        oneof_index=0,
    )
    add_field(
        account_proto,
        name="aliases",
        number=3,
        field_type=descriptor_pb2.FieldDescriptorProto.TYPE_STRING,
        label=descriptor_pb2.FieldDescriptorProto.LABEL_REPEATED,
    )
    add_field(
        account_proto,
        name="profile",
        number=4,
        field_type=descriptor_pb2.FieldDescriptorProto.TYPE_MESSAGE,
        type_name=".csx.contract.Profile",
    )

    pool = descriptor_pool.DescriptorPool()
    add_result = pool.Add(file_proto)
    file_descriptor = pool.FindFileByName(file_proto.name)
    account_descriptor = pool.FindMessageTypeByName("csx.contract.Account")
    profile_descriptor = pool.FindMessageTypeByName("csx.contract.Profile")
    Account = message_factory.GetMessageClass(account_descriptor)
    Profile = message_factory.GetMessageClass(profile_descriptor)

    assert file_descriptor.name == file_proto.name
    assert add_result is None or add_result is file_descriptor
    assert account_descriptor.file is file_descriptor
    assert Account.DESCRIPTOR is account_descriptor
    assert Profile.DESCRIPTOR is profile_descriptor
    assert issubclass(Account, Message)
    assert issubclass(Profile, Message)
    assert message_factory.GetMessageClass(account_descriptor) is Account
    assert message_factory.GetMessageClass(profile_descriptor) is Profile
    return Account, Profile


def assert_presence_contract(Account) -> None:
    account = Account()
    assert account.account_id == 0

    try:
        account.HasField("account_id")
    except ValueError:
        # upb and the pure-Python backend intentionally word this error
        # differently; the API contract is the exception type.
        pass
    else:
        raise AssertionError("plain proto3 scalar unexpectedly supported presence")

    assert not account.HasField("display_name")
    account.display_name = ""
    assert account.HasField("display_name")
    account.ClearField("display_name")
    assert not account.HasField("display_name")

    assert not account.HasField("profile")
    account.profile.city = "Seoul"
    assert account.HasField("profile")


def populated_account(Account):
    account = Account(account_id=LARGE_ACCOUNT_ID, aliases=["ops"])
    account.display_name = ""
    account.profile.city = "Seoul"
    return account


def assert_binary_contract(Account) -> None:
    account = populated_account(Account)
    encoded = account.SerializeToString(deterministic=True)
    assert encoded == (
        b"\x08\x81\x80\x80\x80\x80\x80\x80\x10"
        b"\x12\x00"
        b"\x1a\x03ops"
        b"\x22\x07\x0a\x05Seoul"
    )

    decoded = Account()
    consumed = decoded.ParseFromString(encoded)
    assert consumed == len(encoded)
    assert type(decoded) is Account
    assert decoded == account
    assert decoded.HasField("display_name")


def assert_json_contract(Account) -> None:
    account = populated_account(Account)
    document = json_format.MessageToDict(account)
    assert document == {
        "accountId": str(LARGE_ACCOUNT_ID),
        "displayName": "",
        "aliases": ["ops"],
        "profile": {"city": "Seoul"},
    }

    decoded = json_format.ParseDict(document, Account())
    assert type(decoded) is Account
    assert decoded == account
    assert decoded.HasField("display_name")

    defaults = json_format.MessageToDict(
        Account(),
        always_print_fields_with_no_presence=True,
    )
    assert defaults == {"accountId": "0", "aliases": []}
    assert "displayName" not in defaults
    assert "profile" not in defaults


def main() -> None:
    assert_official_distribution_import()
    Account, _Profile = build_runtime_types()
    assert_presence_contract(Account)
    assert_binary_contract(Account)
    assert_json_contract(Account)
    print("CONTRACT PASS: protobuf 7.35.1 dynamic types, presence, binary, and JSON")


if __name__ == "__main__":
    main()