pydantic 2.13.4: Exclude or include fields across nested collections in Pydantic model serialization using '__all__' rather than bare field names

# Pydantic nested collection serialization masking semantics

## Prior search results

`search_known_solution` returned contracts for URL type coercion (`sha256:39e10d5ed6b42f8dff5b5058de1c88dc65201cb0b044894581fdada51cd99e7e`), `model_copy` trust boundaries (`sha256:a7f8bfd32813f975440d7e5007452b0320c4fef7237de0ea68ad49b75a676d7f`), and FastAPI encoder dictionary nesting (`sha256:b09b062b12f13c860852641882840dceca140b65e173f95222f347f4f95b25b6`). None addressed field masking across nested sequences or mappings during `model_dump` and `model_dump_json`.

## Pinned version

- `pydantic==2.13.4`
- `pydantic-core==2.46.4`
- `annotated-types==0.8.0`
- `typing-extensions==4.15.0`
- `typing-inspection==0.4.2`

## Observed behavior and failure modes

When serializing models with `model_dump(exclude=...)`, `model_dump(include=...)`, or their `model_dump_json` equivalents:

1. **Sequence exclusion trap**:
   - For a single nested model (e.g. `lead: UserProfile`), specifying `exclude={"lead": {"secret_token"}}` strips `secret_token` as expected.
   - For a sequence of models (e.g. `members: list[UserProfile]`), the child dictionary keys in the exclusion mask represent item indices (integers), not model attribute names.
   - Passing `exclude={"members": {"secret_token"}}` matches zero item indices. The serialization completes without error, silently leaving sensitive fields unmasked in the output (a silent green build failure mode).
   - The correct syntax to exclude attributes across all items in a sequence is `exclude={"members": {"__all__": {"secret_token"}}}`. Alternatively, specific integer indices like `{0: {"secret_token"}}` target individual item positions.

2. **Sequence inclusion trap**:
   - Passing bare attribute names to `include`, such as `include={"members": {"name"}}`, matches no integer indices and silently strips all elements from the list, returning `{"members": []}`.
   - The correct syntax to include specific fields across all sequence items is `include={"members": {"__all__": {"name"}}}`.

3. **Mapping behavior**:
   - For dictionary fields (e.g. `departments: dict[str, UserProfile]`), child keys match dictionary keys directly (e.g. `exclude={"departments": {"core": {"secret_token"}}}`).
   - To apply an exclusion to all dictionary values regardless of key name, `"__all__"` must be used: `exclude={"departments": {"__all__": {"secret_token"}}}`. Passing bare model attribute names without `"__all__"` checks for top-level dictionary keys of that name and leaves values unmasked.

{"case":{"believed":"Passing a set of field names like {'members': {'secret'}} to model_dump(exclude=...) excludes that field from each nested model in a sequence.","caseId":"case:sha256:c4e2a91ae3dad53396b374a50a78e62732e1233ca0b0b38ff3041497829766b6","contract":["Passing a bare field set like {'members': {'secret'}} to model_dump(exclude=...) silently leaves secrets present in nested sequence elements because collection keys represent item indices rather than attribute names.","Using {'__all__': {'secret'}} under a sequence or mapping key excludes the field from every nested model item.","Passing integer indices like {0: {'secret'}} under a sequence key excludes fields exclusively from the matching item position.","Passing bare field names to model_dump(include=...) on a sequence matches zero item indices and returns an empty list, whereas '__all__' projects the requested fields for all items.","Under mapping fields, bare keys match only exact dictionary keys rather than value attributes unless '__all__' or an explicit mapping key is provided."],"goal":"Exclude or include fields across nested collections in Pydantic model serialization using '__all__' rather than bare field names","kind":"HOW","packages":["pkg:pypi/pydantic@2.13.4"],"schemaVersion":1,"symbols":["pydantic.BaseModel.model_dump","pydantic.BaseModel.model_dump_json"]},"contractCommand":["python","test/contract.py"],"environment":{"arch":"x64","ecosystem":"pypi","executionContext":"python","language":"python","os":"linux","packageManager":"pip","runtime":"python","schemaVersion":1},"license":"MIT-0","packages":["pkg:pypi/pydantic@2.13.4"],"schemaVersion":1,"symbols":["pydantic.BaseModel.model_dump","pydantic.BaseModel.model_dump_json"],"verifierAdapter":"python@1"}

annotated-types==0.8.0
pydantic==2.13.4
pydantic-core==2.46.4
typing-extensions==4.15.0
typing-inspection==0.4.2

annotated-types==0.8.0
pydantic==2.13.4
pydantic-core==2.46.4
typing-extensions==4.15.0
typing-inspection==0.4.2

"""Package root."""

from pydantic import BaseModel


class UserProfile(BaseModel):
    id: int
    name: str
    secret_token: str
    email: str


class TeamOrganization(BaseModel):
    name: str
    lead: UserProfile
    members: list[UserProfile]
    departments: dict[str, UserProfile]

import json
import sys
from importlib.metadata import version
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent.parent))

from src.models import TeamOrganization, UserProfile


def test_nested_sequence_exclude_semantics() -> None:
    lead = UserProfile(
        id=1, name="Eve", secret_token="lead-secret-123", email="eve@example.com"
    )
    members = [
        UserProfile(
            id=2,
            name="Alice",
            secret_token="member-secret-456",
            email="alice@example.com",
        ),
        UserProfile(
            id=3, name="Bob", secret_token="member-secret-789", email="bob@example.com"
        ),
    ]
    org = TeamOrganization(
        name="Platform",
        lead=lead,
        members=members,
        departments={},
    )

    # 1. Direct nested model: field name set correctly strips the field
    direct_dump = org.model_dump(exclude={"lead": {"secret_token"}})
    assert "secret_token" not in direct_dump["lead"]
    assert direct_dump["lead"]["name"] == "Eve"

    # 2. Sequence nested model trap: passing bare field name does NOT exclude secrets
    # because sequence keys are interpreted as item indices, not field names.
    naive_dump = org.model_dump(exclude={"members": {"secret_token"}})
    assert naive_dump["members"][0]["secret_token"] == "member-secret-456"
    assert naive_dump["members"][1]["secret_token"] == "member-secret-789"

    # 3. Correct sequence exclusion: '__all__' applies the mask across all items
    correct_dump = org.model_dump(exclude={"members": {"__all__": {"secret_token"}}})
    assert "secret_token" not in correct_dump["members"][0]
    assert "secret_token" not in correct_dump["members"][1]
    assert correct_dump["members"][0]["name"] == "Alice"
    assert correct_dump["members"][1]["name"] == "Bob"

    # 4. Integer index exclusion: targets only the specific sequence position
    indexed_dump = org.model_dump(exclude={"members": {0: {"secret_token"}}})
    assert "secret_token" not in indexed_dump["members"][0]
    assert indexed_dump["members"][1]["secret_token"] == "member-secret-789"

    # 5. model_dump_json mirrors the '__all__' sequence exclusion behavior
    json_dump = json.loads(
        org.model_dump_json(exclude={"members": {"__all__": {"secret_token"}}})
    )
    assert "secret_token" not in json_dump["members"][0]
    assert "secret_token" not in json_dump["members"][1]


def test_nested_sequence_include_semantics() -> None:
    org = TeamOrganization(
        name="Platform",
        lead=UserProfile(
            id=1, name="Eve", secret_token="sec", email="eve@example.com"
        ),
        members=[
            UserProfile(
                id=2, name="Alice", secret_token="sec1", email="alice@example.com"
            ),
            UserProfile(
                id=3, name="Bob", secret_token="sec2", email="bob@example.com"
            ),
        ],
        departments={},
    )

    # Bare field name matches no integer index, producing an empty list
    naive_include = org.model_dump(include={"members": {"name"}})
    assert naive_include["members"] == []

    # '__all__' correctly includes only the requested fields across all items
    correct_include = org.model_dump(include={"members": {"__all__": {"name"}}})
    assert correct_include["members"] == [{"name": "Alice"}, {"name": "Bob"}]


def test_nested_mapping_exclude_semantics() -> None:
    dep_core = UserProfile(
        id=4,
        name="Charlie",
        secret_token="dept-secret-111",
        email="charlie@example.com",
    )
    dep_infra = UserProfile(
        id=5, name="Dana", secret_token="dept-secret-222", email="dana@example.com"
    )
    org = TeamOrganization(
        name="Platform",
        lead=UserProfile(
            id=1, name="Eve", secret_token="sec", email="eve@example.com"
        ),
        members=[],
        departments={"core": dep_core, "infra": dep_infra},
    )

    # 1. '__all__' excludes field across all dictionary values
    all_map_dump = org.model_dump(
        exclude={"departments": {"__all__": {"secret_token"}}}
    )
    assert "secret_token" not in all_map_dump["departments"]["core"]
    assert "secret_token" not in all_map_dump["departments"]["infra"]

    # 2. Specific dict key excludes field only for that mapped value
    key_map_dump = org.model_dump(
        exclude={"departments": {"core": {"secret_token"}}}
    )
    assert "secret_token" not in key_map_dump["departments"]["core"]
    assert (
        key_map_dump["departments"]["infra"]["secret_token"] == "dept-secret-222"
    )

    # 3. Bare field name does not match dict keys and leaves all values unmasked
    naive_map_dump = org.model_dump(exclude={"departments": {"secret_token"}})
    assert (
        naive_map_dump["departments"]["core"]["secret_token"]
        == "dept-secret-111"
    )
    assert (
        naive_map_dump["departments"]["infra"]["secret_token"]
        == "dept-secret-222"
    )


if __name__ == "__main__":
    assert version("pydantic") == "2.13.4"
    test_nested_sequence_exclude_semantics()
    test_nested_sequence_include_semantics()
    test_nested_mapping_exclude_semantics()
    print("CONTRACT PASS: Pydantic nested collection serialization masking verified")