python-dotenv 1.2.2: Pin python-dotenv parse_variables AST tokenization and resolve semantics for Literal and Variable atoms alongside parse_stream error tracking and line indexing

# the network Notes: python-dotenv parse_variables AST and parse_stream Semantics

## 1. Search Result
- **`search_known_solution`**: `NO_SAFE_MATCH` on `dotenv.variables.parse_variables`, `dotenv.variables.Atom`, `dotenv.variables.Variable`, `dotenv.variables.Literal`, `dotenv.parser.parse_stream`, `dotenv.parser.Binding`. Existing samples verified high-level `load_dotenv` and `dotenv_values` integration, but did not measure AST tokenization, variable fallback resolution, or stream parser error and line indexing contracts.

## 2. What a Model Would Have Written Instead
- A model would assume `$VAR` syntax (without braces) is tokenized into `Variable` instances rather than left as literal strings.
- A model would assume `parse_stream` raises an exception immediately on malformed lines rather than returning a `Binding` record with `error=True` and `key=None` while preserving stream line indexing.
- A model would assume unassigned keys (`KEY` without `=`) produce parsing errors or default to empty string `""` rather than resolving to `value=None` in `Binding`.
- A model would assume `Variable(name, default).resolve(env)` resolves `default` when `env[name] is None`, but python-dotenv checks `env.get(name, default)` which returns `None` and subsequently evaluates to empty string `""`.

## 3. How the Wrong Version Fails
- Calling `parse_variables("prefix_$VAR")` and expecting a `Variable` node fails silently, treating `$VAR` as plain text literal.
- In stream parsing pipelines, catching parser exceptions instead of checking `binding.error` causes unhandled malformed lines to bypass validation quietly.
- Passing `{"KEY": None}` to `Variable("KEY", default="def").resolve(...)` returns `""` rather than `"def"`, leading to silent configuration degradation when upstream dictionaries contain explicit `None` values.

Clean-room public code sample — generation instructions

Write a brand-new, minimal, self-contained code sample in this clean-room directory.
Do not copy, paraphrase, or reference any existing project source. Work only from this spec.

A csx.json manifest scaffold already exists. Do not recreate it from memory. Preserve its case.goal, packages and symbols; fill its empty case.contract with exact assertions and correct its environment, commands and verifierAdapter for the files you generate.

Goal: Pin python-dotenv parse_variables AST tokenization and resolve semantics for Literal and Variable atoms alongside parse_stream error tracking and line indexing
Kind: HOW

Use EXACTLY these public packages and versions:
  - pkg:pypi/python-dotenv@1.2.2
Demonstrate these symbols/APIs:
  - dotenv.parser.Binding
  - dotenv.parser.parse_stream
  - dotenv.variables.Atom
  - dotenv.variables.Literal
  - dotenv.variables.Variable
  - dotenv.variables.parse_variables

Rules:
  - One focused purpose; the smallest project that proves the goal.
  - Include a contract test (test/contract.*) that runs OFFLINE and exits 0 exactly when the goal behavior works.
  - Pin every dependency with a lockfile so resolution is reproducible.
  - No secrets, credentials, or tokens. No real URLs (only example.com or localhost). No absolute paths.
  - No personal names, emails, company names, or project identifiers of any kind.
  - No binaries and no generated output (node_modules, dist, target, venv, .git, .env).
  - Keep it under 200 files and 256KB packed.

{"case":{"caseId":"case:sha256:e7bfb26914cc81cae0fe5cbfe51e48ec1b85323a32e81b9ba1d9f25cd3367167","contract":["assert parse_variables tokenizes string templates into Literal and Variable atoms","assert Variable atom extracts variable name and optional default from POSIX parameter expansion syntax","assert Variable.resolve substitutes from environment mapping or falls back to default or empty string","assert parse_variables preserves bare unbracketed dollar identifiers as Literal tokens without expansion","assert parse_stream yields Binding namedtuples with 1-indexed original line numbers and parsed key-values","assert parse_stream flags syntax errors with error=True and key=None for malformed or unclosed input lines","assert parse_stream distinguishes valueless keys (value=None) from empty values (value='') without error"],"goal":"Pin python-dotenv parse_variables AST tokenization and resolve semantics for Literal and Variable atoms alongside parse_stream error tracking and line indexing","kind":"HOW","packages":["pkg:pypi/python-dotenv@1.2.2"],"schemaVersion":1,"symbols":["dotenv.parser.Binding","dotenv.parser.parse_stream","dotenv.variables.Atom","dotenv.variables.Literal","dotenv.variables.Variable","dotenv.variables.parse_variables"]},"contractCommand":["python","test/contract.py"],"environment":{"arch":"x64","distro":"ubuntu","ecosystem":"pypi","executionContext":"python","language":"python","libc":"glibc","libcVersion":"2.39","os":"linux","osVersionBucket":"24","packageManager":"pip","runtime":"python","schemaVersion":1},"license":"MIT-0","packages":["pkg:pypi/python-dotenv@1.2.2"],"schemaVersion":1,"subject":"pkg:pypi/python-dotenv@1.2.2","symbols":["dotenv.parser.Binding","dotenv.parser.parse_stream","dotenv.variables.Atom","dotenv.variables.Literal","dotenv.variables.Variable","dotenv.variables.parse_variables"],"verifierAdapter":"python@1"}

python-dotenv==1.2.2

{
  "schemaVersion": 1,
  "goal": "Pin python-dotenv parse_variables AST tokenization and resolve semantics for Literal and Variable atoms alongside parse_stream error tracking and line indexing",
  "kind": "HOW",
  "packages": [
    "pkg:pypi/python-dotenv@1.2.2"
  ],
  "symbols": [
    "dotenv.parser.Binding",
    "dotenv.parser.parse_stream",
    "dotenv.variables.Atom",
    "dotenv.variables.Literal",
    "dotenv.variables.Variable",
    "dotenv.variables.parse_variables"
  ]
}

"""python-dotenv parse_variables AST and parse_stream contract sample."""

"""Safe helpers demonstrating parse_variables AST inspection and stream parsing."""

import io
from typing import Dict, Iterator, List, Mapping, Optional, Union

from dotenv.parser import Binding, parse_stream
from dotenv.variables import Atom, Literal, Variable, parse_variables


def tokenize_template(template: str) -> List[Atom]:
    """
    Parse a string containing POSIX parameter expansions into Atom objects.

    Returns a list of Literal and Variable atoms. Bare dollar expressions without braces
    are treated as Literal tokens.
    """
    return list(parse_variables(template))


def evaluate_template(
    template: str,
    env: Mapping[str, Optional[str]],
) -> str:
    """
    Evaluate a template against the given environment mapping using AST atoms.
    """
    atoms = parse_variables(template)
    return "".join(atom.resolve(env) for atom in atoms)


def parse_dotenv_stream(
    stream: Union[str, io.StringIO],
) -> Iterator[Binding]:
    """
    Yield parsed bindings with line numbering and syntax error markers.
    """
    if isinstance(stream, str):
        stream = io.StringIO(stream)
    return parse_stream(stream)


def extract_valid_bindings(
    stream: Union[str, io.StringIO],
) -> Dict[str, Optional[str]]:
    """
    Extract valid key-value pairs from stream, skipping errors and unkeyed lines.
    """
    result: Dict[str, Optional[str]] = {}
    for binding in parse_dotenv_stream(stream):
        if not binding.error and binding.key is not None:
            result[binding.key] = binding.value
    return result

import io
import sys
from dotenv.parser import Binding, Original, parse_stream
from dotenv.variables import Atom, Literal, Variable, parse_variables


def test_parse_variables_tokenization():
    """Prove parse_variables tokenizes string templates into Literal and Variable atoms."""
    template = "base_${VAR1}_mid_${VAR2:-default_val}_$BARE"
    atoms = list(parse_variables(template))

    assert len(atoms) == 5, f"Expected 5 atoms, got {len(atoms)}: {atoms}"
    assert isinstance(atoms[0], Literal) and atoms[0].value == "base_"
    assert isinstance(atoms[1], Variable) and atoms[1].name == "VAR1" and atoms[1].default is None
    assert isinstance(atoms[2], Literal) and atoms[2].value == "_mid_"
    assert isinstance(atoms[3], Variable) and atoms[3].name == "VAR2" and atoms[3].default == "default_val"
    assert isinstance(atoms[4], Literal) and atoms[4].value == "_$BARE"

    assert all(isinstance(a, Atom) for a in atoms)


def test_variable_resolve_semantics():
    """Prove Variable.resolve substitutes from mapping or falls back to default or empty string."""
    var_no_default = Variable(name="CONFIG_KEY", default=None)
    # Unset resolves to empty string
    assert var_no_default.resolve({}) == ""
    # Set resolves to value
    assert var_no_default.resolve({"CONFIG_KEY": "prod"}) == "prod"
    # Mapped to None resolves to empty string
    assert var_no_default.resolve({"CONFIG_KEY": None}) == ""

    var_with_default = Variable(name="PORT", default="8080")
    # Unset falls back to default
    assert var_with_default.resolve({}) == "8080"
    # Set resolves to value
    assert var_with_default.resolve({"PORT": "3000"}) == "3000"
    # Mapped to None resolves to empty string (None in mapping is not considered unset)
    assert var_with_default.resolve({"PORT": None}) == ""


def test_literal_resolve_semantics():
    """Prove Literal.resolve preserves raw string and ignores environment mapping."""
    lit = Literal(value="static_$VARIABLE_literal")
    assert lit.resolve({}) == "static_$VARIABLE_literal"
    assert lit.resolve({"VARIABLE": "replaced"}) == "static_$VARIABLE_literal"


def test_parse_stream_line_indexing_and_values():
    """Prove parse_stream yields Binding namedtuples with 1-indexed original line numbers."""
    stream_content = (
        "# Line 1: Header comment\n"
        "KEY_EMPTY=\n"
        "KEY_VALLESS\n"
        "KEY_QUOTED='single quoted value'\n"
        'export KEY_DOUBLE="escaped\\nnewline"\n'
    )
    bindings = list(parse_stream(io.StringIO(stream_content)))
    assert len(bindings) == 5

    # 1. Comment line
    assert bindings[0] == Binding(
        key=None,
        value=None,
        original=Original(string="# Line 1: Header comment\n", line=1),
        error=False,
    )

    # 2. Empty value (KEY=)
    assert bindings[1] == Binding(
        key="KEY_EMPTY",
        value="",
        original=Original(string="KEY_EMPTY=\n", line=2),
        error=False,
    )

    # 3. Valueless key (KEY)
    assert bindings[2] == Binding(
        key="KEY_VALLESS",
        value=None,
        original=Original(string="KEY_VALLESS\n", line=3),
        error=False,
    )

    # 4. Quoted value
    assert bindings[3] == Binding(
        key="KEY_QUOTED",
        value="single quoted value",
        original=Original(string="KEY_QUOTED='single quoted value'\n", line=4),
        error=False,
    )

    # 5. Double quoted value with export
    assert bindings[4] == Binding(
        key="KEY_DOUBLE",
        value="escaped\nnewline",
        original=Original(string='export KEY_DOUBLE="escaped\\nnewline"\n', line=5),
        error=False,
    )


def test_parse_stream_syntax_error_flagging():
    """Prove parse_stream flags syntax errors with error=True and key=None for malformed lines."""
    malformed_content = (
        'VALID_ONE="ok"\n'
        'MALFORMED LINE WITHOUT EQUALS AND UNCLOSED "quotes\n'
        'VALID_TWO="also ok"\n'
    )
    bindings = list(parse_stream(io.StringIO(malformed_content)))
    assert len(bindings) == 3

    assert bindings[0].key == "VALID_ONE"
    assert bindings[0].error is False
    assert bindings[0].original.line == 1

    assert bindings[1].key is None
    assert bindings[1].value is None
    assert bindings[1].error is True
    assert bindings[1].original.line == 2

    assert bindings[2].key == "VALID_TWO"
    assert bindings[2].error is False
    assert bindings[2].original.line == 3


if __name__ == "__main__":
    test_parse_variables_tokenization()
    test_variable_resolve_semantics()
    test_literal_resolve_semantics()
    test_parse_stream_line_indexing_and_values()
    test_parse_stream_syntax_error_flagging()

    print("CONTRACT PASS: python-dotenv parse_variables AST and parse_stream verified")