Python packages¶

A full Python package may need more than the compiled extension generated by cppwg. For example, a package may want to add syntactic sugar for wrapped templated classes to enable using the Foo[Bar] subscript syntax in Python instead of the actually wrapped Foo_Bar and more closely mirror the Foo<Bar> usage in C++.

This can be added by hand in the Python package, but can drift from the config whenever the wrapped instantiations change. To help address this, the cppwg genpackage tool can be used to generate this Python package layer from the same config that produced the wrappers, so they stay in step.

How it works¶

Generation is a two-step flow:

When cppwg generates the wrappers it also writes a package model, cppwg_package_model.json. This is a small JSON description of what was wrapped — each module, its compiled extension, and the wrapped classes (with their template instantiations), enums and free functions.
cppwg genpackage reads that model, together with a layout file you provide describing the layout of the Python package, and emits a _generated.py for each Python subpackage. In _generated.py, all the wrapped elements are imported from the compiled-extension, and one TemplateClass subscript stub is added per templated class to allow the Foo[Bar] syntax from Python. The _generated.py can then be imported inside an __init__.py, where other things can be manually added that cannot be derived from the config (see Hand-written code).

`TemplateClass`¶

A templated class is exposed as a TemplateClass subclass whose _instantiations maps template-argument tuples to the concrete wrapped classes. Subscripting the class (Point[2]) looks up the concrete class (Point_2). These can be nested so e.g. MeshFactory<PottsMesh<2>> is subscripted as MeshFactory[PottsMesh[2]] (see examples/cells).

Note

The generated stubs import TemplateClass (and, where you use it, TemplateMethod) from a _syntax.py helper in your package. This helper is not generated — copy it from examples/shapes/src/py/pyshapes/_syntax.py.

Layouts¶

cppwg genpackage supports two layouts, and your supplied layout file chooses between them:

One subpackage per module (default): each cppwg module has its own compiled extension and becomes its own subpackage. The optional module_dirs option controls where each subpackage is written.
One shared extension split into subpackages: everything is compiled into a single extension, which the subpackages option then divides across several subpackages by name.

One subpackage per module¶

Each cppwg module has its own subpackage, which imports the names it needs from the relevant compiled extension with from .<module> import (...). This is the default, and what examples/shapes uses.

package_layout.yaml

examples/shapes/wrapper/package_layout.yaml

package: pyshapes
package_root: ../src/py/pyshapes

Note

package_root is the directory holding the Python subpackage; a relative path is resolved against the layout file’s own directory.

Run:

cppwg genpackage \
  --model wrapper/cppwg_package_model.json \
  --layout wrapper/package_layout.yaml

For the geometry module, this writes:

geometry/_generated.py

"""Generated by cppwg genpackage - do not edit.
...
"""

from ._pyshapes_geometry import (
    Point_2,
    Point_3,
)
from pyshapes._syntax import TemplateClass

__all__ = [
    "Point",
    "Point_2",
    "Point_3",
]


class Point(TemplateClass):
    _instantiations = {
        ("2",): Point_2,
        ("3",): Point_3,
    }

The __all__ lists exactly the names this subpackage re-exports — the imported extension names plus the TemplateClass stub bases — so the from ._generated import * below stays explicit and does not leak helpers such as TemplateClass.

This is imported into the hand-written __init__.py beside it:

geometry/init.py

from ._generated import *  # noqa: F401,F403

The package can then be imported with both the concrete names and the subscript form:

from pyshapes.geometry import Point, Point_2

Point[2]   # -> Point_2
Point[3]   # -> Point_3

Mapping modules to subpackage directories¶

By default each module’s subpackage directory is named after the module, under package_root — so the geometry module above becomes pyshapes/geometry/. The module_dirs option overrides that directory per module: the key is the module name, the value is a directory relative to package_root. Use it to rename a subpackage, nest it, or map a module to "." to place it at the package root. A module not listed keeps the default.

examples/cells has a single module, all, and maps it to "." so its extension sits at the package root rather than in an all/ subfolder:

examples/cells/dynamic/package_layout.yaml

package: pycells
package_root: ../src/py/pycells
module_dirs:
  all: "."

The _generated.py is then written to pycells/_generated.py, and from ._generated import * in pycells/__init__.py surfaces everything at the top level (from pycells import Node).

module_dirs only relocates a module’s files; every module still keeps its own compiled extension. To split a single extension instead, use subpackages (below).

Splitting one shared extension into subpackages¶

In the second layout all wrappers are compiled into a single extension, named by compiled_module, but you still want several Python subpackages. The subpackages option divides that one extension’s contents manually by name. For each subpackage, list the (base) class, enum and free-function names it should own.

package: chaste
package_root: src/py/chaste
compiled_module: _pychaste_all
subpackages:
  core: [FileFinder, OutputFileHandler, RandomNumberGenerator, Timer, ...]
  mesh: [ChastePoint, Element, MutableMesh, Node, ...]
  # ...

Each subpackage’s _generated.py imports only its own names from the shared extension (star-importing it would pull the whole extension into every subpackage). A name listed under no subpackage, or listed but absent from the model, is reported as a warning.

For the mesh subpackage this writes the explicit imports plus a TemplateClass stub per templated class:

mesh/_generated.py

"""Generated by cppwg genpackage - do not edit.
...
"""

from chaste._pychaste_all import (
    Element_1_1,
    Element_1_2,
    Element_1_3,
    Element_2_2,
    Element_2_3,
    Element_3_3,
    # ...
)
from chaste._syntax import TemplateClass


class Element(TemplateClass):
    _instantiations = {
        ("1", "1"): Element_1_1,
        ("1", "2"): Element_1_2,
        ("1", "3"): Element_1_3,
        ("2", "2"): Element_2_2,
        ("2", "3"): Element_2_3,
        ("3", "3"): Element_3_3,
    }


# ... one stub per templated class

As in the per-module layout, a hand-written __init__.py in each subpackage star-imports this via from ._generated import *.

Hand-written code¶

Anything that cannot be derived from the config stays in the hand-written __init__.py, after the from ._generated import * line. A common case is a templated method (TemplateMethod), which the model does not describe:

primitives/init.py

from ._generated import *  # noqa: F401,F403
from pyshapes._syntax import TemplateMethod

# UnitSquare.GetAreaIn[Unit]() — a templated method, so it cannot be
# auto-generated and is attached here.
UnitSquare.GetAreaIn = TemplateMethod("GetAreaIn", UnitSquare.GetAreaIn)

Options¶

`exclude`¶

Some wrapped classes are deliberately not exposed in the Python package. For example, abstract base classes must typically stay wrapped in the compiled extension because concrete C++ subclasses declare them as bases, and C++ APIs pass and return them. However, in most cases they are not meant to be named, instantiated or subclassed from Python. List such classes under exclude so they are held out of every subpackage’s imports and __all__ while remaining registered in the extension:

exclude:
  - AbstractShape
  - AbstractPolygon

exclude works in either layout. examples/shapes (one subpackage per module) uses it to hide the abstract AbstractShape/AbstractPolygon bases while their concrete subclass RegularPolygon stays exposed and still inherits their method bindings.

genpackage warns if an exclude entry no longer matches any wrapped class (e.g. after a rename). In the shared-extension split it additionally warns if an excluded name is also assigned to a subpackage (it would be exposed after all), and flags an un-excluded class that is assigned to no subpackage as a likely oversight. There is no such per-name assignment in the per-module layout, where each module is exposed in full apart from its excludes.

`diagonal_shorthand`¶

For a multi-argument instantiation whose arguments are all equal (a “diagonal”, e.g. Element<2, 2>), also emit a single-argument alias so Element[2] resolves to the same class as Element[2, 2]. Off by default; enable it in the layout file:

diagonal_shorthand: true

The Element stub from the split example above then gains a single-argument alias key for each diagonal instantiation (the ("1",), ("2",) and ("3",) entries):

class Element(TemplateClass):
    _instantiations = {
        ("1", "1"): Element_1_1,
        ("1",): Element_1_1,
        ("1", "2"): Element_1_2,
        ("1", "3"): Element_1_3,
        ("2", "2"): Element_2_2,
        ("2",): Element_2_2,
        ("2", "3"): Element_2_3,
        ("3", "3"): Element_3_3,
        ("3",): Element_3_3,
    }

so Element[2] now resolves to Element_2_2 alongside Element[2, 2].

`flatten_to_root`¶

Also write a top-level _generated.py that re-exports every subpackage’s class, enum and free-function names into the package root, so chaste.Node works alongside chaste.mesh.Node. A name exported by more than one subpackage is reported as an ambiguity warning. Off by default:

flatten_to_root: true

Command-line usage¶

usage: cppwg genpackage [-h] --model MODEL --layout LAYOUT [--overwrite]

options:
  -h, --help       show this help message and exit
  --model MODEL    Path to cppwg_package_model.json (from cppwg).
  --layout LAYOUT  Path to the Python package-layout file (YAML).
  --overwrite      Rewrite files even if unchanged.

The generated _generated.py files are reproducible: re-running cppwg genpackage after a config change updates only what changed, so the layer can be checked in and verified with git diff.

Python packages¶

How it works¶

TemplateClass¶

Layouts¶

One subpackage per module¶

Mapping modules to subpackage directories¶

Splitting one shared extension into subpackages¶

Hand-written code¶

Options¶

exclude¶

diagonal_shorthand¶

flatten_to_root¶

Command-line usage¶

`TemplateClass`¶

`exclude`¶

`diagonal_shorthand`¶

`flatten_to_root`¶