# Python packages

A full Python package may need more than the compiled extension generated by cppwg.
For example, a package may want to add syntactic sugar for wrapped templated
classes to enable using the `Foo[Bar]` subscript syntax in Python instead of the
actually wrapped `Foo_Bar` and more closely mirror the `Foo<Bar>` usage in C++.

This can be added by hand in the Python package, but can drift from the config
whenever the wrapped instantiations change. To help address this, the
`cppwg genpackage` tool can be used to generate this **Python package layer**
from the same config that produced the wrappers, so they stay in step.

## How it works

Generation is a two-step flow:

1. When cppwg generates the wrappers it also writes a **package model**,
   `cppwg_package_model.json`. This is a small JSON description of what was
   wrapped — each module, its compiled extension, and the wrapped classes (with
   their template instantiations), enums and free functions.

2. `cppwg genpackage` reads that model, together with a **layout file** you
   provide describing the layout of the Python package, and emits a
   `_generated.py` for each Python subpackage. In `_generated.py`, all the
   wrapped elements are imported from the compiled-extension, and one
   [`TemplateClass`](#templateclass) subscript stub is added per templated
   class to allow the `Foo[Bar]` syntax from Python. The `_generated.py` can
   then be imported inside an `__init__.py`, where other things can be manually
   added that cannot be derived from the config
   (see [Hand-written code](#hand-written-code)).

(templateclass)=
### `TemplateClass`

A templated class is exposed as a `TemplateClass` subclass whose `_instantiations`
maps template-argument tuples to the concrete wrapped classes. Subscripting the
class (`Point[2]`) looks up the concrete class (`Point_2`). These can be nested
so e.g. `MeshFactory<PottsMesh<2>>` is subscripted as `MeshFactory[PottsMesh[2]]`
(see `examples/cells`).

:::{note}
The generated stubs import `TemplateClass` (and, where you use it,
`TemplateMethod`) from a `_syntax.py` helper in your package. This helper is not
generated — copy it from `examples/shapes/src/py/pyshapes/_syntax.py`.
:::

## Layouts

`cppwg genpackage` supports two layouts, and your supplied layout file chooses
between them:

- **One subpackage per module** (default): each cppwg module has its **own**
  compiled extension and becomes its own subpackage. The optional `module_dirs`
  option controls *where* each subpackage is written.
- **One shared extension split into subpackages**: everything is compiled into a
  **single** extension, which the `subpackages` option then divides across
  several subpackages by name.

## One subpackage per module

Each cppwg [module](reference.md#module-options) has its own subpackage, which
imports the names it needs from the relevant compiled extension with
`from .<module> import (...)`. This is the default, and what `examples/shapes`
uses.

**package_layout.yaml**

`examples/shapes/wrapper/package_layout.yaml`

```yaml
package: pyshapes
package_root: ../src/py/pyshapes
```

:::{note}
`package_root` is the directory holding the Python subpackage; a relative path
is resolved against the layout file's own directory.
:::

Run:

```bash
cppwg genpackage \
  --model wrapper/cppwg_package_model.json \
  --layout wrapper/package_layout.yaml
```

For the `geometry` module, this writes:

**geometry/_generated.py**

```python
"""Generated by cppwg genpackage - do not edit.
...
"""

from ._pyshapes_geometry import (
    Point_2,
    Point_3,
)
from pyshapes._syntax import TemplateClass

__all__ = [
    "Point",
    "Point_2",
    "Point_3",
]


class Point(TemplateClass):
    _instantiations = {
        ("2",): Point_2,
        ("3",): Point_3,
    }
```

The `__all__` lists exactly the names this subpackage re-exports — the imported
extension names plus the `TemplateClass` stub bases — so the
`from ._generated import *` below stays explicit and does not leak helpers such
as `TemplateClass`.

This is imported into the hand-written `__init__.py` beside it:

**geometry/__init__.py**

```python
from ._generated import *  # noqa: F401,F403
```

The package can then be imported with both the concrete names and the subscript form:

```python
from pyshapes.geometry import Point, Point_2

Point[2]   # -> Point_2
Point[3]   # -> Point_3
```

### Mapping modules to subpackage directories

By default each module's subpackage directory is named after the module, under
`package_root` — so the `geometry` module above becomes `pyshapes/geometry/`. The
`module_dirs` option overrides that directory per module: the key is the module
name, the value is a directory relative to `package_root`. Use it to rename a
subpackage, nest it, or map a module to `"."` to place it at the package root. A
module not listed keeps the default.

`examples/cells` has a single module, `all`, and maps it to `"."` so its extension
sits at the package root rather than in an `all/` subfolder:

`examples/cells/dynamic/package_layout.yaml`

```yaml
package: pycells
package_root: ../src/py/pycells
module_dirs:
  all: "."
```

The `_generated.py` is then written to `pycells/_generated.py`, and
`from ._generated import *` in `pycells/__init__.py` surfaces everything at the top
level (`from pycells import Node`).

`module_dirs` only relocates a module's files; every module still keeps its own
compiled extension. To split a *single* extension instead, use `subpackages`
([below](#splitting-one-shared-extension)).

(splitting-one-shared-extension)=
## Splitting one shared extension into subpackages

In the second layout all wrappers are compiled into a **single** extension, named
by `compiled_module`, but you still want several Python subpackages. The
`subpackages` option divides that one extension's contents manually by **name**.
For each subpackage, list the (base) class, enum and free-function names it should own.

```yaml
package: chaste
package_root: src/py/chaste
compiled_module: _pychaste_all
subpackages:
  core: [FileFinder, OutputFileHandler, RandomNumberGenerator, Timer, ...]
  mesh: [ChastePoint, Element, MutableMesh, Node, ...]
  # ...
```

Each subpackage's `_generated.py` imports only its **own** names from the shared
extension (star-importing it would pull the whole extension into every
subpackage). A name listed under no subpackage, or listed but absent from the
model, is reported as a warning.

For the `mesh` subpackage this writes the explicit imports plus a `TemplateClass`
stub per templated class:

**mesh/_generated.py**

```python
"""Generated by cppwg genpackage - do not edit.
...
"""

from chaste._pychaste_all import (
    Element_1_1,
    Element_1_2,
    Element_1_3,
    Element_2_2,
    Element_2_3,
    Element_3_3,
    # ...
)
from chaste._syntax import TemplateClass


class Element(TemplateClass):
    _instantiations = {
        ("1", "1"): Element_1_1,
        ("1", "2"): Element_1_2,
        ("1", "3"): Element_1_3,
        ("2", "2"): Element_2_2,
        ("2", "3"): Element_2_3,
        ("3", "3"): Element_3_3,
    }


# ... one stub per templated class
```

As in the per-module layout, a hand-written `__init__.py` in each subpackage
star-imports this via `from ._generated import *`.

(hand-written-code)=
## Hand-written code

Anything that cannot be derived from the config stays in the hand-written
`__init__.py`, after the `from ._generated import *` line. A common case is a
templated *method* (`TemplateMethod`), which the model does not describe:

**primitives/__init__.py**

```python
from ._generated import *  # noqa: F401,F403
from pyshapes._syntax import TemplateMethod

# UnitSquare.GetAreaIn[Unit]() — a templated method, so it cannot be
# auto-generated and is attached here.
UnitSquare.GetAreaIn = TemplateMethod("GetAreaIn", UnitSquare.GetAreaIn)
```

## Options

### `exclude`

Some wrapped classes are deliberately **not** exposed in the Python package. For
example, abstract base classes must typically stay wrapped in the compiled
extension because concrete C++ subclasses declare them as bases, and C++ APIs
pass and return them. However, in most cases they are not meant to be named,
instantiated or subclassed from Python. List such classes under `exclude` so
they are held out of every subpackage's imports and `__all__` while remaining
registered in the extension:

```yaml
exclude:
  - AbstractShape
  - AbstractPolygon
```

`exclude` works in either layout. `examples/shapes` (one subpackage per module)
uses it to hide the abstract `AbstractShape`/`AbstractPolygon` bases while their
concrete subclass `RegularPolygon` stays exposed and still inherits their method
bindings.

genpackage warns if an `exclude` entry no longer matches any wrapped class (e.g.
after a rename). In the shared-extension split it additionally warns if an
excluded name is **also** assigned to a subpackage (it would be exposed after
all), and flags an un-excluded class that is assigned to no subpackage as a
likely oversight. There is no such per-name assignment in the per-module layout,
where each module is exposed in full apart from its excludes.

### `diagonal_shorthand`

For a multi-argument instantiation whose arguments are all equal (a "diagonal",
e.g. `Element<2, 2>`), also emit a single-argument alias so `Element[2]` resolves
to the same class as `Element[2, 2]`. Off by default; enable it in the layout file:

```yaml
diagonal_shorthand: true
```

The `Element` stub from the split example above then gains a single-argument alias
key for each diagonal instantiation (the `("1",)`, `("2",)` and `("3",)` entries):

```python
class Element(TemplateClass):
    _instantiations = {
        ("1", "1"): Element_1_1,
        ("1",): Element_1_1,
        ("1", "2"): Element_1_2,
        ("1", "3"): Element_1_3,
        ("2", "2"): Element_2_2,
        ("2",): Element_2_2,
        ("2", "3"): Element_2_3,
        ("3", "3"): Element_3_3,
        ("3",): Element_3_3,
    }
```

so `Element[2]` now resolves to `Element_2_2` alongside `Element[2, 2]`.

### `flatten_to_root`

Also write a top-level `_generated.py` that re-exports every subpackage's class,
enum and free-function names into the package root, so `chaste.Node` works
alongside `chaste.mesh.Node`. A name exported by more than one subpackage is
reported as an ambiguity warning. Off by default:

```yaml
flatten_to_root: true
```

## Command-line usage

```text
usage: cppwg genpackage [-h] --model MODEL --layout LAYOUT [--overwrite]

options:
  -h, --help       show this help message and exit
  --model MODEL    Path to cppwg_package_model.json (from cppwg).
  --layout LAYOUT  Path to the Python package-layout file (YAML).
  --overwrite      Rewrite files even if unchanged.
```

The generated `_generated.py` files are reproducible: re-running `cppwg genpackage`
after a config change updates only what changed, so the layer can be checked in and
verified with `git diff`.

:::{seealso}
- See [First steps](first-steps.md) for generating the wrappers themselves.
- See [Templates](templates.md) for choosing which instantiations are wrapped.
:::