Alternative Approaches

This page compares StateTracker to other Python methods for working with combinatorial structures. Understanding these alternatives will help you appreciate what makes StateTracker unique.

The key insight: StateTracker’s power comes from building a computation DAG (directed acyclic graph) of states, where setting the state of a derived state automatically propagates to all parent states. No other approach provides this.

The Problem: Complex Combinatorial Structures

Real-world combinatorial problems often involve more than simple Cartesian products. Consider designing an experiment with:

• 2 control samples

• Plus a treatment arm with 3 treatments x 4 replicates

This is a stack (disjoint union) of a simple state and a product. The total is 2 + 12 = 14 samples, but they have different structure depending on which “arm” is active.

When you iterate through these 14 samples, you need to know:

1. Which arm is active (control or treatment)?

2. If treatment, what are the treatment and replicate indices?

3. If you shuffle, slice, or sample, how do you track all this?

Let’s see how hard this is without StateTracker:

The Manual Approach

# Manual approach: 2 controls + (3 treatments x 4 replicates)
def get_indices_manual(sample_idx):
    """Manually compute which arm is active and the component indices."""
    num_controls = 2
    num_treatments = 3
    num_replicates = 4

    if sample_idx < num_controls:
        # Control arm
        return {"control": sample_idx, "treatment": None, "replicate": None}
    else:
        # Treatment arm: need to decompose into treatment x replicate
        adjusted = sample_idx - num_controls
        treatment = adjusted % num_treatments
        replicate = adjusted // num_treatments
        return {"control": None, "treatment": treatment, "replicate": replicate}


print("Manual enumeration of 14 samples:")
for i in range(14):
    print(f"  Sample {i}: {get_indices_manual(i)}")

Manual enumeration of 14 samples:
  Sample 0: {'control': 0, 'treatment': None, 'replicate': None}
  Sample 1: {'control': 1, 'treatment': None, 'replicate': None}
  Sample 2: {'control': None, 'treatment': 0, 'replicate': 0}
  Sample 3: {'control': None, 'treatment': 1, 'replicate': 0}
  Sample 4: {'control': None, 'treatment': 2, 'replicate': 0}
  Sample 5: {'control': None, 'treatment': 0, 'replicate': 1}
  Sample 6: {'control': None, 'treatment': 1, 'replicate': 1}
  Sample 7: {'control': None, 'treatment': 2, 'replicate': 1}
  Sample 8: {'control': None, 'treatment': 0, 'replicate': 2}
  Sample 9: {'control': None, 'treatment': 1, 'replicate': 2}
  Sample 10: {'control': None, 'treatment': 2, 'replicate': 2}
  Sample 11: {'control': None, 'treatment': 0, 'replicate': 3}
  Sample 12: {'control': None, 'treatment': 1, 'replicate': 3}
  Sample 13: {'control': None, 'treatment': 2, 'replicate': 3}

This works, but now imagine you want to shuffle these samples. You need to:

1. Generate a random permutation

2. Apply it before looking up indices

3. Rewrite all your index math to account for the permutation

And what if you want to slice (take samples 5-10)? Or split into train/test? Each operation requires rewriting the index logic. This quickly becomes unmanageable.

StateTracker’s Solution: The State DAG

StateTracker solves this by building a directed acyclic graph (DAG) of states. You declare the structure once, and StateTracker handles all the index math automatically, including for shuffles, slices, samples, and splits.

from statetracker import Manager, State, product, stack

with Manager():
    # Declare the structure
    control = State(num_values=2, name="control")
    treatment = State(num_values=3, name="treatment")
    replicate = State(num_values=4, name="replicate")

    # Build: controls + (treatments x replicates)
    treatment_arm = product([treatment, replicate])
    samples = stack([control, treatment_arm])

    print("StateTracker enumeration of 14 samples:")
    for state in samples:
        print(
            f"  Sample {state}: control={control.value}, treatment={treatment.value}, replicate={replicate.value}"
        )

StateCounter enumeration of 14 samples:
  Sample 0: control=0, treatment=None, replicate=None
  Sample 1: control=1, treatment=None, replicate=None
  Sample 2: control=None, treatment=0, replicate=0
  Sample 3: control=None, treatment=1, replicate=0
  Sample 4: control=None, treatment=2, replicate=0
  Sample 5: control=None, treatment=0, replicate=1
  Sample 6: control=None, treatment=1, replicate=1
  Sample 7: control=None, treatment=2, replicate=1
  Sample 8: control=None, treatment=0, replicate=2
  Sample 9: control=None, treatment=1, replicate=2
  Sample 10: control=None, treatment=2, replicate=2
  Sample 11: control=None, treatment=0, replicate=3
  Sample 12: control=None, treatment=1, replicate=3
  Sample 13: control=None, treatment=2, replicate=3

The magic is automatic state propagation: when you set the state of the derived state, all parent states (control, treatment, replicate) update automatically. The inactive states show None.

Now adding shuffle, slice, or split is trivial:

from statetracker import Manager, State, product, shuffle, split, stack

with Manager():
    control = State(num_values=2, name="control")
    treatment = State(num_values=3, name="treatment")
    replicate = State(num_values=4, name="replicate")

    treatment_arm = product([treatment, replicate])
    samples = stack([control, treatment_arm])

    # Shuffle - no manual permutation tracking needed!
    shuffled = shuffle(samples, seed=42)

    # Split into train/test - no manual index math!
    train, test = split(shuffled, [0.8, 0.2])

    print(f"Total: {samples.num_values}, Train: {train.num_values}, Test: {test.num_values}")
    print("\nTest set (states still propagate correctly):")
    for state in test:
        print(
            f"  control={control.value}, treatment={treatment.value}, replicate={replicate.value}"
        )

Total: 14, Train: 11, Test: 3

Test set (states still propagate correctly):
  control=0, treatment=None, replicate=None
  control=1, treatment=None, replicate=None
  control=None, treatment=2, replicate=2

You can visualize the state DAG to understand the structure:

from statetracker import Manager, State, product, shuffle, split, stack

with Manager():
    control = State(num_values=2, name="control")
    treatment = State(num_values=3, name="treatment")
    replicate = State(num_values=4, name="replicate")

    treatment_arm = product([treatment, replicate], name="treatment_arm")
    samples = stack([control, treatment_arm], name="samples")
    shuffled = shuffle(samples, seed=42, name="shuffled")
    train, test = split(shuffled, [0.8, 0.2], names=["train", "test"])

    test.print_dag()

test (counter, io=0, n=3)
└── [op=Slice]
    └── shuffled (counter, io=0, n=14)
        └── [op=Shuffle]
            └── samples (counter, io=0, n=14)
                └── [op=Stack]
                    ├── control (counter, io=0, n=2)
                    └── treatment_arm (counter, io=0, n=12)
                        └── [op=Product]
                            ├── treatment (counter, io=0, n=3)
                            └── replicate (counter, io=0, n=4)

Alternative Approaches in Python

Let’s examine what other Python tools offer and why they fall short for problems involving both products and stacks with additional operations.

itertools

Python’s itertools module provides product for Cartesian products and chain for concatenation:

from itertools import chain, product

# itertools.product for Cartesian products
print("itertools.product (2 x 3):")
for i, (t, r) in enumerate(product(range(2), range(3))):
    print(f"  {i}: ({t}, {r})")

# itertools.chain for concatenation
print("\nitertools.chain (2 controls + 3 treatments):")
controls = [("control", i) for i in range(2)]
treatments = [("treatment", i) for i in range(3)]
for i, item in enumerate(chain(controls, treatments)):
    print(f"  {i}: {item}")

itertools.product (2 x 3):
  0: (0, 0)
  1: (0, 1)
  2: (0, 2)
  3: (1, 0)
  4: (1, 1)
  5: (1, 2)

itertools.chain (2 controls + 3 treatments):
  0: ('control', 0)
  1: ('control', 1)
  2: ('treatment', 0)
  3: ('treatment', 1)
  4: ('treatment', 2)

Limitations of itertools:

1. No state propagation: You get tuples, not connected objects. There is no way to ask “given index 4, what are the component values?”

2. No composition: You cannot easily combine product and chain into a single enumerable structure with unified indexing.

3. No tracking for chain: With chain, you lose information about which source contributed each element. You have to encode it manually (as we did with the tuples).

4. Operations do not compose: To shuffle an itertools.product, you must materialize it into a list first. Then you lose the ability to know which original indices correspond to each shuffled position.

NumPy’s unravel_index / ravel_multi_index

NumPy provides functions to convert between flat indices and multi-dimensional indices for arrays:

import numpy as np

# Shape: 2 treatments x 3 replicates
shape = (2, 3)

# Convert flat index 4 to multi-dimensional indices
# Note: numpy uses row-major (C) order by default
indices = np.unravel_index(4, shape)
print(f"Flat index 4 -> treatment={indices[0]}, replicate={indices[1]}")

# Convert back to flat index
flat = np.ravel_multi_index(indices, shape)
print(f"treatment={indices[0]}, replicate={indices[1]} -> flat index {flat}")

Flat index 4 -> treatment=1, replicate=1
treatment=1, replicate=1 -> flat index 4

Limitations of NumPy’s index functions:

1. Only rectangular products: NumPy assumes a regular multi-dimensional array shape. It cannot represent structures like “2 controls + (3 x 4 treatments)” which are not rectangular.

2. No stacks (disjoint unions): There is no NumPy equivalent for StateTracker’s stack operation.

3. No state propagation: You get index tuples, not connected state objects that track state.

4. No composable operations: Slicing, shuffling, or sampling requires manual reimplementation of all the index math.

Manual Index Math (divmod)

You can always compute indices manually using divmod, as shown in the Motivation page:

def get_product_indices(flat_idx, sizes):
    """Convert flat index to component indices for a Cartesian product."""
    indices = []
    for size in sizes:
        flat_idx, idx = divmod(flat_idx, size)
        indices.append(idx)
    return tuple(indices)


# Example: 2 treatments x 3 replicates
sizes = [2, 3]
print("Manual product enumeration:")
for flat in range(6):
    t, r = get_product_indices(flat, sizes)
    print(f"  {flat}: treatment={t}, replicate={r}")

Manual product enumeration:
  0: treatment=0, replicate=0
  1: treatment=1, replicate=0
  2: treatment=0, replicate=1
  3: treatment=1, replicate=1
  4: treatment=0, replicate=2
  5: treatment=1, replicate=2

Limitations of manual index math:

1. Does not compose: Every new operation (stack, slice, shuffle, sample) requires writing new index logic from scratch.

2. Error-prone: Off-by-one errors, wrong divisors, forgetting edge cases. Manual index math is a minefield.

3. No state tracking: You get tuples, not connected objects. You cannot ask “what is the current treatment value?” without recomputing.

4. Tedious for complex structures: As shown in the first example, combining products and stacks manually is already complicated. Adding shuffle or slice makes it worse.

What Makes StateTracker Different

The key difference is that StateTracker builds a computation DAG where:

1. Leaf states represent basic dimensions (control, treatment, replicate, etc.)

2. Operations (product, stack, slice, shuffle, etc.) create derived states

3. State propagation flows automatically from any derived state to all its ancestors

This means you declare your structure once, and StateTracker handles all the index math for every operation you compose on top.

Comparison Summary

Feature	itertools	NumPy	Manual divmod	StateTracker
Cartesian products	Yes	Yes	Yes	Yes
Disjoint unions (stack)	chain (no tracking)	No	Manual	Yes, with tracking
State propagation	No	No	No	Automatic
Composable operations	No	No	No	Yes
Shuffle/slice/sample	Must materialize	Manual	Manual	Built-in
Conflict detection	N/A	N/A	Manual	Automatic
Synchronization	N/A	N/A	Manual	Built-in

When to Use Each Approach

Use itertools when:

• You just need to iterate through all combinations once

• You do not need to track which component indices correspond to each element

• Your structure is simple (just products or just chains, not both)

Use NumPy when:

• You are working with actual array data, not just indices

• Your structure is a regular rectangular grid

• You need fast vectorized operations on the data

Use manual divmod when:

• You have a one-off simple product

• You are comfortable with the math and do not need composition

Use StateTracker when:

• You have complex structures combining products AND stacks

• You need automatic tracking of which components are active

• You want to compose operations (shuffle, slice, sample, split) without reimplementing index math

• You need synchronization between states

• You want automatic conflict detection

Summary

StateTracker’s unique value is automatic child-to-parent state propagation through a computation DAG. This is fundamentally different from:

• itertools: Provides iteration but no state tracking or composition

• NumPy: Provides index math for rectangular arrays but cannot handle stacks or complex compositions

• Manual divmod: Works for simple cases but does not compose and is error-prone

When you need to enumerate combinatorial structures that combine products, stacks, slices, shuffles, samples, or splits, while always knowing which component values correspond to each state, StateTracker is the right tool.

For more details on StateTracker’s capabilities, see:

• Quick Start - Get started with basic usage

• Core Concepts - Understand the state DAG and state propagation

• Operations - Complete reference for all operations