SuperGLM

SuperGLM

Penalised generalised linear model with splines, group penalties, and REML.

Supports Poisson, Gaussian, Gamma, NB2, Tweedie, and Binomial families with group lasso, sparse group lasso, or ridge penalties. Smoothing parameters can be estimated via REML (fit_reml) or cross-validation (cross_validate).

__init__(family='poisson', link=None, penalty=None, selection_penalty=None, spline_penalty=None, penalty_features=None, features=None, splines=None, n_knots=10, degree=3, categorical_base='most_exposed', interactions=None, active_set=False, direct_solve='auto', discrete=False, n_bins=256, tol=1e-06, max_iter=100, convergence='deviance', retain_fit_state=True)

Parameters:

Name	Type	Description	Default
family	str or Distribution	Response distribution. Strings "poisson", "gaussian", "gamma", "binomial" are accepted for parameter-free families. For parameterized families use Distribution objects: Tweedie(p=1.5), NegativeBinomial(theta=1.0), or the families module (e.g. families.tweedie(p=1.5)). For "binomial", y must be in {0, 1} and predict() returns probabilities.	'poisson'
link	str or Link	Link function. Defaults to the family's configured default link.	None
penalty	str or Penalty	Penalty type. One of "group_lasso", "sparse_group_lasso", "group_elastic_net", "ridge", or a Penalty object. Defaults to GroupLasso.	None
selection_penalty	float, {"auto"}, or None	Regularisation strength for the group penalty (feature selection). None (default) and 0.0 disable selection. "auto" explicitly requests calibration to 10% of lambda_max at fit time.	None
spline_penalty	float	Within-group ridge shrinkage for spline smoothing. Defaults to 0.1.	None
penalty_features	str or list[str]	Restrict the selection penalty to specific feature or group names. None (default) applies to all penalizable groups.	None
features	dict[str, FeatureSpec]	Explicit feature specifications mapping column names to feature objects (Spline, Categorical, Numeric, Polynomial). Mutually exclusive with splines.	None
splines	list[str]	Column names to treat as splines in auto-detect mode. All other columns are auto-detected as categorical or numeric. Mutually exclusive with features.	None
n_knots	int or list[int]	Number of interior knots for auto-detect splines.	10
degree	int	B-spline degree for auto-detect splines.	3
categorical_base	str	Base level strategy for auto-detected categoricals.	'most_exposed'
interactions	list[tuple[str, str]]	Pairs of feature names to interact. Interaction type is auto-detected from the parent feature specs.	None
active_set	bool	Use active-set cycling in the BCD solver.	False
direct_solve	('auto', 'gram', 'qr')	Strategy for the direct IRLS solver (lambda1=0). "auto" uses gram-based Cholesky with residual-checked SVD fallback, warning after repeated fallbacks. "gram" forces the gram path without warnings. "qr" uses QR on the materialised weighted design matrix — backward-stable but O(n·p²) per iteration. Intended for smaller datasets.	"auto"
discrete	bool	Use discretized basis matrices for large-n REML (fREML-style).	False
n_bins	int or dict[str, int]	Number of discretization bins per feature when discrete=True.	256
tol	float	Convergence tolerance for IRLS / PIRLS. Default 1e-6. Can also be set per-call via fit(tol=...) or fit_reml(pirls_tol=...). Fit-time values take precedence. Larger values (e.g. 1e-6) converge faster but may stop before near-separated coefficients have stabilised.	1e-06
max_iter	int	Maximum IRLS / PIRLS outer iterations. Default 100.	100
convergence	('deviance', 'coefficients')	Convergence criterion. "deviance" (default) stops when relative deviance change drops below tol — fast, since well-identified coefficients lock in early. "coefficients" (experimental) stops when the maximum relative coefficient change drops below tol. May not converge for near-separated levels where the MLE is at −∞.	"deviance"
retain_fit_state	bool	If True (default), keep training-scale fit state such as the fitted design matrix for later diagnostics. If False, eagerly computes compact inference state after fitting, then releases row-scale training caches while preserving prediction, summaries, and term confidence intervals.	True

fit(X, y, sample_weight=None, offset=None, *, tol=None, max_iter=None, convergence=None, record_diagnostics=False)

Fit the model to data.

Parameters:

Name	Type	Description	Default
X	pandas or eager Polars DataFrame	Feature matrix with columns matching registered features. Lazy frames must be collected before fitting.	required
y	array - like	Response variable.	required
sample_weight	array - like	Observation weights. Their likelihood interpretation is family-specific. Defaults to 1 for all observations. For Tweedie, these are finite, strictly positive EDM prior weights: Y_i ~ Tw_p(mu_i, phi / w_i), so Var(Y_i | x_i) = phi * mu_i**p / w_i. They are not replication counts; zero or non-finite weights are rejected. Non-Tweedie families retain their existing weighting behavior. Weights affect fitting but do not enter the linear predictor or automatically scale the conditional mean. The model mean is mu_i = g**-1(x_i.T @ beta + offset_i).	None
offset	array - like	Offset added to the linear predictor. sample_weight does not supply an offset. To make a raw-count mean scale with exposure, pass offset=np.log(exposure). For a per-exposure response, pass exposure as sample_weight without adding it again to the linear predictor.	None
record_diagnostics	bool	If True, record per-iteration IRLS diagnostics (W range, mu/eta range, step halvings, worst-observation indices) on result.iteration_log. Useful for debugging convergence.	False

Returns:

Type	Description
SuperGLM	The fitted model (self).

fit_reml(X, y, sample_weight=None, offset=None, *, max_reml_iter=20, reml_tol=1e-06, pirls_tol=None, max_pirls_iter=None, lambda2_init=None, interaction_mode='full', runtime_validation='auto', verbose=False, w_correction_order=1)

Fit with REML estimation of per-term smoothing parameters.

fit_reml() is the smoothness-selection path and does not support a selection penalty: configure selection_penalty=None or 0.0. It optimizes a Laplace approximate REML objective over per-term smoothing parameters. For sparse/group selection, use fit() or fit_path(). To let REML shrink spline null spaces, use select=True on the spline terms instead.

Parameters:

Name	Type	Description	Default
X	pandas or eager Polars DataFrame	Feature matrix. Lazy frames must be collected before fitting.	required
y	array - like	Response variable.	required
sample_weight	array - like	Observation weights. Their likelihood interpretation is family-specific. For Tweedie, these are finite, strictly positive EDM prior weights with Var(Y_i | x_i) = phi * mu_i**p / w_i; they are not replication counts. Weights affect fitting but do not enter the linear predictor or automatically scale the conditional mean. Non-Tweedie families retain their existing weighting behavior.	None
offset	array - like	Offset term.	None
max_reml_iter	int	Maximum REML outer iterations (default 20).	20
reml_tol	float	Convergence tolerance on log-lambda (default 1e-6).	1e-06
pirls_tol	float	Inner PIRLS/IRLS convergence tolerance. Defaults to constructor tol (1e-6). Pass explicitly to override.	None
max_pirls_iter	int	Maximum inner PIRLS iterations per REML step. Defaults to constructor max_iter (100).	None
lambda2_init	float	Initial per-group lambda. Defaults to self.lambda2.	None
interaction_mode	('full', 'fast_candidate')	"full" runs ordinary REML. "fast_candidate" caps REML outer updates for interaction models and then runs the normal final refit, intended for screening candidate interactions before a full final model fit.	"full"
runtime_validation	('auto', 'full', 'skip')	Controls the post-fit public-runtime parity diagnostic. "auto" validates small fits and skips the full training-row diagnostic for large fits or fast candidate interaction fits. "full" always validates; "skip" skips validation while still canonicalizing the public prediction state.	"auto"
verbose	bool	Print progress.	False
w_correction_order	int	Order of the W(rho) implicit-differentiation correction. 1 gives the exact objective and gradient with a modified-Newton outer Hessian (default, fast). 2 also includes the available exact d²W/dη² Hessian cross-terms from Wood (2011, Appendix C). Only affects the exact REML path.	1

Returns:

Type	Description
SuperGLM	The fitted model (self).

fit_path(X, y, sample_weight=None, offset=None, *, n_lambda=50, lambda_ratio=0.001, lambda_seq=None)

Fit a regularization path from lambda_max down to lambda_min.

Warm-starts each lambda from the previous solution.

predict(X, offset=None)

Predict the response mean for new data.

Parameters:

Name	Type	Description	Default
X	pandas or eager Polars DataFrame	Eager input features with the same columns used during fitting.	required
offset	NDArray or None	Optional offset added to the linear predictor before applying the inverse link.	None

Returns:

Type	Description
NDArray	Predicted mean on the response scale (inverse-link of eta).

design_summary()

Describe fitted design storage and static route eligibility.

The summary does not build an accelerated matrix or prove that an eligible kernel executed. Fit and REML traces remain authoritative for actual dispatch.

relativities(with_se=False, centering='native')

Extract plot-ready relativity DataFrames for all features.

Parameters:

Name	Type	Description	Default
centering	('native', 'mean')	"native" (default) returns the canonical fitted term contribution under the model's identifiability constraint. "mean" is a reporting convenience that shifts so the geometric mean of relativities = 1 — useful for cross-feature comparison but not the fitted term decomposition.	"native"

term_inference(name, *, with_se=True, simultaneous=False, n_points=200, alpha=0.05, n_sim=10000, seed=42, centering='native')

Per-term inference: curve, uncertainty, and metadata in one object.

Parameters:

Name	Type	Description	Default
centering	('native', 'mean')	"native" (default) returns the canonical fitted term contribution under the model's identifiability constraint. "mean" is a reporting convenience that shifts so the geometric mean of relativities = 1.	"native"

plot(terms=None, *, kind='global', ci='pointwise', X=None, sample_weight=None, show_density=True, show_knots=False, show_bases=False, scale='response', ci_style='band', categorical_display='auto', grouped_level_display='auto', engine='matplotlib', n_points=200, figsize=None, title=None, subtitle=None, plotly_style=None, alpha=0.05, n_sim=10000, seed=42, centering='native', **kwargs)

Plot model terms.

Single entry point for all plotting. Dispatches based on terms:

• None — all main effects in a grid.
• "age" — one main effect.
• ["age", "region"] — subset of main effects.
• "age:region" — one interaction.

Parameters:

Name	Type	Description	Default
terms	str, list of str, or None	Which term(s) to plot. None plots all main effects.	None
kind	('global', 'local')	"global" shows model-wide fitted effects (default). "local" is reserved for per-row explanations (not yet implemented).	"global"
ci	(None, False, 'pointwise', 'simultaneous', 'both')	Confidence interval style. None or False disables bands.	None
X	pandas or eager Polars DataFrame	Training data for density overlays.	None
sample_weight	array - like	Frequency weights / sample_weight for density overlays.	None
show_density	bool	Show sample_weight/observation density (strip for continuous, bars for categorical). Default True.	True
show_knots	bool	Show interior knot ticks (spline terms only).	False
show_bases	bool	Initial visibility for coefficient-weighted spline basis contributions in the Plotly explorer. Only meaningful when scale="link"; ignored in response-scale mode and by the matplotlib renderer.	False
scale	('response', 'link')	"response" (default) shows the fitted effect on the inverse-link scale (relativities). With centering="native", this is the exponentiated fitted term contribution under the model's identifiability constraint — not a portfolio-average relativity. "link" shows the additive link-scale contribution eta(x) = B(x) @ beta, with optional basis decomposition overlays. Only used by the Plotly renderer.	"response"
ci_style	('band', 'lines')	Plotly CI presentation. "band" (default) draws filled confidence bands. "lines" draws line-only CI bounds with no fill.	"band"
categorical_display	('auto', 'bars', 'markers', 'bars+markers')	Plotly categorical rendering mode. "auto" (default) uses bars+markers up to 30 levels and markers-only above that.	"auto"
grouped_level_display	('auto', 'expanded', 'collapsed')	Display option for grouped categorical levels in main-effect plots. "auto" collapses grouped ordered-categorical terms and leaves unordered categoricals expanded. This is a plotting-only option; scoring, inference tables, and exports remain expanded over the original levels.	"auto"
engine	('matplotlib', 'plotly')	Plotting backend. "matplotlib" is the chart/export path for single terms and grids. "plotly" is the interactive main-effect explorer path, with a response/link scale toggle and term selector. For main effects, Plotly requires at least two terms (or terms=None); use engine="matplotlib" for a single-term chart. Requires the plotly optional dependency (pip install superglm[plotting]).	"matplotlib"
centering	('native', 'mean')	"native" (default) returns the canonical fitted term contribution under the model's identifiability constraint. "mean" is a reporting convenience that shifts so the geometric mean of relativities = 1.	"native"
n_points	int	Grid resolution for spline/polynomial curves.	200
figsize	tuple	Figure size override.	None
title	str	Figure-level title and subtitle.	None
subtitle	str	Figure-level title and subtitle.	None
plotly_style	dict	Plotly main-effect explorer style overrides. Supported keys include line_color, bar_color, density_fill_color, density_edge_color, error_bar_color, text_color, and text_outline_color. Ignored by the matplotlib renderer.	None
alpha	float	Significance level for CIs (default 0.05).	0.05
n_sim	int	Posterior simulations for simultaneous bands.	10000
seed	int	Random seed for simultaneous bands.	42
**kwargs		Forwarded to the underlying renderer (e.g. ncols for grid plots, colormap for interactions).	{}

Returns:

Type	Description
Figure or Figure

Examples:

>>> fig = model.plot(engine="plotly", X=X_train, sample_weight=w)
>>> fig.show()                      # interactive main-effect explorer
>>> fig.write_html("effects.html") # standalone HTML export

metrics(X, y, sample_weight=None, offset=None)

Compute comprehensive diagnostics for the fitted model.

refit_unpenalised(X, y, sample_weight=None, offset=None, *, keep_smoothing=True)

Refit with only active features and no selection penalty.

drop1(X, y, sample_weight=None, offset=None, *, test='Chisq')

Drop-one deviance analysis for each feature.

estimate_theta(X, y, sample_weight=None, offset=None, **kwargs)

Estimate NB theta via profile likelihood, refit, and return result.

estimate_p(X, y, sample_weight=None, offset=None, *, fit_mode='fit', phi_method='mle', method='auto', ci_alpha=None, **kwargs)

Estimate Tweedie p via profile likelihood, refit, and return result.

Parameters:

Name	Type	Description	Default
X	pandas or eager Polars DataFrame	Feature matrix. Lazy frames must be collected before fitting.	required
y	array - like	Response variable.	required
sample_weight	array - like	Finite, strictly positive EDM prior weights, not replication or frequency weights. The Tweedie variance convention is Var(Y_i | x_i) = phi * mu_i**p / w_i. Remove zero-weight rows consistently from X, y, sample_weight, and offset before calling this method.	None
offset	array - like	Offset added to the linear predictor.	None
fit_mode	('fit', 'reml', 'inherit')	Fitting regime for each candidate p evaluation.	"fit"
phi_method	('pearson', 'mle')	How to profile out Tweedie dispersion phi at each candidate p. "mle" (default) maximizes the likelihood in phi; the joint fast path uses exact derivatives and defensive searches use a nested scalar optimization. "pearson" is an explicit faster plug-in and does not support likelihood-ratio confidence intervals.	"pearson"
method	('auto', 'joint_ml', 'brent', 'grid', 'grid_refine', 'profile_opt')	Search strategy. "auto" (default) uses safeguarded exact joint ML for ordinary MLE profiles and Brent otherwise. "joint_ml" explicitly requests that fast path within its stable p range and falls back defensively otherwise. "brent" uses bounded scalar optimisation. "grid" does exhaustive grid search. "grid_refine" does a coarse grid + local Brent refinement. "profile_opt" uses a general-purpose optimizer on logit-transformed p.	"auto"
ci_alpha	float	Significance level for an explicitly requested likelihood-ratio profile confidence interval. For example, 0.05 computes a 95% interval and caches it for model.summary(alpha=0.05). The default None performs no confidence-interval evaluations.	None