API Reference

API Reference#

Public API for using limulus from Python.

Top-Level Functions#

Convenience functions that can be executed as one-shot operations without creating a session.

limulus.submit(code, *, backend='auto', **datasets)[source]#

One-shot execution (when session management is not needed).

Parameters:

  • code (str) – Data Step DSL text to execute.

  • backend (str) – Runtime backend to use, one of "rust", "python", or "auto". Defaults to "auto".

Return type:

SubmitResult

Examples

>>> import limulus, pyarrow as pa
>>> result = limulus.submit("data out; set inp; run;", inp=pa.table({"x": [1, 2]}))
>>> result.success
True
limulus.run(code, *, backend='auto', **datasets)[source]#

Alias for submit().

Return type:

SubmitResult

Session#

The main class responsible for dataset management and Data Step execution.

class limulus.Session(*, backend='auto', runtime_backend=None, parser_backend='python', options=None)[source]#

Bases: object

dataset(name)[source]#

Returns an operation view for the specified dataset.

Use this when you want to chain keep() / drop() / where() / rename() / sort() calls.

Parameters:

name (str) – Name of the dataset to create a view for.

Return type:

DatasetView

Returns:

A DatasetView instance.

Examples

>>> df = session.dataset("bmi").where("bmi > 20").keep(["name", "bmi"]).to_pandas()
property datasets: DatasetCatalog#

Returns the session catalog (DatasetCatalog).

Supports dict-like access (session.datasets["name"]), in operator, and iteration.

Note

session["name"] is syntactic sugar for session.datasets["name"].

delete(*names, missing_ok=True)[source]#

Alias for unload().

Return type:

bool

property dictionary: DictionaryProvider#
get_log()[source]#

Most recent SubmitResult.

This method is equivalent to log and is provided for cases where method-style access is preferred.

Returns None until the first submit()/run() call.

Return type:

SubmitResult | None

get_option(name=None, default=None)[source]#

Returns session options as a value or dictionary.

When name is omitted, returns a copy of all options. When a list of keys is given, returns a dictionary for those keys. When a single key is given, returns its value or default.

Return type:

Any

include(path)[source]#

Reads a DSL file and executes it via submit().

Parameters:

path (str) – Path to a UTF-8 encoded Data Step script.

Return type:

SubmitResult

Returns:

The resulting SubmitResult.

load(name, data)[source]#

Registers a single dataset in the session catalog.

Parameters:

  • name (str) – Dataset name referenced from Data Step code (case-insensitive).

  • data (Any) –

    Data to register. Supported types:

    • pyarrow.Table

    • polars.DataFrame

    • pandas.DataFrame

    • list[dict] (list of dicts; normalized to Arrow-backed storage)

Return type:

None

Examples

>>> import limulus, pyarrow as pa
>>> session = limulus.Session()
>>> session.load("mydata", pa.table({"x": [1, 2, 3]}))
loads(datasets=None, **named_datasets)[source]#

Registers multiple datasets at once.

Parameters:

  • datasets (Mapping[str, Any] | None) – A mapping of dataset names to data.

  • **named_datasets (Any) – Can also be passed as keyword arguments (compatible with datasets).

Return type:

None

Examples

>>> session.loads({"a": df_a, "b": df_b})
>>> session.loads(a=df_a, b=df_b)  # also accepted as keyword arguments
property log: SubmitResult | None#

Most recent SubmitResult.

Returns None until the first submit()/run() call.

register_format(name, formatter, *, namespace=None)[source]#
Return type:

None

register_informat(name, parser, *, kind=None)[source]#
Return type:

None

run(code, *, backend=None)[source]#

Alias for submit().

Return type:

SubmitResult

sql(query, out=None, *, target=None)[source]#

Executes SQL against session datasets.

Parameters:

  • query (str) – SQL text executed against datasets currently loaded in the session.

  • out (str | None) – Optional explicit output dataset name.

Returns:

A pyarrow.Table containing the query result.

SQL execution is backed by the Polars SQL engine. https://docs.pola.rs/api/python/stable/reference/sql/index.html If the SQL starts with CREATE TABLE name AS ..., the result is also stored in the session catalog under name.

submit(code, *, backend=None, show_result=False)[source]#

Submits Data Step code to the session and executes it.

Resulting datasets are added to or updated in the session catalog. Datasets created by a previous submit() call can be referenced in subsequent calls.

Parameters:

  • code (str) – Data Step DSL text to execute. Multiple DATA ... RUN; blocks may be included.

  • backend (str | None) – Optional runtime backend override for this call only. One of "rust", "python", or "auto". When given, temporarily overrides the session-level backend setting.

  • show_result (bool) – Controls notebook/REPL representation of the returned SubmitResult. Defaults to False for Session.submit() to keep successful calls quiet.

Return type:

SubmitResult

Returns:

A SubmitResult. Check result.success to determine whether errors occurred.

Examples

>>> rc = session.submit("""
... data result;
...   set mydata;
...   bmi = round(weight / height**2, 0.1);
...   keep name bmi;
... run;
... """)
>>> print(session["result"].to_pandas())
to_arrow(name)[source]#

Returns a dataset as a pyarrow.Table.

Parameters:

name (str) – Dataset name.

Returns:

pyarrow.Table.

to_pandas(name)[source]#

Converts a dataset to a pandas.DataFrame and returns it.

Parameters:

name (str) – Dataset name.

Returns:

pandas.DataFrame.

to_polars(name)[source]#

Converts a dataset to a polars.DataFrame and returns it.

Parameters:

name (str) – Dataset name.

Returns:

polars.DataFrame.

unload(*names, missing_ok=True)[source]#

Removes one or more datasets from the session.

Parameters:

  • *names (str | Sequence[str]) – Dataset name(s) to remove. Accepts strings or sequences, e.g. "a", "b" or ["a", "b"].

  • missing_ok (bool) – If False, raises KeyError for datasets that do not exist. Defaults to True (silently ignores missing datasets).

Return type:

bool

Returns:

True if at least one dataset was removed.

Examples

>>> session.unload("tmp")
>>> session.unload("a", "b")
>>> session.unload(["a", "b"])
property work: DatasetCatalog#

Alias for datasets.

DatasetView#

A view class returned by :meth:Session.dataset for chained operations.

class limulus.session.DatasetView(session, name)[source]#

Bases: object

A view for chained operations on a dataset.

Obtained via Session.dataset(). You can chain select() / keep() / drop() / where() / rename() / sort() calls.

Examples

>>> view = session.dataset("bmi")
>>> df = view.where("bmi > 20").keep(["name", "bmi"]).to_pandas()
apply_options(*, keep=None, drop=None, rename=None, where=None, out=None)[source]#

Applies keep/drop/rename/where in a single call.

Parameters:

  • keep (Sequence[str] | None) – List of column names to retain.

  • drop (Sequence[str] | None) – List of column names to remove.

  • rename (Mapping[str, str] | None) – A {old_name: new_name} mapping dict.

  • where (str | None) – Filter expression (e.g. "age > 13").

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

assign(out=None, **assignments)[source]#

Adds or replaces columns using ordered expressions or literals.

Parameters:

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

  • **assignments (Any) – Column assignments evaluated from left to right. String values are interpreted as expressions. Non-string values are treated as literals.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

Note

Assignments are translated to ordered Polars expressions and evaluated from left to right. Expressions or functions that cannot be translated to the column-oriented assign pipeline raise ValueError instead of falling back to Python row materialization.

astype(mapping, out=None, *, alias=None)[source]#

Casts columns using a {column: dtype} mapping.

Parameters:

  • mapping (Mapping[str, str]) – Mapping from column name to dtype string.

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

  • alias (str | Mapping[str, str] | None) – Optional new column name, or {source: alias} mapping, for writing cast results into new columns instead of overwriting the source.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

Note

Column names are resolved case-insensitively when the match is unique.

cast(mapping, out=None, *, alias=None)[source]#

Alias for astype().

Return type:

DatasetView

property dictionary: pyarrow.Table#
drop(columns, out=None)[source]#

Removes the specified columns (equivalent to DROP).

Parameters:

  • columns (Sequence[str]) – List of column names to remove.

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

keep(columns, out=None)[source]#

Retains only the specified columns (equivalent to KEEP).

Parameters:

  • columns (Sequence[str]) – List of column names to keep.

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

property name: str#
rename(mapping, out=None)[source]#

Renames columns (equivalent to RENAME).

Parameters:

  • mapping (Mapping[str, str]) – A {old_name: new_name} mapping dict.

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

select(columns, out=None)[source]#

Retains only the specified columns.

Parameters:

  • columns (Sequence[str]) – List of column names to keep.

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

sort(by, out=None, *, nodupkey=False)[source]#

Sorts the dataset by columns.

Parameters:

  • by (Sequence[str]) – List of column names to sort by. Can also be a list of (column_name, "ascending" | "descending") tuples to specify direction.

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

  • nodupkey (bool) – When True, keeps only the first row for each unique key defined by by after sorting.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

to_arrow()[source]#

Returns this dataset as a pyarrow.Table.

to_pandas()[source]#

Converts this dataset to a pandas.DataFrame and returns it.

to_polars()[source]#

Converts this dataset to a polars.DataFrame and returns it.

transpose(*, by=None, id=None, var=None, out=None)[source]#

Transposes the dataset using PROC TRANSPOSE-like defaults.

Parameters:

  • by (str | Sequence[str] | None) – Grouping columns preserved in the output.

  • id (str | None) – Single column name expanded into output column names.

  • var (str | Sequence[str] | None) – Value columns to transpose. When omitted, uses all non-by/id columns.

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

Note

This helper is intended as a convenience API with PROC TRANSPOSE-like defaults. Performance is not a primary goal. For performance-critical reshaping, prefer Arrow or Polars directly.

unload(*, missing_ok=True)[source]#

Removes this dataset from the session.

Parameters:

missing_ok (bool) – If False, raises KeyError when the dataset does not exist.

Return type:

bool

Returns:

True if the dataset was successfully removed.

where(expression, out=None)[source]#

Filters rows using a simple comparison expression (equivalent to WHERE).

Parameters:

  • expression (str) – Filter expression, e.g. "age > 13", "sex = 'M'".

  • out (str | None) – Output dataset name. If omitted, overwrites this view’s dataset.

Return type:

DatasetView

Returns:

A DatasetView for the resulting dataset.

SubmitResult#

The return value of :meth:Session.submit.

class limulus.SubmitResult(success, datasets=<factory>, log=<factory>, elapsed_seq=0.0, display_result=True)[source]#

Bases: object

datasets: Mapping[str, Any]#
display_result: bool = True#
elapsed_seq: float = 0.0#
format_log()[source]#
Return type:

str

log: tuple[LogEntry, ...]#
print_log()[source]#
Return type:

None

success: bool#

LogEntry#

A class representing a single entry in the execution log.

class limulus.LogEntry(code, severity, message, location='', stage='', span=None, labels=<factory>, notes=<factory>, source_text=None)[source]#

Bases: object

code: str#
labels: tuple[DiagnosticLabel, ...]#
location: str = ''#
message: str#
notes: tuple[str, ...]#
severity: str#
source_text: str | None = None#
span: DiagnosticSpan | None = None#
stage: str = ''#

Column-Oriented Helpers#

Session and DatasetView expose reshape and derived-column helpers as part of the public API.
Dataset names are resolved case-insensitively, and a work. prefix is stripped during lookup. These helpers include procedure-like operations such as sort(...) and transpose(...), along with assign(...) for fast, simple column-oriented transformations.

assign(...) evaluates assignments from left to right and treats string values as expressions rather than string literals. It is intended for simple derived-column work such as literals, arithmetic and comparison expressions, case when, and the built-in helper functions. Unsupported constructs raise ValueError.

Registries#

Built-in format / informat families currently cover w., w.d, w.d., zw., zw.d, zw.d., commaw., commaw.d, commaw.d., plus named forms such as e8601da., e8601dt., yymmdd6., yymmdd8., yymmdd10., and time.. For numeric formats, w is the total width, d is the number of fractional digits, and omitted d defaults to 0. Bare w / zw / commaw without any period are rejected. limulus does not space-pad values when the rendered text is shorter than w; zw.d zero-fills instead.

The shared registry also supports exact-match dict catalogs:

  • Session.register_format(name, mapping) for numeric put(...) lookups

  • Session.register_format(name, mapping, namespace="character") for $name. character-format lookups

  • Session.register_informat(name, mapping) for float64-only input(...) lookups

Dict put(...) catalogs return text and fall back to the original value on no-match. Dict input(...) catalogs return null on no-match. Callable formatter / parser registration remains supported for advanced Python-side hooks; callable custom input(...) may still need kind= so assign(...) can determine the output dtype.

DatasetView.astype(...) / DatasetView.cast(...) also support out= so type-converted results can be written into a new dataset instead of replacing the source. When the cast result should be written into a new column instead of replacing the source, use alias=.

They also expose dynamic dictionary metadata views:

  • session.dictionary.tables: dataset-level metadata for the current session

  • session.dictionary.columns: column-level metadata for all session datasets

  • session.dictionary("name"): column metadata for one dataset

  • session.dataset("name").dictionary: dataset-scoped dictionary view

Diagnostics#

Execution diagnostics now carry structured source metadata in addition to message text.

  • DiagnosticSpan: source offset and line/column range for a primary location

  • DiagnosticLabel: labeled span metadata for renderer input

  • LogEntry: stage, span, labels, notes, and optional source text preserved from execution diagnostics

SubmitResult.format_log() uses these fields to render excerpt-based plain-text diagnostics when source text is available, and falls back to the older message-only format otherwise.

The same renderer path is used across parse, validate, and execute stages. This also applies to non-DSL parse points such as case when, simple filter / SQL classification, and runtime regex parsing, so source-aware diagnostics share the same display contract regardless of where the error originated.

Contents