Design#

Concept#

Bringing Data Step Skills into Python Workflows

While the rapid evolution of Python’s data processing ecosystem is a strength, keeping up with changes, managing dependencies, and shifting from row-oriented to column-oriented thinking can impose unexpected costs.
Data Step, on the other hand, has been used for decades with virtually unchanged syntax, and is unlikely to change significantly going forward.
The name limulus — the horseshoe crab, a living fossil — reflects this same enduring stability.
Although it has some limitations, row-oriented and procedural notation can often express certain types of processing more naturally.

While generative AI has made Python coding easier, thorough review remains especially important for data processing logic.
This library was created with the belief that carrying Data Step skills into Python can open new doors — whether for reviewing code, working on personal projects, or working independently outside a dedicated analytics environment.

For these reasons, this library is not a complete reimplementation or drop-in replacement for SAS language.
Rather than rarely-used edge cases or SAS language-specific quirks, it prioritizes practical functionality, safe and predictable behavior, and draws on design ideas from other modern libraries.

Architecture Overview#

Session.submit()
  -> DataStepExecutor.execute()
      1) apply the configured macro hook
      2) parse the complete source, identify DATA blocks, and skip recognized PROC/macros
      3) check program-level parse diagnostics before executing any block
      for each DATA block:
        4) validate dataset references and declarations
        5) plan source-owned helpers and output targets
        6) resolve and prepare Arrow inputs (options, BY, SET/MERGE, helper columns)
        7) assemble the execution and output-shaping plans
        8) execute the Rust row runtime
        9) remove internal variables and restore output types/metadata
       10) make successful outputs available to subsequent blocks
  -> Session.datasets update on successful submission

1. Unsupported PROC / Macro Skip#

The parser recognizes and skips a narrow subset of unsupported syntax so mixed source files can still reach the Data Step parser.

Current skip scope:

%let ...;
%put ...;
%* ...; macro comments
%macro ... %mend; and %macro ... %mend name;
proc ... run; and proc ... quit;

These constructs are skipped rather than executed. Open-code macro expansion and procedure semantics are still outside the supported surface.

2. Block Splitting#

Code passed to submit() is split into DATA ... RUN; units and executed sequentially.
This means that when multiple blocks are passed at once, the output of an earlier block can be referenced as input by subsequent blocks. The complete source is parsed before any block executes; program-level parse errors stop the submission at this stage.

3. Parsing#

Data Step code is currently parsed using lark. The parser produces a structured internal representation that can be inspected programmatically.

For example, the following one-line text is recognized as the structure shown below:

data out; set iris(in=in1) ; where sepal_length > 5; if species ^= 'setosa'; keep species sepal_length sepal_width ;run;

Current parsed structure (simplified):

Program
  DATA block
    DATA: output dataset "out"
    SET: input dataset "iris", IN variable "in1"
    WHERE: greater-than comparison
      column: sepal_length
      number: 5
    IF (row filter): not-equal comparison
      column: species
      string: "setosa"
    KEEP: species, sepal_length, sepal_width
    RUN: end of block

Statements, dataset references, options, and expressions are separated before execution starts.

4. Input Resolution#

SET / MERGE can read datasets loaded into the session or produced by earlier DATA blocks. Dataset and column names are case-insensitive, and the work. prefix is optional for dataset names. Column helpers such as select, where, and assign also resolve column names case-insensitively when the match is unique. Input options, BY ordering, and SET/MERGE preparation are handled in Python before Rust executes the rows.

5. Rust Row Engine#

Row-oriented processing tends to be less efficient than column-oriented processing.
To improve execution speed, a Rust-based runtime module is provided.
Rust is the only Data Step row engine. auto, explicit rust, and the python preference all select it. Errors are returned without retrying a Python row engine.

Supported apply() calls keep the row loop in Rust and invoke only the named Python callable. Other expressions are evaluated by the native runtime.

6. Session Helpers and Metadata Views#

In addition to submit(), Session and DatasetView provide user-facing helpers for common column-oriented tasks.

transpose(...) offers a compact reshape helper for common wide/long conversions.
assign(...) adds or replaces columns from expressions, literals, and case when logic while keeping left-to-right assignment order.
assign(...) and the DSL both support limited built-in put(...), input(...), and hour(...) conversion helpers.
Session maintains a shared format / informat registry. Built-in families cover best. / w. / w.d / w.d., zw. / zw.d / zw.d., commaw. / commaw.d / commaw.d., and named forms such as e8601da., e8601dt., yymmdd6., yymmdd8., yymmdd10., and time.. User catalogs must be mappings serializable to the Rust runtime; callable format/informat registration is unsupported.
astype(...) / cast(...) remain helper-oriented type conversion APIs, can materialize into a new dataset by using out=, and can write cast results into new columns by using alias=.
sql(...) provides practical session-level querying, including read queries, CREATE TABLE ... AS ..., and DROP TABLE ....
dictionary.tables, dictionary.columns, and dataset-scoped dictionary views expose session metadata such as labels, row counts, and Arrow column types.

7. Row Loop Processing#

The Rust runtime manages the basic per-row loop processing and its associated automatic variables.

_N_: incremented on each row iteration
_ERROR_: initialized to 0 at the start of each row; set to 1 on execution errors
BY assigns FIRST.<var> / LAST.<var> automatic variables per row
Output destination is controlled with DATA out1 out2; and OUTPUT out1;
If no explicit OUTPUT is present, the default output destination (usually the first DATA target) is used

8. Output Conversion and Session Update#

Processed results are finalized as Arrow tables and then reflected in the catalog. This allows session["name"] to be retrieved as an Arrow Table.

In the output stage, internal temporary variables are also removed in a backend-agnostic way. This includes helper variables created by IN=, INDSNAME=, END=, and FIRST./LAST. (including renamed aliases).

Dataset labels are stored in Arrow schema metadata under memlabel, and column labels are stored in each Arrow field’s custom metadata. This allows label information produced by DATA ... (label="...") and LABEL statements to survive round-trips through Session.

Successful submissions add their output tables to Session.datasets.

9. Stage-aware Diagnostics and Logs#

Pipeline diagnostics keep stage information as a dedicated field (stage) rather than embedding it in diagnostic messages. Session.submit() propagates this value into log entries so each log line can identify the pipeline stage independently.

When source text is available, log output also includes short source excerpts. This makes parse, validation, and execution errors easier to read without requiring users to inspect internal parser details.

Input Data#

Arrow, Polars and Pandas are supported as public input formats. For CSV, Parquet, sas7bdat, or other file formats, load them first with any library of your choice.
Internally, limulus normalizes supported inputs to Arrow for data exchange, so Arrow or Polars inputs are recommended for best performance.

Performance#

Handling large datasets is a key motivation for using Python, so performance is an explicit design consideration.

Row-oriented processing is inherently slower than columnar processing, which operates on entire columns at once.
For that reason, the main Data Step runtime focuses on predictable row-loop semantics, while user-facing helpers such as assign() can use a column-oriented expression path for supported syntax.

For wide or performance-sensitive pure column operations, assign(...), cast(...) / astype(...), and direct Arrow / Polars usage are often better fits than forcing row-oriented logic. limulus aims to keep those boundaries explicit rather than promising blanket parity with columnar engines.

For reference, limulus uses the following iris-like neutral scenario for benchmark comparisons:

Processing scenario:

data setosa_like others;
  set flowers;
  where sepal_length > 4.5;
  sepal_area = round(sepal_length * sepal_width, 0.01);
  petal_ratio = round(petal_length / petal_width, 0.01);
  if petal_length > 2.5 then do;
    segment = "others";
    output others;
  end ;
  else do;
    segment = "setosa_like";
    output setosa_like;
  end ;
run;

For row-oriented execution, limulus is still slower than polars.iter_rows, but it remains substantially faster than pandas.iterrows for the same workload. limulus also carries a fixed overhead of roughly 50 ms for parsing, preprocessing, and related setup work regardless of input size. As the dataset grows and that fixed-cost share becomes smaller, limulus tends to run at about four times the runtime of Polars and about one third of the runtime of pandas for this row-oriented benchmark. This comparison uses pandas.iterrows(), which is often chosen for its simple column-name-based access even though pandas.itertuples() is usually much faster and can be much closer to Polars row iteration performance for the same workload.

rows	limulus_ms	pandas_iterrows_ms	polars_iterrows_ms
10000	94.17	147.32	9.06
100000	455.67	1433.75	111.30
1000000	4220.09	14675.22	980.06

When the same transformation is expressed through limulus.assign(...), performance improves substantially because the helper uses a Polars-based columnar execution path internally. In practice, this makes it much closer to Polars than to row-oriented execution. This path also carries a fixed overhead of roughly 50 ms, so it remains somewhat slower than other columnar-style processing at small sizes. However, at around 100k rows it can already run faster than polars.iter_rows for the same transformation.

rows	limulus_assign_ms	pandas_ms	polars_ms
10000	54.11	1.93	0.81
100000	63.57	8.57	2.72
1000000	108.52	105.83	28.25

Design

Contents