Apache Hamilton (Incubating)

Hamilton is a lightweight in-process framework to define, execute, and observe directed acyclic graphs (DAGs) that express data transformations. In Hamilton, one can express complex DAGs of transformations, e.g. from dataframe transformations (using pandas, polars, PySpark), machine learning pipelines, through to regular software engineering API request and LLM API based workflows. Observability hooks are built into the framework. The Hamilton UI is a self-hostable service to capture observability output from workflow runs.