Write a dataframe transformation
Writes pandas or polars code for a described transformation with built-in checks on row counts, nulls, key uniqueness and join cardinality. Use when reshaping, joining or aggregating data.
Dataframe code usually fails silently, not loudly: a join on a key that is not unique multiplies rows, a left join leaves nulls that later vanish in an aggregation, a string key with trailing spaces matches nothing, and dates parsed in the wrong format shift by months. The output looks plausible and is wrong. Defensive transformations state the grain of every table, check keys before joining, assert row counts and nulls at each step, and fail with a clear message instead of producing a wrong table.
Write code that turns this input:
into this output:
- State the grain (what one row represents) and the key of each input and of the output.
- Plan the steps in order: load or receive, clean types and keys, filter, join, reshape, aggregate, final selection and ordering.
- Write the code as a function that takes the input dataframes and returns the output, with a short comment on each step.
- After each step that can change row counts or introduce nulls, add a check:
- keys: uniqueness on the side that should be unique, before every join;
- joins: the expected cardinality (pandas
merge(..., validate="many_to_one"), polarsjoin(..., validate="m:1")) and a count of unmatched keys; - row counts: expected equal, smaller or larger than before, and by how much;
- nulls: in key columns and in columns the output requires;
- aggregates: totals that should be preserved (for example the sum of amounts before and after reshaping).
- Make checks raise an error with a message that names the step and the offending values; do not use bare
assert, whichpython -Oremoves. - Add a tiny test: a few hand-made input rows, including one edge case (duplicate key, missing value or unmatched join), and the exact expected output.
- Use idiomatic, vectorised : for pandas, method chaining where it stays readable,
.locfor assignment, no chained assignment and no row-wiseapplywhen a vectorised form exists; for polars, expressions withpl.col, and the lazy API for large data. - Write code compatible with current stable releases, and name any feature that needs a recent version.
- Do not guess column names, types or business rules. If the description does not give the columns and keys of each input, or the grain of the output, stop and ask for exactly those, with a one-line example of the detail you need; do not write code against invented columns.
- For smaller gaps (for example which duplicate to keep, or how to treat unmatched rows), choose the safest behaviour, list it under Assumptions, and make it easy to change.
- Keep it self-contained: imports at the top, no reading from paths you invented; take dataframes as parameters.
Assumptions
Bullets: grains, keys and every assumption made.
Code
One fenced Python block with the function and its checks.
What the checks catch
A table: check | step | the failure it prevents.
Test
A fenced Python block with the small test and its expected output.
2 required values still a placeholder; the assistant will ask for them.
details
- kind
- Prompt: a task you run by name to get one finished thing back
- domain
- Data analysis
- category
- Data exploration
- level
- Intermediate
- made for
- Data analyst, Data scientist, Data engineer
- risk
- read-only
- version
- v1.1.0 · experimental
- reviewed
- 2026-10-02
- works in
- Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Antigravity, OpenCode, Windsurf, Zed, Continue, AGENTS.md, ChatGPT, claude.ai
use in
npx @hermes-hq/hodios install write-dataframe-transformation --target claude-codenpx skills add hermes-hq/hodios-dist --skill write-dataframe-transformation -a claude-codeclaude plugin marketplace add hermes-hq/hodios-distclaude plugin install hodios-data-analysis@hodiosThe plugin brings every entry in this domain at once.
pairs well with
All of Data explorationReconcile two datasets
Reconciles two datasets that should agree, such as bank versus ledger or CRM versus billing, by matching records, listing mismatches and explaining likely causes. Use for month-end checks.
reconcile-datasetsConsulting statistician
Consulting statistician who asks how the data were produced before analysing them, chooses methods that fit the question, checks assumptions and refuses to over-claim. Use for any data analysis.
statisticianAnalyse website analytics
Analyses website analytics (GA4 or similar) for traffic sources, landing pages, engagement and conversion, flags tracking problems first and gives prioritised actions. Use as a marketer or site owner.
analyze-web-analyticsClassify text records
Classifies free-text records such as tickets, feedback or expenses into a given set of categories, with a confidence level and an explicit Other bucket, and returns a table.
classify-text-recordsAnalyse survey results
Analyses quantitative survey responses with cleaning, tabulation, cross-tabs and optional weighting, and states the caveats about sample and response bias. Use before reporting survey numbers.
analyze-survey-resultsData analyst
Acts as a data analyst who starts from the decision, sanity-checks data before trusting it and states uncertainty plainly. Use as a standing analyst persona or subagent for data questions.
data-analyst