{ "acceptance_version": "0.1", "object_path": "examples/dr-duke-s-phytochemical-and-ethnobotanical-databases", "manifest": "agent/tools.json", "published_results_path": "results.json", "notes": "The published run is three calls to one unit, analysis.mc_chi2, and a Holm correction. The published calls reproduce every block of results.json that a recipe covers. The off-reference calls take the slices a referee asks for next, and they qualify claims 1 to 3. Leaving out CHEM_MEANS -- six columns the dictionary defines as a per-chemical, per-plant-part summary across species (species count, mean and standard deviation of concentration) -- takes every member of the pre-registered family above 0.05, before and after Holm, including R1, the one published Holm survivor (claim 2). CHEM_MEANS is the largest lever but not the only one: the primary's uncorrected p = 0.032 is also lost when YIELDS or FARMACY_NEW is left out instead, and R1 ends just above the Holm line without FARMACY_NEW. What holds in every family slice here is the direction claim 2 reports: dropping the ID key lowers the p-value, without CHEM_MEANS, YIELDS or FARMACY_NEW alike. With both numeric-heavy tables (CHEM_MEANS and YIELDS) out, the other 13 tables show no detectable association. claims.json v1 states claims 1 to 3 without these qualifications. Every literal was produced by calling the tool, at the published 20,000 replicates unless the call says otherwise.", "calls": [ { "call_id": "meta-published", "tool": "source_profile", "args": {}, "expect": { "kind": "published_pointer", "result_path": "/meta" }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "source-md5-published", "tool": "source_profile", "args": {}, "select": "/source_md5", "expect": { "kind": "published_pointer", "result_path": "/meta/source_md5" }, "recipe_id": "duke-phytochemical-source-md5", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "n-columns-published", "tool": "source_profile", "args": {}, "select": "/n_columns", "expect": { "kind": "published_pointer", "result_path": "/meta/n_columns" }, "recipe_id": "duke-phytochemical-columns", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "required-published", "tool": "degenerate_fields", "args": {}, "select": "/required/distinct_values", "expect": { "kind": "published_pointer", "result_path": "/degenerate_fields/required/distinct_values" }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "size-published", "tool": "degenerate_fields", "args": {}, "select": "/size/within_type_unique_sizes", "expect": { "kind": "published_pointer", "result_path": "/degenerate_fields/size/within_type_unique_sizes" }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "primary-contingency-published", "tool": "association_test", "args": {}, "select": "/contingency", "expect": { "kind": "published_pointer", "result_path": "/primary/contingency" }, "recipe_id": "duke-phytochemical-primary", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "primary-p-published", "tool": "association_test", "args": {}, "select": "/p_mc", "expect": { "kind": "published_pointer", "result_path": "/primary/p_mc" }, "recipe_id": "duke-phytochemical-primary", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "r1-drop-id-published", "tool": "association_test", "args": { "drop_columns": [ "ID" ], "seed": 20240609 }, "select": "/p_mc", "expect": { "kind": "published_pointer", "result_path": "/R1_drop_id/p_mc" }, "recipe_id": "duke-phytochemical-r1-drop-id", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "R1 is this tool with ID dropped, on the stream the published run gave it (seed + 2)." }, { "call_id": "r2-binary-text-published", "tool": "association_test", "args": { "type_grouping": "text_vs_nontext", "seed": 20240610 }, "select": "/p_mc", "expect": { "kind": "published_pointer", "result_path": "/R2_binary_text/p_mc" }, "recipe_id": "duke-phytochemical-r2-binary-text", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "R2 is this tool with type collapsed to Text vs Non-text, on the stream the published run gave it (seed + 3)." }, { "call_id": "structural-columns-dropped-off-reference", "tool": "association_test", "args": { "drop_columns": [ "ID", "CREATED" ] }, "expect": { "kind": "literal", "value": { "chi2": 38.51914215512988, "dof": 14, "p_mc": 0.00044997750112494374, "cramers_v": 0.46915816495463375, "cramers_v_bias_corrected": 0.3747757153669678, "n": 175, "n_tables": 15, "type_levels": [ "Double", "Short Text" ], "n_cells": 30, "cells_expected_lt5": 22, "contingency": { "ACTIVITIES": { "Double": 0, "Short Text": 6 }, "AGGREGAC": { "Double": 1, "Short Text": 7 }, "ASSAY": { "Double": 2, "Short Text": 3 }, "CHEMICALS": { "Double": 1, "Short Text": 6 }, "CHEM_MEANS": { "Double": 5, "Short Text": 0 }, "CODES": { "Double": 2, "Short Text": 3 }, "COMMON_NAMES": { "Double": 1, "Short Text": 4 }, "DOSAGE": { "Double": 0, "Short Text": 3 }, "ETHNOBOT": { "Double": 1, "Short Text": 17 }, "FARMACY_NEW": { "Double": 2, "Short Text": 22 }, "FNFTAX": { "Double": 3, "Short Text": 20 }, "PARTS": { "Double": 0, "Short Text": 4 }, "REFERENCES": { "Double": 0, "Short Text": 5 }, "SUPERACT": { "Double": 0, "Short Text": 2 }, "YIELDS": { "Double": 19, "Short Text": 36 } }, "design": { "type_grouping": "declared", "drop_columns": [ "CREATED", "ID" ], "exclude_tables": [], "mc_replicates": 20000, "seed": 20240607 } } }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "R1 removes ID, the only Long Integer column and present once in every table, because it is a structural constant. It keeps CREATED, the only Date With Time column, present once in 9 of the 15 tables: a record-creation timestamp rather than domain content. Removing both leaves a Short Text vs Double contrast over 175 columns: chi2 = 38.52 on 14 df, p_mc = 0.00045, Cramer's V = 0.47 (bias-corrected 0.37). This bears out the part of claim 2 that concerns the key: stripping structural columns sharpens the association rather than creating it. It says nothing about claim 2's Holm survival, and this sharper contrast leans on CHEM_MEANS as the family does: see the next call." }, { "call_id": "strict-contrast-chem-means-excluded-off-reference", "tool": "association_test", "args": { "drop_columns": [ "ID", "CREATED" ], "exclude_tables": [ "CHEM_MEANS" ] }, "expect": { "kind": "literal", "value": { "chi2": 21.08267214946976, "dof": 13, "p_mc": 0.06819659017049147, "cramers_v": 0.3521586553334913, "cramers_v_bias_corrected": 0.2176532906851906, "n": 170, "n_tables": 14, "type_levels": [ "Double", "Short Text" ], "n_cells": 28, "cells_expected_lt5": 21, "contingency": { "ACTIVITIES": { "Double": 0, "Short Text": 6 }, "AGGREGAC": { "Double": 1, "Short Text": 7 }, "ASSAY": { "Double": 2, "Short Text": 3 }, "CHEMICALS": { "Double": 1, "Short Text": 6 }, "CODES": { "Double": 2, "Short Text": 3 }, "COMMON_NAMES": { "Double": 1, "Short Text": 4 }, "DOSAGE": { "Double": 0, "Short Text": 3 }, "ETHNOBOT": { "Double": 1, "Short Text": 17 }, "FARMACY_NEW": { "Double": 2, "Short Text": 22 }, "FNFTAX": { "Double": 3, "Short Text": 20 }, "PARTS": { "Double": 0, "Short Text": 4 }, "REFERENCES": { "Double": 0, "Short Text": 5 }, "SUPERACT": { "Double": 0, "Short Text": 2 }, "YIELDS": { "Double": 19, "Short Text": 36 } }, "design": { "type_grouping": "declared", "drop_columns": [ "CREATED", "ID" ], "exclude_tables": [ "CHEM_MEANS" ], "mc_replicates": 20000, "seed": 20240607 } } }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "The Short Text vs Double contrast of the previous call, with CHEM_MEANS left out as well: chi2 = 21.08 on 13 df, p_mc = 0.068, bias-corrected V = 0.22, n = 170. The sharpest form of the association also falls short of 0.05 without CHEM_MEANS. It stays closer to the line than the primary does without CHEM_MEANS (p_mc = 0.221, next call), so what the other 14 tables hold is weak evidence, not none." }, { "call_id": "chem-means-excluded-off-reference", "tool": "association_test", "args": { "exclude_tables": [ "CHEM_MEANS" ] }, "expect": { "kind": "literal", "value": { "chi2": 45.500330447543504, "dof": 39, "p_mc": 0.22133893305334734, "cramers_v": 0.28032897786958894, "cramers_v_bias_corrected": 0.10511249906834956, "n": 193, "n_tables": 14, "type_levels": [ "Date With Time", "Double", "Long Integer", "Short Text" ], "n_cells": 56, "cells_expected_lt5": 46, "contingency": { "ACTIVITIES": { "Date With Time": 1, "Double": 0, "Long Integer": 1, "Short Text": 6 }, "AGGREGAC": { "Date With Time": 1, "Double": 1, "Long Integer": 1, "Short Text": 7 }, "ASSAY": { "Date With Time": 0, "Double": 2, "Long Integer": 1, "Short Text": 3 }, "CHEMICALS": { "Date With Time": 1, "Double": 1, "Long Integer": 1, "Short Text": 6 }, "CODES": { "Date With Time": 0, "Double": 2, "Long Integer": 1, "Short Text": 3 }, "COMMON_NAMES": { "Date With Time": 1, "Double": 1, "Long Integer": 1, "Short Text": 4 }, "DOSAGE": { "Date With Time": 0, "Double": 0, "Long Integer": 1, "Short Text": 3 }, "ETHNOBOT": { "Date With Time": 1, "Double": 1, "Long Integer": 1, "Short Text": 17 }, "FARMACY_NEW": { "Date With Time": 1, "Double": 2, "Long Integer": 1, "Short Text": 22 }, "FNFTAX": { "Date With Time": 1, "Double": 3, "Long Integer": 1, "Short Text": 20 }, "PARTS": { "Date With Time": 1, "Double": 0, "Long Integer": 1, "Short Text": 4 }, "REFERENCES": { "Date With Time": 1, "Double": 0, "Long Integer": 1, "Short Text": 5 }, "SUPERACT": { "Date With Time": 0, "Double": 0, "Long Integer": 1, "Short Text": 2 }, "YIELDS": { "Date With Time": 0, "Double": 19, "Long Integer": 1, "Short Text": 36 } }, "design": { "type_grouping": "declared", "drop_columns": [], "exclude_tables": [ "CHEM_MEANS" ], "mc_replicates": 20000, "seed": 20240607 } } }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "CHEM_MEANS is 6 of the 199 columns: ID plus five Double columns. The dictionary's own column definitions make it a per-chemical, per-plant-part summary across species: chem_id and part_id are keys into other tables, cnt is the number of plant species containing that chemical in that part, and avg and std are the mean and standard deviation of concentration (PPM). It has no Short Text and no CREATED column. Leaving it out moves chi2 from 62.47 on 42 df to 45.50 on 39 df, p_mc from 0.032 to 0.221, and the bias-corrected V from 0.19 to 0.11. It is the largest single lever on claim 1 but not the only one: family-yields-excluded-off-reference and family-farmacy-new-excluded-off-reference also take the primary's uncorrected p above 0.05 (0.087 and 0.053). Claim 1's uncorrected significance does not survive leaving out any one of these three tables; claims.json v1 does not say so." }, { "call_id": "chem-means-and-yields-excluded-off-reference", "tool": "association_test", "args": { "exclude_tables": [ "CHEM_MEANS", "YIELDS" ], "mc_replicates": 2000 }, "expect": { "kind": "literal", "value": { "chi2": 24.195793666140577, "dof": 36, "p_mc": 0.9515242378810594, "cramers_v": 0.24263252914614605, "cramers_v_bias_corrected": 0.0, "n": 137, "n_tables": 13, "type_levels": [ "Date With Time", "Double", "Long Integer", "Short Text" ], "n_cells": 52, "cells_expected_lt5": 44, "contingency": { "ACTIVITIES": { "Date With Time": 1, "Double": 0, "Long Integer": 1, "Short Text": 6 }, "AGGREGAC": { "Date With Time": 1, "Double": 1, "Long Integer": 1, "Short Text": 7 }, "ASSAY": { "Date With Time": 0, "Double": 2, "Long Integer": 1, "Short Text": 3 }, "CHEMICALS": { "Date With Time": 1, "Double": 1, "Long Integer": 1, "Short Text": 6 }, "CODES": { "Date With Time": 0, "Double": 2, "Long Integer": 1, "Short Text": 3 }, "COMMON_NAMES": { "Date With Time": 1, "Double": 1, "Long Integer": 1, "Short Text": 4 }, "DOSAGE": { "Date With Time": 0, "Double": 0, "Long Integer": 1, "Short Text": 3 }, "ETHNOBOT": { "Date With Time": 1, "Double": 1, "Long Integer": 1, "Short Text": 17 }, "FARMACY_NEW": { "Date With Time": 1, "Double": 2, "Long Integer": 1, "Short Text": 22 }, "FNFTAX": { "Date With Time": 1, "Double": 3, "Long Integer": 1, "Short Text": 20 }, "PARTS": { "Date With Time": 1, "Double": 0, "Long Integer": 1, "Short Text": 4 }, "REFERENCES": { "Date With Time": 1, "Double": 0, "Long Integer": 1, "Short Text": 5 }, "SUPERACT": { "Date With Time": 0, "Double": 0, "Long Integer": 1, "Short Text": 2 } }, "design": { "type_grouping": "declared", "drop_columns": [], "exclude_tables": [ "CHEM_MEANS", "YIELDS" ], "mc_replicates": 2000, "seed": 20240607 } } }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "With both numeric-heavy tables out -- CHEM_MEANS and YIELDS, 62 columns holding 24 of the 37 Double columns -- the other 13 tables show no detectable association: chi2 = 24.20 on 36 df, p_mc = 0.95, bias-corrected V = 0. A p this high means the observed table is more even than most permutations, and the one-per-table ID and CREATED columns supply much of that evenness: with both dropped the same 13 tables give p_mc = 0.43 (next call). Neither is evidence that type is independent of table in the rest of the schema; the slice was chosen after seeing where the residuals sit, so it locates the signal rather than testing the remainder afresh. Run at 2,000 replicates to keep the surface affordable: at p = 0.95 the Monte-Carlo standard error is about 0.005, so the published 20,000 would not change the reading." }, { "call_id": "strict-contrast-chem-means-and-yields-excluded-off-reference", "tool": "association_test", "args": { "drop_columns": [ "ID", "CREATED" ], "exclude_tables": [ "CHEM_MEANS", "YIELDS" ], "mc_replicates": 2000 }, "expect": { "kind": "literal", "value": { "chi2": 12.08619879575762, "dof": 12, "p_mc": 0.42828585707146427, "cramers_v": 0.3241872619839881, "cramers_v_bias_corrected": 0.0, "n": 115, "n_tables": 13, "type_levels": [ "Double", "Short Text" ], "n_cells": 26, "cells_expected_lt5": 20, "contingency": { "ACTIVITIES": { "Double": 0, "Short Text": 6 }, "AGGREGAC": { "Double": 1, "Short Text": 7 }, "ASSAY": { "Double": 2, "Short Text": 3 }, "CHEMICALS": { "Double": 1, "Short Text": 6 }, "CODES": { "Double": 2, "Short Text": 3 }, "COMMON_NAMES": { "Double": 1, "Short Text": 4 }, "DOSAGE": { "Double": 0, "Short Text": 3 }, "ETHNOBOT": { "Double": 1, "Short Text": 17 }, "FARMACY_NEW": { "Double": 2, "Short Text": 22 }, "FNFTAX": { "Double": 3, "Short Text": 20 }, "PARTS": { "Double": 0, "Short Text": 4 }, "REFERENCES": { "Double": 0, "Short Text": 5 }, "SUPERACT": { "Double": 0, "Short Text": 2 } }, "design": { "type_grouping": "declared", "drop_columns": [ "CREATED", "ID" ], "exclude_tables": [ "CHEM_MEANS", "YIELDS" ], "mc_replicates": 2000, "seed": 20240607 } } }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "The previous slice with ID and CREATED dropped too: a Short Text vs Double contrast over the 115 remaining columns of 13 tables, chi2 = 12.09 on 12 df, p_mc = 0.43, bias-corrected V = 0. Without the structural columns' evenness the p-value falls from 0.95 to 0.43, still far from significance. Run at 2,000 replicates: at p = 0.43 the Monte-Carlo standard error is about 0.011." }, { "call_id": "family-primary-published", "tool": "pre_registered_family", "args": {}, "select": "/primary", "expect": { "kind": "published_pointer", "result_path": "/primary" }, "recipe_id": "duke-phytochemical-primary", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "family-r1-published", "tool": "pre_registered_family", "args": {}, "select": "/R1_drop_id", "expect": { "kind": "published_pointer", "result_path": "/R1_drop_id" }, "recipe_id": "duke-phytochemical-r1-drop-id", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "family-r2-published", "tool": "pre_registered_family", "args": {}, "select": "/R2_binary_text", "expect": { "kind": "published_pointer", "result_path": "/R2_binary_text" }, "recipe_id": "duke-phytochemical-r2-binary-text", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "family-multiplicity-published", "tool": "pre_registered_family", "args": {}, "select": "/multiple_comparison", "expect": { "kind": "published_pointer", "result_path": "/multiple_comparison" }, "recipe_id": "duke-phytochemical-multiplicity", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "family-chem-means-excluded-off-reference", "tool": "pre_registered_family", "args": { "exclude_tables": [ "CHEM_MEANS" ] }, "select": "/multiple_comparison", "expect": { "kind": "literal", "value": { "method": "Holm-Bonferroni", "family_size": 3, "family": { "primary": { "p_mc": 0.22133893305334734, "p_holm": 0.46827658617069146 }, "R1_drop_id": { "p_mc": 0.15609219539023048, "p_holm": 0.46827658617069146 }, "R2_binary_text": { "p_mc": 0.7054647267636618, "p_holm": 0.7054647267636618 } } } }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "Without CHEM_MEANS no member of the pre-registered family is below 0.05, before or after Holm: raw p_mc 0.221 (primary), 0.156 (R1) and 0.705 (R2), corrected to 0.468, 0.468 and 0.705. R1 is the only published Holm survivor (Holm p = 0.033, claim 2), so claim 2's survival depends on these six columns. The other half of claim 2 holds in this slice too: dropping ID still lowers the p-value, from 0.221 to 0.156. claims.json v1 states claim 2 without this qualification; revising it would be a new version of the object, not an edit to this one." }, { "call_id": "family-yields-excluded-off-reference", "tool": "pre_registered_family", "args": { "exclude_tables": [ "YIELDS" ] }, "select": "/multiple_comparison", "expect": { "kind": "literal", "value": { "method": "Holm-Bonferroni", "family_size": 3, "family": { "primary": { "p_mc": 0.08679566021698915, "p_holm": 0.08679566021698915 }, "R1_drop_id": { "p_mc": 0.010599470026498675, "p_holm": 0.03179841007949602 }, "R2_binary_text": { "p_mc": 0.030848457577121145, "p_holm": 0.06169691515424229 } } } }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "Leaving out YIELDS instead: the table with the most columns (56) and the most Double columns (19). Raw p_mc 0.087 (primary), 0.011 (R1) and 0.031 (R2); Holm 0.087, 0.032 and 0.062. The primary loses its uncorrected significance, but R1 still survives Holm, so R1's survival does not depend on every numeric-heavy table: among the exclusions in this file it is CHEM_MEANS that removes it decisively." }, { "call_id": "family-farmacy-new-excluded-off-reference", "tool": "pre_registered_family", "args": { "exclude_tables": [ "FARMACY_NEW" ] }, "select": "/multiple_comparison", "expect": { "kind": "literal", "value": { "method": "Holm-Bonferroni", "family_size": 3, "family": { "primary": { "p_mc": 0.052747362631868404, "p_holm": 0.10549472526373681 }, "R1_drop_id": { "p_mc": 0.01784910754462277, "p_holm": 0.05354732263386831 }, "R2_binary_text": { "p_mc": 0.09669516524173791, "p_holm": 0.10549472526373681 } } } }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "Leaving out FARMACY_NEW, the table with the most Short Text columns after YIELDS (22 of its 26). Raw p_mc 0.053 (primary), 0.018 (R1) and 0.097 (R2); Holm 0.105, 0.054 and 0.105. Removing a table from the text-heavy side also leaves no member below 0.05 after correction. R1 lands at 0.054, about 1.3 Monte-Carlo standard errors above the line (3 x p_mc at p_mc = 0.018 and 20,000 replicates has a standard error of about 0.003), so this one is a boundary result rather than a clear loss. The association is a contrast between a few numeric-heavy and a few text-heavy tables, and its significance moves with either end." }, { "call_id": "holm-published", "tool": "holm", "args": {}, "select": "/1", "expect": { "kind": "published_pointer", "result_path": "/multiple_comparison/family/R1_drop_id/p_holm" }, "recipe_id": "duke-phytochemical-multiplicity", "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true }, { "call_id": "holm-two-member-family-off-reference", "tool": "holm", "args": { "p_values": [ 0.03249837508124594, 0.011049447527623619 ] }, "expect": { "kind": "literal", "value": [ 0.03249837508124594, 0.022098895055247238 ] }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "Both robustness checks re-test the primary's hypothesis on nearly the same columns -- R1 on 184 of its 199, R2 on all 199 with type coarsened -- so the three p-values are far from independent, and Holm, which holds under any dependence, is conservative here. Which members are counted decides the boundary reading in claim 1: over the primary and R1 alone, Holm gives 0.0325 and 0.0221, both below 0.05; over the primary and R2 alone it gives 0.065 for both (next call). The primary crosses 0.05 because R2 is in the family, not because the family has three members. The family of three is what was pre-registered and it governs the published claims; these two calls are a sensitivity reading, not a licence to pick a smaller family after seeing the p-values." }, { "call_id": "holm-primary-r2-family-off-reference", "tool": "holm", "args": { "p_values": [ 0.03249837508124594, 0.04269786510674466 ] }, "expect": { "kind": "literal", "value": [ 0.06499675016249187, 0.06499675016249187 ] }, "rel_tolerance": 1e-06, "abs_tolerance": 1e-09, "enabled": true, "notes": "The primary and R2 as a family of two, at their published p_mc: Holm gives 0.065 for both, the same boundary value the family of three gives the primary." } ] }