-
Notifications
You must be signed in to change notification settings - Fork 96
Comparing changes
Open a pull request
base repository: datajoint/datajoint-python
base: master
head repository: datajoint/datajoint-python
compare: feat/1424-self-upstream
- 8 commits
- 9 files changed
- 2 contributors
Commits on Jun 10, 2026
-
fix(#1429): cascade through FK chain for part_integrity="cascade"
Diagram.cascade(part_integrity="cascade") used to derive each Part's Master restriction by joining `master_ft.proj() & child_ft.proj()` on shared attribute names. This failed when the Part referenced its Master indirectly — through another Part with renamed FK columns (`.proj()` in the definition) or via a Part-of-Part chain that does not directly inherit Master's PK names. The intermediate Parts' restrictions were also skipped. Replace the proj-join shortcut with an upward walk of the actual FK graph from the Part to its Master, applying symmetric (upward) counterparts of the existing propagation rules at each edge. Key changes in src/datajoint/diagram.py: - New Diagram._apply_propagation_rule_upward — mirror of the existing forward propagation method. Same three rules (shared-PK copy, aliased reverse-rename, non-aliased projection) applied in the reverse direction (child → parent). - New Diagram._propagate_part_to_master — walks nx.shortest_path (Master → Part) and applies the upward rules along each real edge, transparently skipping the integer-named alias nodes that the graph inserts for aliased FKs. Restricts intermediate Parts too (the chain case from #1429 Case 2). Materializes the Master's restriction via to_arrays() so the subsequent forward cascade back down to Master's other Parts produces literal `WHERE ... IN (values)` clauses rather than self-referential subqueries (avoiding MySQL error 1093). - New Diagram._find_real_edge_props — looks up edge props for parent → child via the direct edge OR through an alias node. - _propagate_restrictions: seed-is-Part case. When the cascade starts at a Part (e.g. `Master.PartB.delete(part_integrity="cascade")`), the main loop's part_integrity block — nested inside the out_edges iteration — cannot fire because a leaf Part has no out-edges. Trigger the upward propagation explicitly for the seed before the main loop. - Diagram.cascade: expand nodes_to_show to include any node that the part_integrity propagation pulled in (the master and its descendants), so counts() and __iter__ report the full cascade subgraph. Tests in tests/integration/test_cascade_delete.py — three new mysql tests covering both #1429 cases plus an end-to-end delete. Full regression: 8 + 15 + 33 mysql tests pass. Slated for DataJoint 2.3.
Configuration menu - View commit details
-
Copy full SHA for bbbbadf - Browse repository at this point
Copy the full SHA bbbbadfView commit details -
fix(#1429): address MilagrosMarin review on #1468
- Drop unused `propagated_edges` parameter from `_propagate_part_to_master` and its call sites. The parameter was vestigial after the design switched from edge-blocking to materialization at the master. - Document two limitations in the docstring: - Single FK path: nx.shortest_path returns one path; non-shortest paths are not applied. - Memory cost of materialization: to_arrays() pulls matching master PKs into Python memory. - Add test_cascade_three_level_part_chain covering PartC → PartB → PartA → Master. Confirms intermediate Parts are restricted at every hop, not just the first. All 36 mysql tests in cascade_delete + cascading_delete + dependencies + semantic_matching pass.Configuration menu - View commit details
-
Copy full SHA for 9094a64 - Browse repository at this point
Copy the full SHA 9094a64View commit details -
Configuration menu - View commit details
-
Copy full SHA for df8d73d - Browse repository at this point
Copy the full SHA df8d73dView commit details
Commits on Jun 23, 2026
-
feat(#1423): Diagram.trace() for upstream restriction propagation
Implements T2.2.a of the provenance trinity (datajoint-docs#183). Upstream mirror of Diagram.cascade(): walks the FK graph from a restricted seed to every ancestor with OR convergence — an ancestor entity is included if reachable through any FK path from the seed. Reuses the upward propagation primitives added by #1468 (_apply_propagation_rule_upward / _find_real_edge_props) applied here in a generalized form (any child → any parent, not just Part → Master). Branch note: stacked on fix/1429-cascade-part-part-renamed-fk (#1468) for the upward primitives. Will rebase onto master after #1468 lands. What's added: - src/datajoint/diagram.py: - New @classmethod Diagram.trace(table_expr) — mirror of cascade(), walks ancestors instead of descendants, trims to ancestor subgraph. - New _propagate_restrictions_upstream(start_node) — multi-pass walk over in_edges, applies the upward rules at each real edge. Alias-node transparent. - New __getitem__(key) — supports both Table subclass/instance (returns pre-restricted QueryExpression) and string (returns pre-restricted FreeTable). Raises DataJointError for tables outside the trace's subgraph. - Bugfix in _apply_propagation_rule_upward Backward Rule 3: previous code projected child to its OWN PK (child_ft.proj()) which excluded non-primary FK columns. Now projects to the FK columns via proj(*attr_map.keys()), correctly carrying them into the parent restriction for non-primary-FK cases. Caught by test_trace_or_convergence_two_paths. - src/datajoint/dependencies.py: - New load_all_upstream() — symmetric to load_all_downstream. Iteratively discovers upstream schemas reachable via reverse FK edges, expanding the graph until convergence. - src/datajoint/adapters/{base,mysql,postgres}.py: - New find_upstream_schemas_sql(schemas_list) on each adapter, symmetric to find_downstream_schemas_sql. - tests/integration/test_trace.py (new, 8 tests covering single-hop, multi-hop, renamed FK, OR convergence across two paths, non-ancestor rejection, string indexing → FreeTable, counts(), leaf-table seed). All 8 trace tests pass on MySQL. Regression: test_cascade_delete + test_cascading_delete + test_dependencies + test_semantic_matching — 36 tests pass, no regressions from the Rule 3 fix. Slated for DataJoint 2.3.
Configuration menu - View commit details
-
Copy full SHA for 8d9d242 - Browse repository at this point
Copy the full SHA 8d9d242View commit details -
fix(#1423): gate Diagram.__getitem__ on _mode == "trace"
The trace-mode __getitem__ I added shadowed networkx.DiGraph's standard adjacency-dict lookup for ALL Diagrams, not just trace results. ERD tests (and any other code that does diagram[node_name] for adjacency) were getting DataJointError("not in this trace's subgraph") instead of the adjacency dict. Fix: short-circuit non-trace diagrams (no _mode attribute or _mode != "trace") to super().__getitem__(key) before any trace-specific logic runs. Tests: - 5 previously-failing erd tests now pass (test_erd, test_diagram_algebra, test_repr_svg, test_make_image, test_part_table_parsing). - 8/8 trace tests still pass.Configuration menu - View commit details
-
Copy full SHA for 03dd0a7 - Browse repository at this point
Copy the full SHA 03dd0a7View commit details -
feat(#1424): self.upstream property for pre-restricted ancestor access
Implements T2.2.b of the provenance trinity. Inside make(), self.upstream exposes a pre-constructed Diagram.trace(self & key) so users can read declared ancestors with provenance-safe, ergonomic syntax. Branch stacked on feat/1423-diagram-trace (#1471) for Diagram.trace(). What's added: - src/datajoint/autopopulate.py: - AutoPopulate._upstream class attribute (default None) — instance storage for the per-make() trace. - AutoPopulate.upstream property — returns the trace if set, raises DataJointError with a clear "only available inside make()" message otherwise. The error includes the fallback pattern (dj.Diagram.trace(self & key)) so the user knows the escape hatch. - In _populate_one, set self._upstream = Diagram.trace(self & dict(key)) immediately before the make() invocation block. Construction is lazy at this layer (graph copy only); the SQL fetch fires when the user accesses self.upstream[T].fetch(...). - The existing `finally` block (line 716) that resets _allow_insert now also resets _upstream to None, so subsequent attribute access raises a clear error rather than silently returning a stale trace from the previous make() call. What's not changed: - make() signature: unchanged — key remains a dict, make_kwargs work as before. self.upstream is a new attribute on self, not a new parameter. - Tripartite make pattern: self._upstream is set once before all three make() invocations, so all three phases see the same upstream view. Tests in tests/integration/test_autopopulate.py (5 new): - test_upstream_provides_pre_restricted_ancestor — basic case: make() reads self.upstream[Subject].fetch1("name") and the value is correctly pre-restricted to the current key. - test_upstream_rejects_non_ancestor — self.upstream[Unrelated] raises DataJointError ("not in this trace"). Inherited from Diagram.__getitem__. - test_upstream_unset_outside_make — accessing the property outside of make() raises with the helpful "only available inside make()" message. - test_upstream_cleared_after_make — after populate() completes, accessing the property on a fresh instance still raises (verifies the finally cleanup; no stale state). - test_upstream_seen_across_tripartite_make — both make_fetch / make_compute / make_insert see the same self.upstream value. Full regression: 17/17 autopopulate tests pass on MySQL. Slated for DataJoint 2.3. Blocked on #1471 (Diagram.trace) merging; T2.2.c (strict_provenance) is stacked on this PR.
Configuration menu - View commit details
-
Copy full SHA for 7e4130f - Browse repository at this point
Copy the full SHA 7e4130fView commit details
Commits on Jul 1, 2026
-
Configuration menu - View commit details
-
Copy full SHA for 52ff2ea - Browse repository at this point
Copy the full SHA 52ff2eaView commit details -
Configuration menu - View commit details
-
Copy full SHA for b61abf2 - Browse repository at this point
Copy the full SHA b61abf2View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff master...feat/1424-self-upstream