How to Detect and Qualify Breaking API Changes Across Downstream Repositories

Key takeaways

• Code search overestimates migration impact. In the Inngest experiment, 941 raw matches ended at 2 repositories that were actually affected.
• Dependency state and package boundaries matter as much as source matches. Of 111 active repositories with old-interface code, 105 were intentionally on the older major version.
• Many apparent migration targets need no action because they are intentionally pinned, already migrated or already being fixed.
• In our first frozen held-out evaluation, Ziek classified 28 of 30 repositories correctly. The final 48 of 48 includes documented fixes and label corrections, so the first result is the stronger measure.
• A live Ray contribution shows the opposite case: a repository where migration was genuinely warranted and could be verified across multiple Azure SDK versions. The pull request is open and under review.
• The emerging opportunity is not automatic pull request generation. It is downstream change intelligence plus verified remediation.

How does Ziek differ from basic code and dependency automation?

What is software change propagation?

Software change propagation is the process of determining how an upstream API, SDK, package or dependency change affects downstream codebases, and what action should be taken in each one.

Ziek approaches it as a sequence of five questions. Each can fail on its own, and each needs a different kind of evidence.

The first step is a precise description of the change: the package, the old interface, the replacement and the version boundary. Step two is search, and it is the easy part. Steps three and four are where the experiments below found most of the difficulty, and most of the value. Step five, producing a patch and proving it works, matters only for the repositories that survive the first four.

Why is code search not enough for API migrations?

Code search finds text. A breaking API change is a statement about executable behavior in a specific dependency context. Between the two sits a set of cases that look identical in a list of search results and call for different actions:

• Documentation, plans and notes. Markdown files, agent skill files, handoff notes and data corpora contain the old API name without using it.
• Comments and commented-out code. A comment explaining that the old API was removed matches the old API name.
• Generated and vendored copies. Snapshot files and committed copies of third-party code contain the old interface but are not maintained by the repository.
• Intentional old-major pins. A repository can depend on the older major version on purpose, often because the vendor still supports it.
• Monorepo package boundaries. A match in one package says nothing about the dependency declared by a different package in the same repository.
• Lockfile and version indirection. The declared range, a shared workspace version catalog and the resolved lockfile version can disagree.
• Already migrated code. Repositories that completed the migration often keep a comment or an old file that still matches.
• Migrations in progress. Some repositories are partway through, with branches or pull requests already open.

Search tooling also limits what a count means. GitHub's legacy code search returns at most 1,000 results per query. In the Inngest experiment, one query reported 9,968 total matches while 1,000 were retrievable, so any figure derived from that query is a floor, not a count. Only the main old-interface query (941 matches) was fully enumerated.

What did the Inngest migration experiment show?

What we observed

The Inngest TypeScript SDK 4.0.0 was released on 2026-03-16. It removed the EventSchemas export that v3 applications use to type their events, and it changed the createFunction signature: v3 takes options, a trigger and a handler, while v4 takes options that contain the triggers, and a handler. Both changes are visible in the published type definitions (EventSchemas appears in the 3.54.2 package and not in 4.0.0 or 4.21.1), and the npm registry lists a v3-lts tag that keeps v3 available, so only repositories that moved to v4 are broken.

A broad search for the old new EventSchemas interface showed what looked like a large migration population. Ziek then ran a read-only census of those public GitHub repositories and analyzed the survivors in depth:

The population collapsed for ordinary reasons. Most repositories were intentionally pinned to the older major version. In the manual audit, roughly half of the 210 active matches were documentation, plans, skill files or data rather than executable code. Among repositories already on v4, 4 of the 12 matches were comments or commented-out lines. Our benchmark also included monorepos where a flagged dependency belonged to a different package than the code that matched, and repositories where migration work was already underway. A later analysis found that the "last pushed" date used to define "active" is itself a weak signal, because bots and automation move it without a human commit.

The event did not clear the bar we set before collecting data, which required at least three repositories actually affected on their current default branch. It found two.

The important result was not that Ziek found more migration targets. It was that it eliminated most of them.

How do you determine whether a repository is actually affected?

Ziek's impact-analysis pipeline takes one candidate repository and one upstream software change and produces a structured record with the evidence behind its conclusion. It checks:

• The current default branch. Re-fetch it and confirm the old interface still exists there, instead of trusting a search index.
• Dependency resolution. Read the declared version and the lockfile version, including shared workspace catalogs, and classify the result: pinned to the old major, a compatible old-major range, a range that can reach the new major, already on the new major, ambiguous, or absent.
• Package boundaries. In a monorepo, tie each match to the manifest of the package that owns the file.
• Executable code versus everything else. Classify each match as code, configuration, test, documentation, comment, plan or notes, generated file, vendored copy or example.
• Migration state. Detect code that is already migrated or partly migrated.
• Existing fixes. Look for open pull requests, migration branches, broader rewrites and issues covering the same change.
• Activity and licence. Use human commits rather than the last-push date, and record the licence.
• How a fix could be verified. Determine whether tests, a type check, a build or an install can verify it, or whether verification would need credentials or a live service.

Each repository then lands in one category:

When should a migration automatically become a pull request?

A migration should become a pull request only when the repository is affected on its current default branch and sending a fix is also warranted: the repository is active and independently maintained, nobody is already fixing it, its licence and contribution norms allow outside changes, and the fix can be verified. Being affected is not enough.

That makes "should anything be sent?" a separate decision from "is this repository affected?", with three outcomes: migration warranted, no action needed and requires engineering review. Ziek's design rules are that this decision is made on its own rather than copied from the impact result, that the default is to send nothing, and that unresolved ambiguity goes to an engineer instead of being acted on.

How accurate was the Ziek downstream-impact benchmark?

What we observed

Ziek built a benchmark of 48 Inngest repositories with hand-audited labels: 18 used to develop the pipeline and 30 held out. The pipeline was frozen before the held-out run.

The first frozen held-out result is the stronger generalization measure: 28 of 30. The final 48 of 48 result includes documented post-test improvements. Both misses in the frozen run were corrected afterward, and the corrections are recorded:

• One was a real pipeline gap: a truncated repository tree hid the manifest that owned the code. It was fixed after the run and covered by a test.
• One was an error in the human label: the manual audit had missed orphan workflow files. The pipeline was right to escalate, and the label was corrected.
• On the development set, three pipeline issues were fixed and three labels were revised after seeing output.

Limits that matter more than the headline score:

• The benchmark contains no clean case where migration was warranted. 36 of 48 repositories are intentional v3 pins. Recommending nothing unnecessarily is a necessary but weak result, because a system that never recommends anything also scores zero. The "migration warranted" path was exercised only by synthetic unit tests.
• The same person wrote the pipeline and the audit, the audit was shallow, and the held-out set is about 97% one outcome.
• Finding the repositories (step two) was not tested. They came from the census.
• Two repositories out of 48 (4.2%) still needed an engineer to look.

When is migration work actually warranted?

What we observed

Having built a system that is good at suppressing false positives, we deliberately went looking for the positive cases: repositories that were genuinely affected, active, licensed, independently maintained, not already migrating and locally verifiable. Those cases were much rarer than expected.

The test was written before any downstream data was collected and was not lowered afterward. An upstream change passed only if, on current default branches, there were at least 8 independent active repositories on the old interface, at least 3 genuinely incompatible, at least 2 plausible candidates for a migration, at least 1 with repository-native verification, and at least 1 that was active, licensed, independently maintained, not migrating and not pinned.

A sweep of 167 popular npm packages found 31 with a new major version released on or after 2026-04-09. Three were prescreened with bounded samples of repositories that had already declared the new major version:

All three were bounded, search-ordered samples with hint-limited file scans, so they are lower bounds, not census counts.

Finding: maintained repositories that intentionally adopt a new major version usually migrate while doing so. Repositories still broken on the new major skewed toward unlicensed, dormant or otherwise low-quality targets. This is a finding about these samples, not a universal claim.

For an SDK provider, this matters directly: sending indiscriminate fixes to every repository that matches a search would mostly create noise.

What about changes consumers do not explicitly opt into?

What we observed

The next hypothesis was that changes which reach a repository without a deliberate upgrade would leave more genuine breakage. Examples include package restructuring, broad or unpinned dependency resolution, transitive changes, provider-side deprecations and security-forced upgrades. Ziek evaluated six such events:

setuptools 82 was the nearest miss and the best example. The sample was 340 repositories with an open issue mentioning No module named 'pkg_resources' created after the release, of which 30 independent, active repositories were inspected. Six still imported pkg_resources on their default branch and 24 did not, because maintained repositories mostly fixed it within weeks. Of the six, four were genuinely affected and had not chosen the change, one was mid-migration with a pull request open, and one guarded the import with a fallback. One affected repository, Graveyard-Keeper-Savefile-Editor, was a clear case for a fix, confirmed with a deterministic local reproduction (it has no native tests). Three others needed engineering review: one has CI that runs only on self-hosted GPU hardware, one maintainer chose to document a setuptools pin, and one is a one-star script with no tests.

Conclusion: changes consumers do not opt into produced a higher density of genuine impact than ordinary major-version migrations, but still not a large, clean remediation population. Because the setuptools sample was drawn from open issues, it selects breakage that maintainers can already see.

A case where migration was warranted: Ray and the Azure SDK split

What we observed

The upstream change: azure-mgmt-resource 25.0.0 moved the Deployments operation group out of ResourceManagementClient into a separate package, azure-mgmt-resource-deployments, exposed as DeploymentsMgmtClient. Version 26.0.0 also stopped exporting ResourceManagementClient from the top-level azure.mgmt.resource module. The break begins at 25.0.0, not 26.

Ray's Azure autoscaler still relied on the legacy interface. In an earlier ten-repository benchmark, Ziek classified it as a genuine migration need: the relevant tests passed on the pinned 24.x release and failed once only azure-mgmt-resource was upgraded, with the failure raised from Ray's own checked-out source. The migration then had to meet a real constraint: support modern SDKs without breaking existing 24.x environments. The compatibility path that was implemented:

• The legacy 24.x path remains supported.
• The modern split client is used when it is installed.
• If neither interface is available, the code raises an error that names the package to install.

The reported test matrix covers azure-mgmt-resource 24.0.0 without the deployments package (36 passed), 24.0.0 with azure-mgmt-resource-deployments 2.0.0 (36 passed), 25.0.0 with 2.0.0 (36 passed), and 26.0.0 with 2.0.0 (36 passed). The change was also checked with the targeted tests, the relevant pre-commit hooks, git diff --check and Ray's raydepsets --check, which validates the regenerated dependency lock files.

What was not done: there was no live Azure deployment and no full Ray Bazel or CI run locally. The pull request is ray-project/ray#66714. As of October 8, 2026 it is open with no approving review.

Why Ray differs from the suppressed cases. It had an actually incompatible interface, not a comment or a pin. It is a large, actively maintained repository. The migration path was concrete. Compatibility could be verified across SDK versions. And no equivalent active migration was found. That combination is what the other repositories in these experiments mostly lacked, and it is why downstream impact analysis matters: without it, Ray would have been one match among hundreds.

What does the Ray review tell us?

What we observed

Ray's automation labeled the pull request and a Ray maintainer assigned themselves. That maintainer then wrote to two code owners: "Hi @marosset and @alimaazamat, do you have more context on those SDKs? Could you help take a look?" An automated reviewer, Gemini Code Assist, suggested creating one AzureCliCredential and reusing it for both clients. The suggestion is valid: the change added a second credential instantiation.

The review also illustrates an important part of this problem: external SDK changes often require domain-specific context. In this case, a Ray maintainer asked contributors with Azure SDK context to review the migration.

Hypothesis

One hypothesis behind Ziek is that packaging the upstream change, downstream impact, compatibility constraints and verification evidence together can make that context easier to evaluate. We have not yet measured whether it reduces review time, and one review is not evidence for it.

Why this matters beyond individual pull requests

What we think this could become

Today, an upstream provider ships release notes, a migration guide and a changelog. Then hundreds or thousands of downstream teams independently work out whether they are affected, where they are affected, whether to migrate, how to migrate and whether the result is correct. The same investigation is repeated in every repository, and the provider mostly sees the outcome as support tickets and migration lag.

The Ziek thesis is that software providers should be able to distribute not just documentation about a change, but structured downstream impact and verified remediation. Depending on what the evidence supports for each repository, the output could be:

• an impact report,
• migration guidance,
• a task for a coding agent,
• a verified patch,
• a pull request, or
• an explicit "no action needed".

This is a product thesis, not a claim of current product maturity. The evidence so far is the set of experiments in this article.

What would this look like for an SDK or API provider?

Target workflow, not a deployed product

1. The provider ships an SDK or API change.
2. Ziek identifies downstream repositories that are potentially exposed.
3. Ziek determines which of them are actually affected.
4. Consumers that already migrated, are intentionally pinned or are unaffected are filtered out.
5. For affected consumers, Ziek produces evidence explaining why the change matters to them.
6. Where remediation is appropriate, Ziek prepares and verifies the migration.
7. The provider sees a downstream impact map, instead of waiting for fragmented support tickets and migration lag.

This is the workflow Ziek is built toward. It has not been demonstrated at enterprise scale, and the experiments above cover individual repositories and bounded samples.

Why this could become infrastructure

What we think this could become

The durable problem is not one SDK migration. The recurring primitive is: upstream software change, then downstream impact, then action. Changes of this kind occur continuously across SDKs, APIs, cloud services, libraries, models, security policies and dependency graphs. In the events we examined, the same pattern appeared in both the npm and PyPI ecosystems: many apparent targets, few genuine ones.

The potential infrastructure layer is the system that maps those changes to the codebases that depend on them. If that is right, what could become hard to replicate is accumulated evidence about how real repositories declare dependencies, pin versions, migrate and verify, not the ability to generate a diff. That is a hypothesis. It has not been validated with providers or customers.

How is Ziek different from dependency update tooling?

Dependency update tools are excellent at managing declared package-version changes. Ziek is exploring a different layer: interpreting what an upstream software change means for each downstream codebase and deciding whether remediation is warranted.

That difference shows up in cases where a version bump is the wrong unit:

• Exposure without a declaration change. A repository can be broken by an unpinned or undeclared dependency, such as setuptools 82 removing pkg_resources, with no edit to its own manifest.
• Documentation-only references. A match in documentation is not a breakage.
• Intentional pins. 105 of the 111 active Inngest repositories with old-interface code were on v3 on purpose.
• Already-migrated code. The old name survives in comments after the migration.
• No-action decisions. An explicit "no action needed, with the reason" is a result, not a failure.

Ziek is not a replacement for these tools, and this article makes no claim about how any specific tool works internally.

What is Ziek?

Ziek helps software providers understand how upstream API, SDK and dependency changes propagate into downstream codebases. It identifies which repositories are actually affected, distinguishes real migration work from false positives, and produces verified remediation when action is warranted. Put briefly, Ziek is infrastructure for understanding and distributing software changes across dependent codebases.

Ziek is an early, experimental system. It is not production-mature. It is being validated through public engineering experiments, benchmarks and upstream contributions. See the Ziek homepage and the founder page.

What remains unproven?

• Finding every downstream repository remains imperfect. GitHub search caps and rate limits affected every experiment above.
• Coverage is event-specific. The Inngest detector covers two break modes, so how often it misses other break modes is unmeasured.
• The rules that decide when remediation is warranted are partly hand-designed.
• Clean cases where migration is warranted remain sparse, and the benchmark contains none.
• Whether API and SDK providers want downstream-impact intelligence is unvalidated.
• There is no customer or pilot evidence yet.
• There is no proof yet that Ziek consistently detects breakage before maintainers react.
• The Ray pull request is still under review and may change or be closed.

What are we testing next?

The next research question is whether Ziek can identify downstream exposure before maintainers publicly react. The metric is the lead time between the earliest technically detectable exposure and the first public maintainer reaction. There are no results to report yet.

Bottom line

Finding references to an old API is easy. The difficult part is determining which downstream repositories are actually affected, which have intentionally stayed on an old version, which have already migrated, whether remediation should be sent at all, and how to verify it safely. In the experiments above, the number of repositories that needed a fix was a small fraction of the number that matched a search. A system that cannot say "no action needed" will open the wrong pull requests.

Frequently asked questions

What is software change propagation?

Software change propagation is the process of determining how an upstream API, SDK, package or dependency change affects downstream codebases, and what action should be taken in each one. It covers finding potentially affected repositories, determining which are actually affected, deciding whether any remediation should be sent, and producing and verifying the migration.

How do you detect which repositories are affected by a breaking API change?

Start from a precise description of the change: the package, the old interface, the replacement and the version boundary. Search for the old interface to find candidates, then check each candidate's current default branch by resolving the dependency version, separating executable code from documentation and comments, and checking for intentional pins and migrations already underway.

Why isn't code search enough?

Code search matches text, not executable use, and GitHub caps results at 1,000 per query. In the Inngest experiment, 941 matches reduced to 2 repositories that were actually affected, because most matches were pins, documentation, comments, vendored copies or already-migrated code.

When should a migration automatically become a pull request?

Only when the repository is affected on its current default branch and sending a fix is warranted: the repository is active, licensed, independently maintained, not already migrating, not intentionally pinned, and the fix can be verified. Affected repositories that cannot be verified independently should go to engineering review, and the rest need no action.

How is Ziek different from dependency update tooling?

Dependency update tools are excellent at managing declared package-version changes. Ziek is exploring a different layer: interpreting what an upstream software change means for each downstream codebase and deciding whether remediation is warranted.

What does Ziek do?

Ziek analyzes how upstream API, SDK and dependency changes affect downstream codebases. It identifies which repositories are genuinely affected, filters cases where no action is needed, and helps produce verified remediation where migration is warranted.

Who is Ziek for?

Ziek is aimed at SDK and API providers, developer platform teams, and engineering organizations that manage large dependency surfaces. It is early, and there are no customers or pilots to report yet.

Is Ziek production-ready?

No. Ziek is currently being validated through public engineering experiments, benchmarks and upstream contributions.

Sources

• ray-project/ray pull request #66714, status checked October 8, 2026.
• inngest-js, the Inngest TypeScript SDK repository, and the inngest package metadata on the npm registry. Release dates, the v3-lts tag and the published type definitions were checked against the registry packages.
• azure-mgmt-resource on PyPI and azure-mgmt-resource-deployments on PyPI. The 25.0.0 breaking-change notes are in the package changelog.
• setuptools 82.0.0 on PyPI, released 2026-02-08. The 81.0.0 package ships pkg_resources and the 82.0.0 package does not, checked by inspecting both published wheels.
• Ziek census, downstream-impact benchmark and remediation-opportunity experiment reports (internal working artifacts, dated 2026-10-06).

Observed counts come from those artifacts. Statements labeled hypothesis or "what we think this could become" are not results.