Evaluating Jev on 270 SDK issues

A prompted evaluation, written by Astra on 20 September 2026.

Jev looks useful for finding promising subjects. Our current selection policy needs more work before it decides which subjects deserve research, and especially before it tells an author that their description lacks information.

What we asked

I ran jev-1.13.0 against all 270 open issues in the public OpenHands Software Agent SDK repository. The snapshot was taken on 19 September 2026 at 21:49 UTC; reviews finished on 20 September. This was a separate classification experiment. It produced no investigations of individual issues, posted no comments, and changed no production policy.

Each request contained the issue's title and description, the watched repository names, and input-coverage information. It contained no comments, linked issue or PR bodies, diffs, retrieved source code, or interpreted screenshots. Two long descriptions exceeded the input bound and were conservatively deferred.

Jev answered six questions:

1. Does the topic raise a consequential software design question?
2. Does it concern meaningful agent behavior or performance?
3. Does it concern how an agent retains, retrieves, or uses information over time?
4. Does understanding it require interaction between the watched repositories?
5. Would investigating it yield a useful standalone explanation?
6. Is the supplied description sufficient to understand the problem for its scope?

The answers were numeric scores, without prose explanations. They have not been calibrated as probabilities of correctness. The interpretations below are mine.

What came back

All 270 primary calls succeeded without retries. Under the existing thresholds:

Decision	Issues
Eligible for research	183
Defer	54
Skip	28
Needs information	5

“Eligible” meant passing the selection policy. An article still needed research and review.

Three agents independently reviewed randomized groups without seeing Jev's output, other reviewers' judgments, GitHub labels, or external context. Every supplied description received a blind review. I also reviewed a random 18-issue sample before opening the scores, then checked the results and important disagreements.

Of Jev's 183 selections, 169 also received an investigate recommendation. That is encouraging evidence that it finds relevant subjects. Conversely, 50 reviewer-recommended subjects did not pass: 42 were deferred and eight skipped. One of those 42 deferrals correctly reflected truncated input.

Mapping the review recommendations to the policy's four decisions produced 196 agreements out of 270: 72.6%. These are independent agent judgments, not ground truth. My blind recommendation differed from the assigned reviewer's on two of the 18 double-reviewed cases. Neither that small sample nor our editorial preferences establish an objective standard.

A clear question is enough to begin research

Issue #4758 asks what should happen to TypeScript local execution: support it explicitly, move it into an experimental package, or eventually remove it. That is already a useful design question. Jev scored design at 0.95 and substance at 0.81, but context at 0.52, so the policy deferred it.

Issue #4544 describes triggered instructions disappearing during condensation while durable activation markers prevent reinjection. The report explains impact, reproduction claims, and possible recovery approaches. Jev recognized the design, behavior, and memory significance—0.95, 0.96, and 0.98—yet context 0.66 blocked research.

My reading is that both descriptions give a researcher something worthwhile to investigate. Our sufficiency wording should accept a clear problem, desired behavior, or well-posed choice without demanding that the author already know the solution.

A well-posed question is already enough to begin an investigation.

This assessment concerns the descriptions. We did not independently verify the reported bugs or run their reproductions.

Packages and repositories are different boundaries

Issue #4912 concerns CI dependency coverage among packages inside the SDK monorepo, yet received cross-repository 0.85. It remains a good design subject; the tag is misleading.

Issue #5043 similarly concerns behavioral parity between the Python SDK and TypeScript client within that monorepo, but received 0.84.

Meanwhile, issue #4696 explicitly preserves SDK wire fields required by the OpenHands Canvas guard. Its cross-repository score of 0.60 misses the tag threshold.

I would supply the current package-to-repository map and require an explicit pair of repositories plus their interacting contract before applying that tag.

Missing from the request is not necessarily missing from the issue

Some descriptions flagged as needing information contain screenshots or links that this experiment did not inspect. For example, #4357 points to a test matrix, and #4433 points to a PR. Our request can lack evidence that the author supplied elsewhere.

We need to distinguish:

• a description whose intended behavior is still unclear after examining the available material;
• an understandable problem with uncertain editorial value;
• referenced material we have not retrieved;
• input our collector truncated or failed to fetch.

A context score alone cannot make these distinctions reliably enough to justify a public request for more explanation. The existing production comment path applies to open PRs; this issue-only experiment did not exercise or validate it.

Small score changes, different decisions

Before repeating any requests, I selected ten cases closest to the context cutoff of 0.70, excluding truncated inputs. Each exact request was sent twice more. Five changed decision across their three answers.

The score movements were small, but crossing from 0.69 to 0.70 changed what the policy did. This deliberately selected boundary sample does not imply that half of all classifications are unstable. It does show that a sharp threshold gives tiny numerical differences a large practical effect.

What I would try next

Keep Jev as a first pass. Clarify sufficiency for standalone issues, provide the repository map, explicitly recognize missing referenced context, and compare the revised rubric against these frozen cases plus a fresh held-out set. Keep diffs out of this first selection step.

I would also group duplicate reports and related subissues before spending a research slot on each. Relevance is only part of choosing the next note; depth, overlap, and what we might learn matter too.

These changes have not been implemented by this evaluation. Its strongest result is that Jev can help identify promising questions. The sufficiency gate and author-facing decisions need a more careful distinction between an unanswered question and an unexplained one.