What data can software companies contribute to AI research?

Look across the software workflow

Software companies generate more than source code. A development task can leave a trail through a support request, an issue, a repository, a review conversation, test results and a release. When those records remain connected, they can help an AI team examine how a problem was understood, how a change was made and what evidence supports the result.

The most useful starting point is a complete example of work. Choose one task and ask which records let someone understand the problem without relying on the original developer's memory. Then identify what would be required to reproduce or check it.

Map the records to a possible research use

These record types can support different research questions. Assess the relationships and limitations alongside the files themselves.

A fictional case from report to release

Imagine a software team receives a report that exporting a large date range omits the final day's records. The support request links to issue APP-204. The issue names an affected version and provides a reproduction. A repository revision captures the code before the fix. Change request CR-217 links back to the issue and adds a regression test. A review asks for a timezone edge case. A later release record identifies the shipped version.

An assessment should preserve the links among these items and the order in which information appeared. The issue title is not enough to reconstruct the task. The final patch alone does not show what was known beforehand. A passing test without its environment may not be reproducible.

This example is fictional. It shows a useful record structure, not a claim about a customer, an accepted dataset or a demonstrated model capability.

A historical record and an evaluation task are different things

A connected history provides evidence from which a task may be designed. An evaluation also needs an instruction, a controlled starting state, available tools and a way to judge the result. Turning a case into a repeatable task requires checking those additional pieces.

SWE-bench provides a public example of this structure. Its original tasks connect GitHub issues and pull requests, prepare the repository at the base revision and use tests that fail before the change and pass afterward to help evaluate a proposed fix. That methodology illustrates why a repository snapshot and outcome checks matter; it does not establish the suitability of a private software archive.

Test results still need interpretation. A test can miss an important requirement, rely on an unstable external service or expose the intended solution. Describe those limits and review the task against the capability it is meant to measure.

Read how the original SWE-bench tasks are constructed

Treat agent logs as evidence of an attempt

If a company retains agent interaction logs, identify the run, the task, the available tools and the software state associated with the attempt. Record whether tool responses, errors and intermediate changes are present. Logs with missing steps may still be informative, but their limitations should be explicit.

A conversation ending with a success message is not sufficient evidence that the requested change happened. Check the relevant result, such as the changed repository, test outcome or application state. Anthropic's evaluation guide distinguishes the recorded interaction from the final outcome and recommends task-appropriate grading.

Keep the intended use clear. Historical human work, agent-generated attempts and synthetic examples have different provenance. A researcher should be able to tell which material they are examining.

Read Anthropic's distinction between transcripts and outcomes

Record the boundaries of the source

Start by identifying which repositories, organisations and time periods the source covers. A company may have its own product code alongside client-owned work, third-party packages and records governed by other agreements. Flag those distinctions for review before an export is shared.

Inspect for credentials, personal information, customer content and confidential details in code history as well as current files. Deleting a value from the latest version does not remove it from earlier revisions. An assessment should also consider attachments, build output and logs.

Describe what the archive does not retain. Some teams have source code but no issue history; others have issues and final patches but no usable test environment. Missing evidence changes the research use that can reasonably be considered.

What to include in a first description

A useful initial description explains one representative workflow and the broader material around it. It can be prepared without transferring private code.

• The product or engineering domain and the kinds of tasks recorded.
• The systems holding issues, code, reviews, tests, support and releases.
• Available history, approximate case counts and how those counts were measured.
• Identifiers linking a request to a change, test run and outcome.
• The availability of starting revisions and reproducible environments.
• Known gaps, exclusions and source-rights questions.
• How frequently new linked cases appear and how exports could be maintained.

Start with one understandable case

Trace one case from request to outcome before specifying a full transfer. That exercise reveals which joins work, which records are missing and how much context a reviewer needs. Keep a wider inventory alongside it so an unusually complete example is not mistaken for typical coverage.

Once the source structure is clear, the next discussion can focus on the intended capability, preparation effort and permitted use. The aim is to identify a package a research team can understand and assess, while keeping the source's limitations visible.

Discuss your records

Start with a general description of your systems and available history. Explore a data partnership before sharing business records.