EvalPort Adds External Adapter for safelabs-eval After Reviewing Its Public Evaluation Models

Open-source AI evaluation tools become more useful when test cases and results can move between ecosystems. A recent integration around safelabs-eval provides a concrete example: an independent maintainer of EvalPort reviewed the project’s public data structures, proposed an adapter, implemented it outside the Safe Labs AI repository, and merged the work into EvalPort.

The interaction began publicly in GitHub Issue #1. The EvalPort maintainer, posting as adhabnr-ux and identifying as an independent contributor not affiliated with safelabs-eval, said they had reviewed safelabs/prompts/schemas.py, safelabs/runner.py, and safelabs/scoring/models.py. They proposed mappings from safelabs-eval PromptEntry, EvalRecord, EvalResult, detector output, and verdict levels into EvalPort’s portable TestCase, grader, and result structures.

The design discussion happened in public

The proposal raised three concrete design questions: how FAIL and VULNERABLE should map into numeric scores, whether the integration should live inside safelabs-eval or externally in EvalPort, and how to preserve raw model responses without making the adapter the only retention point.

Waqar Javed, maintainer of safelabs-eval, confirmed the core mapping, recommended keeping FAIL and VULNERABLE distinguishable through metadata while sharing a failing numeric score, preferred an external adapter maintained on the EvalPort side, and agreed that raw responses should pass through unchanged.

The adapter was then built and merged

The EvalPort maintainer subsequently implemented the adapter in the external EvalPort repository and merged it through pull request #31. The completed adapter mapped PromptEntry to TestCase, detector outputs to custom graders, EvalRecord to result objects, and EvalResult to a portable ResultSet.

The maintainer reported 29 passing tests, with no failures or skips. Importantly, the test set included two integration tests against the real installed safelabs-eval package rather than only duck-typed fixtures. The adapter also validated generated suites and result sets against EvalPort’s own validation routines.

Why this matters

The value of the integration is not that it proves safelabs-eval is broadly adopted. It does not. What it does prove is narrower and more useful: an unaffiliated developer inspected the public interfaces closely enough to design an interoperability layer, implemented that layer in an independently maintained repository, and tested it against the installed package.

That kind of external technical engagement is an early signal that a project’s data model is understandable outside the team that created it. It also creates a path for evaluation artifacts to be moved into a common interchange format instead of remaining locked inside one framework.

The project has changed since the adapter was proposed

safelabs-eval has continued to evolve. Its current repository now ships 300 adversarial prompts, 30 per internal ASI category, with three difficulty tiers plus provenance and MITRE ATLAS metadata. It also documents seven framework adapters: LangChain, CrewAI, AutoGen/ag2, LlamaIndex, OpenAI Agents SDK, Google ADK, and Semantic Kernel.

Because the corpus and metadata schema have expanded, interoperability should be treated as versioned engineering rather than a one-time compatibility claim. The safest interpretation of the EvalPort work is therefore historical and concrete: the adapter was built against the public safelabs-eval shapes available at the time, passed its reported tests, and established an external integration pattern that can be updated as both projects evolve.

Resource links