Academic Affiliate
The work will entail:
This position involves supporting the Text Retrieval Conference (TREC, trec.nist.gov), a NIST project to
conduct evaluations of research-and-development level technology in the areas of information retrieval,
search, natural language processing, and multimedia search. The TREC project builds datasets for
measuring these types of AI technologies, using human-created labeled data. The process for building
those datasets involves holding an open evaluation exercise to collect outputs that represent the state
of the art at that time. At the end of the evaluation cycle, participants in TREC receive detailed
measurements of their system’s effectiveness, and NIST releases a new dataset built from that
evaluation, along with a volume of academic papers describing the research behind the participants’
work.
Evaluation of search, natural language processing, multimedia, and generative information systems
- A PhD degree in Computer Science.
- 5+ years of relevant experience.
- Experience building systems with large language models as components.
- Familiarity with Docker and Git.
- Familiarity with multiple scripting languages.
- Ability to develop prototypes of tools needed to analyze data.
- Strong oral and written communication skills.
Key responsibilities will include but are not limited to:
- Developing software to support topic development, relevance assessment, and generative
output annotation. - Developing scoring software for evaluation outputs.
- Maintaining and developing evaluation support systems that allow participants to register,
submit outputs, and see evaluation results. - Conducting and/or contributing to research on evaluating information access systems.
Research project opportunities include but are not limited to:
- Automating the evaluation of AI-generated outputs.
- Leveraging human insight and expertise with AI support for data annotation and evaluation.
- Designing metrics for Ai-generated outputs in information-seeking contexts.
- Interfaces for annotating data that support consistency and identify errors.
- Quality control processes for data annotation.
- Leaderboard designs that support cooperative research instead of competition.