Academic affiliate
This position involves supporting the Text Retrieval Conference (TREC, trec.nist.gov), a NIST project to conduct evaluations of research-and-development level technology in the areas of information retrieval, search, natural language processing, and multimedia search. The TREC project builds datasets for measuring these types of AI technologies, using human-created labeled data. The process for building those datasets involves holding an open evaluation exercise to collect outputs that represent the state of the art at that time. At the end of the evaluation cycle, participants in TREC receive detailed measurements of their system’s effectiveness, and NIST releases a new dataset built from that evaluation, along with a volume of academic papers describing the research behind the participants’ work.
Evaluation of search, natural language processing, multimedia, and generative information systems
- A PhD degree in Computer Science.
- 5+ years of relevant experience.
- Experience building systems with large language models as components.
- Familiarity with Docker and Git.
- Familiarity with multiple scripting languages.
- Ability to develop prototypes of tools needed to analyze data.
- Strong oral and written communication skills.
Key responsibilities will include but are not limited to:
- Developing software to support topic development, relevance assessment, and generative output annotation.
- Developing scoring software for evaluation outputs.
- Maintaining and developing evaluation support systems that allow participants to register, submit outputs, and see evaluation results.
- Conducting and/or contributing to research on evaluating information access systems.