PREP0004070 LLM Performance Metrics Researcher

PREP Research Associate

Opportunity No.: PREP0004070

This position is part of the National Institute of Standards and Technology (NIST) Professional Research Experience (PREP) program. NIST recognizes that its research staff may wish to collaborate with researchers at academic institutions on specific projects of mutual interest, and thus requires that such institutions must be the recipient of a PREP award. The PREP program requires staff from a wide range of backgrounds to work on scientific research in many areas. Employees in this position will perform technical work that underpins the scientific research of the collaboration.

The position is in the Applied Economics Office (AEO), a part of the Engineering Laboratory (EL) at NIST, which provides economic products and services through research and consulting to industry and government agencies in support of productivity enhancement, economic growth, and international competitiveness, with a focus on improving the life-cycle quality and economy of constructed facilities and manufacturing processes that support social and economic functions. AEO is integrated within EL’s major research thrusts. AEO delivers high quality research and tool development that informs and assists stakeholders in their decision-making processes. The position will collaborate directly with the software development team in EL’s ELDST (Engineering Laboratory Data, Security, & Technology) that oversees AEO’s software development projects.

APPLY ONLINE HERE

LLM Peformance Metrics Researcher

Project Title:
Performance and efficacy of machine learning and artificial intelligence implementation in economics and life cycle system science decision support

Project Description:

The sponsor seeks a computer scientist (US citizen) to join their team in researching the performance and efficacy of machine learning and artificial intelligence (large language models - LLMs) across a variety of both research-oriented and public-facing software applications using standardized science-based metrics. The initial focus is on developing LLM-based outputs, and comparing their performance relative to manually developed outputs, including drafting annotated bibliographies and literature reviews, writing code, web applications that incorporate LLMs to enhance capabilities, and LLM-based web applications.

The ideal candidate will have a strong background in integrating LLM APIs, React, and front-end programming, and will be responsible for transforming models into usable APIs or integrated tools for production. They will also be responsible for monitoring, troubleshooting and enhancing model efficiency and scalability. The candidate should also be well-versed in handling data preprocessing, and analysis for model training. The candidate should be aware of various prompt engineering techniques, implementing intelligent prompt caching (e.g., Redis), understanding of vector stores (e.g., Pinecone), and efficient token management. The candidate should have knowledge of implementing AI security protocols, including guardrails and techniques to prevent prompt injection. The candidate should also have a working knowledge of RAG (Retrieval Augmented Generation). Additional research tasks may be assigned based on candidate’s skillset and priorities.

Key Responsibilities: 

• Develop user interfaces for web application
• Assist with special software development projects as assigned
• Write and implement efficient code
• Work closely with other developers
• Statistically compare performance across code and tool designs
• Draft manuscripts documenting the methodology and results

Desired Qualifications: 

• US Citizen
• Master’s degree in Computer Science or related field
• At least 2 years of professional experience
• At least 1 year as development team lead for at least one web application using the software stacks listed below
• Experience with state management in React (RxJS)
• Experience working on cloud technologies (AWS, Azure)
• Working knowledge of RAG (Retrieval Augmented Generation)
• Proficient with integrating one or more LLMs into applications (e.g., OpenAI, Gemini, Llama)
• Proficient with HTML, CSS, Typescript, React, and Python
• Proficient using any UI Component libraries (e.g., Ant Design, Material UI, etc.)
• Proficient with Node.js
• Working knowledge of building quick prototypes using Streamlit (or similar) and LLMs
• Proficient with JSON
• Proficient with Vite, Nginx, GitHub, Docker, and Portainer
• GPU programming or data visualization experience a plus
• Evidence of strong oral and written communication skills, including authorship on at least 2 technical publications)
• Strong logical thinking and problem solving
• Excellent attention to detail

Other Details:

• Full-time: the participant is expected to work 40 hours a week
• Location: the participant will work at the NIST Gaithersburg Campus.
• Duration: this is expected to be a one-year position. Extensions are sometimes granted depending on the availability of funds.
• For questions related to the research project or the nature of the work in this position, please contact Dr. Joshua D. Kneifel (joshua.kneifel@nist.gov). For questions related to the online application or NIST PREP more generally, please contact msu-nistprep@morgan.edu.

Privacy Act Statement

Authority: 15 U.S.C. § 278g-1(e)(1) and (e)(3) and 15 U.S.C. § 272(b) and (c)

Purpose: The National Institute for Standards and Technology (NIST) hosts the Professional Research Experience Program (PREP) which is designed to provide valuable laboratory experience and financial assistance to undergraduates, post-bachelor’s degree holders, graduate students, master’s degree holders, postdocs, and faculty.

PREP is a 5-year cooperative agreement between NIST laboratories and participating PREP Universities to establish a collaborative research relationship between NIST and U.S. institutions of higher education in the following disciplines including (but may not be limited to) biochemistry, biological sciences, chemistry, computer science, engineering, electronics, materials science, mathematics, nanoscale science, neutron science, physical science, physics, and statistics. This collection of information is needed to facilitate the administrative functions of the PREP Program.

Routine Uses: NIST will use the information collected to perform the requisite reviews of the applications to determine eligibility, and to meet programmatic requirements. Disclosure of this information is also subject to all the published routine uses as identified in the Privacy Act System of Records Notices: NIST-1: NIST Associates.

Disclosure: Furnishing this information is voluntary. When you submit the form, you are indicating your voluntary consent for NIST to use the information you submit for the purpose stated.