← All projects

EECS 591 Distributed Systems Fall 2023

Evaluating LLMs on Distributed-System Specifications

Team course project

Archived course report (PDF)

This team project evaluated LLM understanding of TLA+, a language for specifying how systems behave. We tested protocol recognition after removing name clues, and the detection and explanation of logical errors in two-phase commit, Paxos, and Raft specifications.

The team reviewed 440 responses from GPT-3.5, GPT-4, PaLM, and Llama-2. The evaluation distinguished flagging a bug from correctly explaining it.

Llama-2: identifying a bug versus explaining it
ResponseTrials
Flagged the specification as incorrect79 / 80
Included an accepted bug explanation15 / 80
2023 results for two-phase commit and Paxos: ten buggy specifications each, tested four times, with manual grading (Tables 4–5).

Flagging a problem did not reliably mean explaining it. All inputs in this comparison contained bugs, so these counts measure responses to buggy specifications, not overall detector accuracy or formal verification.

Project team: Houming Chen, Daniel Hou, Oh Jun Kweon.