Skip to main content

Tech Talk: Operations: Evidence, Signals & Blameless Postmortems

Pixel art of a staircase descending left to right. At the top, an operator is woken at night by a pager. In the middle, a student reads a helpful message at a laptop. At the bottom, a robot with a green checkmark stops a bad file at a gate in daylight. Tagline: Take the Person Out of the Loop.

Reminders​

  • TODAY: first-ticket scope agreed on the issue, before you leave
  • TODAY: no standup post. Your team's working-session post replaces it
  • WEDNESDAY: Reading Presentations 1, pre-reads posted today
  • THURSDAY: Game Day 1: The Queue Nobody Drains, announced in advance
  • THU OCT 29: first ticket review-ready, demoed at Demo Day 2

CS 4535: Software Design & Delivery

Tech Talk: Operations: Evidence, Signals & Blameless Postmortems​

©2026 Jonathan Bell, CC-BY-SA

You Get This Email​

We're seeing a widespread problem with Pawtograder not recording student submissions for our project assignment. Only a small number of students' submissions are actually being recorded. The vast majority of students who have pushed real, substantial changes show nothing: no score, "not submitted" on the dashboard, even though their commits are clearly real work in their GitHub repos.

Course staff of another course on Pawtograder · Tue Sep 29

2 minutes, with the person next to you. You're on call. Write down the first three things you'd check, and for each one, where the answer lives.

Where Would the Answer Live?​

The database
Logs
Dashboards & alerts
GitHub
The code & its config
Ask a person

Keep this board. We'll come back and see which columns actually answered it.

Learning Objectives​

After this session, you'll be able to:

  1. Explain why "what happened?" could be answered from what Pawtograder recorded and declared, and say where its evidence lives and what was already gone
  2. Say what tells a failure apart from normal behavior, and why the software should record it at the moment it knows
  3. Write a blameless postmortem action item, and say who has to be awake for it to work

Pawtograder Wasn't Losing Them​

240 student repositories. 13 students had pushed. 44 student pushes reached Pawtograder, and every one was recorded.

43 of the 44 were rejected, each with the same message: "it is identical to the assignment's starter code." 40 of those were real work.

One rejected push was a 10-file change. And one student the staff named as "working" had a submission with no files at all: someone had clicked "make active" on a rejected row.

One File Nobody Edited​

The assignment's pawtograder.yml, in the course's grader repository:

submissionFiles:
files:
# The default is to submit all java, arr, md, and python files.
# You should almost definitely change this.
- '**/*.java'
- '**/*.arr'
- '**/*.py'
- '**/*.md'

The project is TypeScript. Pawtograder decides "empty" by comparing only the files these patterns match against the starter code. In the starter code, that's one file: README.md.

Any push that didn't touch a Markdown file looked identical. The one student who added a .md file got through.

Who Answered Each Question?​

QuestionWhere the answer livedWhat put it there
Can I look without breaking anything?A read-only replica of the database, and a narrowly scoped cluster loginInfrastructure decision
Were the pushes recorded?Submission rows, kept on purpose when rejectedSoftware decision
Why were they rejected?An error row with a machine-readable type: empty_submissionSoftware decision
What was it comparing?The config, copied into the database, and the stored hash of each starter fileSoftware decision
Which code was running?The image tag pinned in the deploy config, which names one commitInfrastructure decision
Who made the empty row "active"?An audit table of every change, with who made itSoftware decision
Did any pushes go missing?GitHub's record of pushes, compared with oursOutside: our logs were gone

Almost every answer was in the database, because someone designed it to be there.

Why the Fix Was a Commit​

Definition. Infrastructure as code (IaC) means the servers, containers, networks and settings a system runs on are described in files under version control, and a tool makes the running system match those files. You change the system by changing a file, not by clicking in a console.

In Pawtograder

  • A Helm chart (a template for Kubernetes resources) plus one values file per environment, such as values-prod.yaml
  • Kubernetes itself: every service is a file describing what should run
  • pawtograder.yml per assignment: configuration as code, the same idea for app settings

What it buys you

  • Review. A change to production is a pull request
  • Reproduce. Staging and previews come from the same files
  • Answer. "What is running?" has a written answer

Where the Evidence Lives​

Definition. Observability is how well you can answer a new question about a running system from what it already records, without shipping new code. What it records is telemetry.

AnswersPawtograder today
ErrorsWhat broke, where, in which release, for whomSentry → Bugsink, every event tagged with the commit
LogsWhat happened, line by line, in one processOnly what a pod still holds: about two hours
MetricsHow much, how often, how fast, over timePrometheus, graphed in Grafana, with 37 alert rules
TracesWhere one request spent its time, across servicesOff

Definition. An alert is a rule over telemetry that notifies a person when it's true: "this number has been above that line for ten minutes."

The Number Was Already There​

Pawtograder already publishes this metric, every minute, per course:

web_workflow_errors_recent{class_id="…", category="empty_submission", window="1h"}

For six days it counted one course's pushes being rejected as empty. Of the 37 alert rules, none looks at it.

It also counts every new repository's first push. That push is the starter code, and rejecting it is correct. A misconfigured course and a normal Monday look the same in this number.

Normal. The commit changes nothing compared with the starter code.

Misconfigured. The commit changes files, and none of them match submissionFiles.

The software knew the difference at the moment it rejected the push.

Blameless Postmortem: What Changes?​

Definition. A blameless postmortem assumes everyone acted reasonably on what they knew, and asks what in the system let the failure happen. Its output is action items: changes that make it not happen again.

What let it happen: a default that was right for Java and wrong for TypeScript · a file nobody had a reason to open · no signal to anyone for six days

4 minutes, whole room. Write one action item. Who has to be awake for it to work?

Action itemWho has to act
Page an operator when a course's pushes change only ignored filesAn operator, who then emails the course staff
Tell the course staff directly, on their dashboardThe staff, who own the file
Tell the student at push time: you changed 10 files, and this assignment only collects .java, .py, .md, .arrThe student, who asks a TA that day
Check the patterns when the assignment is created: they match only README.md in this starterNobody. It never ships
Default the patterns from the starter's own filesNobody

Each row down takes a person out of the loop. The alert is the top row.

Working Session​

1 · Read your feedback (7 min)

Sort your Demo Day sheets into four piles:

  • questions you couldn't answer
  • findings people didn't believe
  • pushback on a risk or a cut
  • the "not yet" reasons

Pick 2 or 3 to act on.

2 · Break it down to Nov 23 (15 min)

Each milestone becomes team: issues. Every issue has:

  • an owner and a size
  • the milestone it belongs to
  • one line: how would we know it broke, and who finds out first?

3 · Your first ticket (while you work)

I come to each table. Before I leave it, every member's scope is written as a comment on their issue, and I've agreed to it there.

4 · Post it (last 5 min)

In your team channel:

  • what you're changing because of the feedback, and why
  • a link to your issues, filtered by milestone
  • each member's first-ticket issue

Up Next​

Wed Oct 21, Reading Presentations 1

Then Thu Oct 22, Game Day 1: The Queue Nobody Drains. It's announced, and the response is narrated. Bring what you learned today.

Before then:

  • Pre-reads for Wednesday's presentations, posted today
  • First ticket: scope agreed on the issue today. Start the first PR in the stack this week
  • Observability: your function responds first on Thursday. Have Grafana and Sentry open