Tech Talk: Operations: Evidence, Signals & Blameless Postmortems

Reminders
- TODAY: first-ticket scope agreed on the issue, before you leave
- TODAY: no standup post. Your team's working-session post replaces it
- WEDNESDAY: Reading Presentations 1, pre-reads posted today
- THURSDAY: Game Day 1: The Queue Nobody Drains, announced in advance
- THU OCT 29: first ticket review-ready, demoed at Demo Day 2
CS 4535: Software Design & Delivery
Tech Talk: Operations: Evidence, Signals & Blameless Postmortems
©2026 Jonathan Bell, CC-BY-SA
You Get This Email
We're seeing a widespread problem with Pawtograder not recording student submissions for our project assignment. Only a small number of students' submissions are actually being recorded. The vast majority of students who have pushed real, substantial changes show nothing: no score, "not submitted" on the dashboard, even though their commits are clearly real work in their GitHub repos.
Course staff of another course on Pawtograder · Tue Sep 29
2 minutes, with the person next to you. You're on call. Write down the first three things you'd check, and for each one, where the answer lives.
Where Would the Answer Live?
Keep this board. We'll come back and see which columns actually answered it.
Learning Objectives
After this session, you'll be able to:
- Explain why "what happened?" could be answered from what Pawtograder recorded and declared, and say where its evidence lives and what was already gone
- Say what tells a failure apart from normal behavior, and why the software should record it at the moment it knows
- Write a blameless postmortem action item, and say who has to be awake for it to work
Pawtograder Wasn't Losing Them
240 student repositories. 13 students had pushed. 44 student pushes reached Pawtograder, and every one was recorded.
43 of the 44 were rejected, each with the same message: "it is identical to the assignment's starter code." 40 of those were real work.
One rejected push was a 10-file change. And one student the staff named as "working" had a submission with no files at all: someone had clicked "make active" on a rejected row.
One File Nobody Edited
The assignment's pawtograder.yml, in the course's grader repository:
submissionFiles:
files:
# The default is to submit all java, arr, md, and python files.
# You should almost definitely change this.
- '**/*.java'
- '**/*.arr'
- '**/*.py'
- '**/*.md'
The project is TypeScript. Pawtograder decides "empty" by comparing only the files these patterns match against the starter code. In the starter code, that's one file: README.md.
Any push that didn't touch a Markdown file looked identical. The one student who added a .md file got through.
Who Answered Each Question?
| Question | Where the answer lived | What put it there |
|---|---|---|
| Can I look without breaking anything? | A read-only replica of the database, and a narrowly scoped cluster login | Infrastructure decision |
| Were the pushes recorded? | Submission rows, kept on purpose when rejected | Software decision |
| Why were they rejected? | An error row with a machine-readable type: empty_submission | Software decision |
| What was it comparing? | The config, copied into the database, and the stored hash of each starter file | Software decision |
| Which code was running? | The image tag pinned in the deploy config, which names one commit | Infrastructure decision |
| Who made the empty row "active"? | An audit table of every change, with who made it | Software decision |
| Did any pushes go missing? | GitHub's record of pushes, compared with ours | Outside: our logs were gone |
Almost every answer was in the database, because someone designed it to be there.
Why the Fix Was a Commit
Definition. Infrastructure as code (IaC) means the servers, containers, networks and settings a system runs on are described in files under version control, and a tool makes the running system match those files. You change the system by changing a file, not by clicking in a console.
In Pawtograder
- A Helm chart (a template for Kubernetes resources) plus one values file per environment, such as
values-prod.yaml - Kubernetes itself: every service is a file describing what should run
pawtograder.ymlper assignment: configuration as code, the same idea for app settings
What it buys you
- Review. A change to production is a pull request
- Reproduce. Staging and previews come from the same files
- Answer. "What is running?" has a written answer
Where the Evidence Lives
Definition. Observability is how well you can answer a new question about a running system from what it already records, without shipping new code. What it records is telemetry.
| Answers | Pawtograder today | |
|---|---|---|
| Errors | What broke, where, in which release, for whom | Sentry → Bugsink, every event tagged with the commit |
| Logs | What happened, line by line, in one process | Only what a pod still holds: about two hours |
| Metrics | How much, how often, how fast, over time | Prometheus, graphed in Grafana, with 37 alert rules |
| Traces | Where one request spent its time, across services | Off |
Definition. An alert is a rule over telemetry that notifies a person when it's true: "this number has been above that line for ten minutes."
The Number Was Already There
Pawtograder already publishes this metric, every minute, per course:
web_workflow_errors_recent{class_id="…", category="empty_submission", window="1h"}
For six days it counted one course's pushes being rejected as empty. Of the 37 alert rules, none looks at it.
It also counts every new repository's first push. That push is the starter code, and rejecting it is correct. A misconfigured course and a normal Monday look the same in this number.
Normal. The commit changes nothing compared with the starter code.
Misconfigured. The commit changes files, and none of them match submissionFiles.
The software knew the difference at the moment it rejected the push.
Blameless Postmortem: What Changes?
Definition. A blameless postmortem assumes everyone acted reasonably on what they knew, and asks what in the system let the failure happen. Its output is action items: changes that make it not happen again.
What let it happen: a default that was right for Java and wrong for TypeScript · a file nobody had a reason to open · no signal to anyone for six days
4 minutes, whole room. Write one action item. Who has to be awake for it to work?
| Action item | Who has to act |
|---|---|
| Page an operator when a course's pushes change only ignored files | An operator, who then emails the course staff |
| Tell the course staff directly, on their dashboard | The staff, who own the file |
Tell the student at push time: you changed 10 files, and this assignment only collects .java, .py, .md, .arr | The student, who asks a TA that day |
Check the patterns when the assignment is created: they match only README.md in this starter | Nobody. It never ships |
| Default the patterns from the starter's own files | Nobody |
Each row down takes a person out of the loop. The alert is the top row.
Working Session
1 · Read your feedback (7 min)
Sort your Demo Day sheets into four piles:
- questions you couldn't answer
- findings people didn't believe
- pushback on a risk or a cut
- the "not yet" reasons
Pick 2 or 3 to act on.
2 · Break it down to Nov 23 (15 min)
Each milestone becomes team: issues. Every issue has:
- an owner and a size
- the milestone it belongs to
- one line: how would we know it broke, and who finds out first?
3 · Your first ticket (while you work)
I come to each table. Before I leave it, every member's scope is written as a comment on their issue, and I've agreed to it there.
4 · Post it (last 5 min)
In your team channel:
- what you're changing because of the feedback, and why
- a link to your issues, filtered by milestone
- each member's first-ticket issue
Up Next
Wed Oct 21, Reading Presentations 1
Then Thu Oct 22, Game Day 1: The Queue Nobody Drains. It's announced, and the response is narrated. Bring what you learned today.
Before then:
- Pre-reads for Wednesday's presentations, posted today
- First ticket: scope agreed on the issue today. Start the first PR in the stack this week
- Observability: your function responds first on Thursday. Have Grafana and Sentry open