|
Scientific debugging your project
Research is the systematic falsification of assumptions. Most weeks, an experiment we tried will not work; that is normal, expected, and how research advances. What matters is not whether it worked, but what you learned from it.
What your advisor already knows and what they don't
As your advisor, I already know, a priori, that roughly 80% of your weekly reports will say “the idea did not work” or “the numbers went down”. This does not surprise me. Negative results are the default in research; if they were rare, we would not need experiments.
What I do not know is why it did not work. That is the part you must bring to the meeting. If your report is just “I ran the experiment and accuracy dropped 2%”, you have told me nothing I did not already expect. If your report is “accuracy dropped 2%, and here is why my hypothesis was wrong”, now we can actually make progress.
Pull me back from the cloud
Between our meetings, I live “in the cloud” of assumptions: I have a mental model of your data, your model, your baselines, and how they should behave. These assumptions come from intuition, prior work, and analogies to other projects. They are almost always partially wrong; they just haven't been checked against reality yet.
You are the one who touches the data and runs the code, so you have the ground truth. Every meeting is a chance to pull me back from the cloud and tell me which of my (and your own) assumptions the data has refuted this week. Without you doing this, I will keep building on top of assumptions that a run of the model would have already disproved.
So when you report a negative result, do not stop at “it did not work”. Ask: which specific assumption did this experiment invalidate?
What a good failure report looks like
The table below contrasts weak reports (just the outcome) with strong reports (the outcome, plus which assumption was falsified and how you know).
| Weak report (outcome only) |
Strong report (which assumption was wrong and why) |
| "I tried the new loss but accuracy dropped 2%." |
"The new loss dropped accuracy 2%. We assumed the auxiliary term would push logits apart on hard cases, but the confusion matrix shows the drop is entirely on easy examples where logits were already well separated — the extra term is adding noise to already-confident predictions. The 'hard cases' assumption was wrong." |
| "The model does not converge." |
"The model diverges in the first 200 steps with the new initialization. We assumed the pre-trained backbone's activations would match the range the decoder expects, but a quick print shows the decoder inputs are ~10× larger than expected, so softmax saturates immediately. The assumption that backbone and decoder scales match was wrong." |
| "Our method is slower than the baseline." |
"Our method is 3× slower than the baseline. We assumed most cost would be in the attention block, but profiling shows 60% of runtime is in the CPU-side negative-pair sampling that we had treated as free. The assumption that sampling cost is negligible was wrong; we need to precompute it or move it to GPU." |
| "I retrained on the larger dataset, no improvement." |
"Retrained on the 10× larger dataset, no improvement in mAP. We assumed we were data-limited, but training loss plateaus at nearly the same value on both dataset sizes — so we are model-capacity-limited, not data-limited. The 'more data will help' assumption was wrong." |
| "The baseline is very hard to beat." |
"Cannot beat the baseline on Dataset X. We assumed our method would help most on long-tail classes, but per-class accuracy shows we are on par on rare classes and losing on frequent ones. The assumption about where the gain would come from was wrong; the mechanism we designed helps a different slice of the data than we thought." |
| "Adding a second view of each object did not improve performance." |
"Adding a second (side) view per object did not improve mAP. We assumed the extra view would resolve occlusion ambiguity, but per-example accuracy is unchanged on occluded instances and slightly worse on unoccluded ones — the fusion module appears to be averaging away discriminative single-view features. The assumption that occlusion was the dominant error source was wrong; the bottleneck is somewhere else." |
| "Removing blurry frames from the videos did not help." |
"Dropping the 15% blurriest frames from training did not improve action-recognition accuracy. We assumed motion blur was hurting the temporal encoder, but a class-wise breakdown shows blurry frames are concentrated in fast-motion classes (running, throwing) where they are actually the most informative cue — removing them threw away useful signal. The 'blur is noise' assumption was wrong for this task." |
Notice that in every “strong” report, three things are present: the outcome, the specific assumption that was invalidated, and the concrete evidence (confusion matrix, print statement, profiler, per-class breakdown) that invalidated it. This is what turns a negative result into information.
A checklist before you say “it did not work”
Before you show up to the meeting with “it did not work”, run through these:
What assumption did I start from when I proposed this experiment? Write it down in one sentence.
What did I expect to see if the assumption were correct?
What did I actually see?
Which of the two — the assumption or the implementation — is more likely responsible for the gap? What is the cheapest experiment that would tell us?
Is there a plot, table, or number I can show that makes the failure legible in ten seconds?
If you can answer these, your weekly meeting will be useful even when the experiment failed. In fact, especially when it failed — a well-analyzed failure often changes the direction of the project more than a successful run does.
Positive results are not exempt
The same discipline applies when the numbers go up. Before you get excited, doubt the result and try to break it: is the improvement within noise across seeds? Is the test set really disjoint from training? Does the qualitative output match the quantitative claim, or did the metric go up for a reason unrelated to what we were trying to fix? A positive result you cannot explain is only marginally more useful than a negative one you cannot explain.
Summary
Failure is the default. Reporting failure is not the news; explaining why it failed is.
Your advisor lives on assumptions between meetings. Your job in the meeting is to tell them which assumptions the week's experiments have refuted.
Every negative result should name the invalidated assumption and show the evidence.
“I made no progress this week” is fine, as long as it is not for lack of trying and analyzing. “I made no progress and I do not know why” is not.
|