I personally found this to be a poor article on the subject. There's very little actual content about debugging, he spent more time talking about theories of intelligence than debugging.
I've recently been thinking about this, thinking about making a course. It's my personal experience that his approach is flawed. Fundamentally, step one is not develop a theory. That is the worst thing you can do, it can waste a lot of time. I still sometimes guess what the problem is, but all I really use that for today is where to look and that's sometimes a waste of my time as I skipped step 1.
Step 1 is recreate it. No theorizing, no thinking, no faffing around. Can you recreate the bug exactly? If you can't, find out why. Talk to the bug reporter, find out what they were doing, this is a people skill you need to learn as a programmer. Maybe you need access to the live data to see the state an object is in. There's only a tiny percentage of bugs (usually race conditions) that can't easily be recreated. If you can't recreate it, that's when you turn to logging to try and catch the conditions to recreate it, but logging is the last thing you should try and I rarely ever have to add it (we're talking 1% of bugs). I often see junior programmers littering code with log statements when trying to debug, I feel this is a bad method, but can understand the temptation, especially when they don't really understand what the code is doing (an understandable position I was in in my earlier days).
Step 2 is isolate it. If the code is simple and it's obviously one method, no further work needed. If the code is complex, you need to really isolate the exact problem. Say you've got the wrong value displaying for a basket total, where is that coming from? Is it not updating, or is the total function the problem? Isolate the exact step in the process that's going wrong. Sometimes it's hard to isolate and logging is needed again, but same advice as before, logging is a rare necessity.
Step 3 is make it easy to run a recreation. If you can run it in 10 secs or less, that's great. If it's one method of a complex web call, make a test endpoint that only calls that method that's failing and exactly recreates the conditions. Then you can run it quickly without all the cruft. In an extreme case, this can be taking the code completely out of the context and putting it in a tiny standalone app.
Step 4 is read + understand the code. Is there an obvious logic bug? Don't jump to conclusions. Still don't get what's going on? Step through the code, breakpoints. Understand what's going wrong. Breakpoints and stepping is fine, inspecting variables, etc.
Step 5 is fix it. This should be fairly simple now. This is also the time to think about the future. Is this part of the code a constant headache? Is it buggy because it's over-complicated? It might need a bit of a refactor to make sure this doesn't happen again. This is a judgement call, unnecessary refactors are expensive, might even introduce new bugs, but over-complicated code will lead to more bugs. It not being written in your favourite style is generally not a legitimate reason to refactor.
Step 6 is testing the fix, recreating the conditions from step 1. Don't rely on QA if you have them. Ensure you've done your job.
(Step 7 get rid of your logging statements + test it again! You just changed the code)
Now you can skip steps, be far less formal about it. Sometimes it's obvious. Sometimes it's hard and you may need to iterate over 4/5/6 a few times. But never, ever skip steps 1 + 6. Recreate it. Test your fix. Because if you skip those steps, and you "fix" it, you don't actually know it's fixed.
I 99% agree with you, I would 100% agree with you if you put step 6 between steps 3 and 4. In my opinion you should always (where possible) be able to deterministically and automatically recreate a bug before you even think about fixing it, the automatic part being your test. I say this because human nature being what it is you will forget how you recreated the bug, or your recreation will have a bunch of extraneous stuff in that really isn't needed.
Unless you can prove the kind of nondeterminism which triggers the bug. Sometimes it is much harder to write a testcase to reproduce a nondeterministic bug than proving formally that it isn't possible after the fix.
I fully agree with all of this for code. However I have often thought the same techniques and mental model approach for debugging any complex system could be boiled down. I also attempted to write a course along the same lines and kept throwing it away.
In addition to your steps though, I often include a hypothesis combined with various likelihood probabilities (sure there COULD be a JVM bug, but is it likely?), and I attack highest probabilities first.
Thanks for saving me the trouble of writing this post. ;-)
I'll just add this: once you get to Step 3, conditional breakpoints are your best friend. Sometimes they're too time-consuming and you have to resort to manually "instrumenting" the code to get the conditional breakpoint, but the principle is the same.
And all bets are off when considering realtime behaviour such as concurrency or fairness.
Neither debugger nor logs will help (they change behaviour and reproducibility is low) so you have to get to the basics and make a formal review. As in describe the expected algorithm, its preconditions, postconditions and then verify if this contract is met.
Yep, know it well, there's a whole class of applications that don't play well with a debugger. But, most developers aren't working on those applications...
No, not normally. It's fairly rare, and when it does crop up, it is typically handled with pre-baked solutions that are used to spawn a worker thread/process for the purposes of off-loading work from the main UI thread, with minimal communication with the main UI thread via posted messages.
Thanks for your feedback. This article was largely intended to explore why some folks just don't seem good at debugging, and don't seem to get better at it. This is where self-theories comes in. It does a rather good job of explaining that and has a large body of evidence supporting its validity. And importantly, that we can actually do something about that. I mean, in bold, at the top, it says "Understanding the psychology of learning strategies leads to effective problem-solving skills." So I think you should have been prepared for an article that wasn't going to talk about debugging strategy in depth.
I think you might have misunderstood what I meant with "develop a general theory." I say this because you've changed it to "develop a theory," which is explicitly not what I actually wrote. It literally means something completely different in the context of science. A general theory is just a concept. It's the formation of ideas. It's the process of information gathering. Where you go from "this is a bug" to "this is a bug in the logging module" to "this is a bug in the logging module when I'm producing more log messages than I can write" to "this is a concurrency bug" to "aha the actual bug is somewhere else, let me look there", etc.
Your first tangible goal should be reproducing the bug, because that is how a hypothesis is tested. But you don't magically start with a hypothesis unless you have very good intuition, which means this isn't a new bug for you, and so you didn't really need to employ any complicated strategies. My article is not about how to handle obvious, run-of-the-mill bugs.
And so between the implicit step 0 (there is a bug) and step 1 (I can reproduce it), you end up assuming quite a lot.
I mention in the article that reproducing a bug is really crucial, because that is how we know we've found where the gap in our mental model is. You've gone on to describe some ways to do that. I explicitly did not take this approach, because such a list would be exhaustingly long, and my article is already long for an ACM Queue article.
If you've reproduced the bug, you must have already figured out what was wrong. How did you do that? Were you staying up all night staring at the problem instead of taking deliberate breaks? Were you collaborating with colleagues or stubbornly tackling the problem alone? Were you scientific in your approach? These are all things that stem from the root of the problem, and are maladaptive strategies that folks engage in. Strategies that prevent them from getting from step 0 to step 1.
So it seems odd to me that you would call my "approach" flawed. To me, it seems you're not actually starting from the point of "I do not understand the problem." If you go to github, pick a random issue in a random project in a random language, and go to fix it, what is the likelihood that the first actual step you take will be reproducing the bug? This is where people actually start from, and this is where we are every time we encounter a new bug.
I'm not sure I agree with the order of your steps. If you are new to software (new job, entirely new codebase), and one of your first tasks is a bugfix, how are you possibly going to have step 4 ordered before step 1? You need to have some kind of mental model. You're going to need to talk with colleagues, interact with people. Gather some understanding of the system architecture. This is a learning process and an information sharing process. _This is the point of my article._ Thank you for taking the time to share your experience with us!
I'm sorry you didn't like my article. It sounds like we agree more than you seem to think we do, even if I'm a bit critical of your steps. I think we have different audiences in mind, perhaps.
The paper is almost entirely focused on incremental learning. It then throws a very brief outline about debugging at the end. What is outlined is pretty much step 5 in my list. I feel that it's twisting the actual content of the article to assert that it naturally follows that "your first tangible goal is to be reproducing because that's how a hypothesis is tested". If that was the belief, why not mention it? It really is lacking a tangible framework for entering a debugging mindset.
The words recreate or replicate don't appear in the article. Reproduce is used once at the bottom of the article in relation to the actual advice ("so it seems like the bug has been reproduced at this point"), and once tangentially in the discussion about tools. How can there be a discussion about a debugging mindset without those words?
As for step 4 coming before step 1? I feel talking about one month in an entire career is moving the goal posts. The abstract talks about how much time all programmers spend debugging, with no talk about the first month of a programmers career. And it talks about teaching debugging, a mindset to put in place before that first job.
That is what my steps were attempting to describe, a method for teaching how to enter a debugging mindset.
Yes, the paper is almost entirely focused on learning because it is a paper about applying somewhat recent research in social psychology into the area of debugging. It is not a paper about debugging. I can see how you might be disappointed if you expected a paper on debugging, but that is not what this is, and it says so at the top, in bold letters, above the abstract.
Anyway, I disagree with the rest of the points you make here, but I don't really feel like arguing about things that my paper is not (and was not) intended to be. I'm sorry you couldn't get much out of it, and thank you again for your feedback!
I've recently been thinking about this, thinking about making a course. It's my personal experience that his approach is flawed. Fundamentally, step one is not develop a theory. That is the worst thing you can do, it can waste a lot of time. I still sometimes guess what the problem is, but all I really use that for today is where to look and that's sometimes a waste of my time as I skipped step 1.
Step 1 is recreate it. No theorizing, no thinking, no faffing around. Can you recreate the bug exactly? If you can't, find out why. Talk to the bug reporter, find out what they were doing, this is a people skill you need to learn as a programmer. Maybe you need access to the live data to see the state an object is in. There's only a tiny percentage of bugs (usually race conditions) that can't easily be recreated. If you can't recreate it, that's when you turn to logging to try and catch the conditions to recreate it, but logging is the last thing you should try and I rarely ever have to add it (we're talking 1% of bugs). I often see junior programmers littering code with log statements when trying to debug, I feel this is a bad method, but can understand the temptation, especially when they don't really understand what the code is doing (an understandable position I was in in my earlier days).
Step 2 is isolate it. If the code is simple and it's obviously one method, no further work needed. If the code is complex, you need to really isolate the exact problem. Say you've got the wrong value displaying for a basket total, where is that coming from? Is it not updating, or is the total function the problem? Isolate the exact step in the process that's going wrong. Sometimes it's hard to isolate and logging is needed again, but same advice as before, logging is a rare necessity.
Step 3 is make it easy to run a recreation. If you can run it in 10 secs or less, that's great. If it's one method of a complex web call, make a test endpoint that only calls that method that's failing and exactly recreates the conditions. Then you can run it quickly without all the cruft. In an extreme case, this can be taking the code completely out of the context and putting it in a tiny standalone app.
Step 4 is read + understand the code. Is there an obvious logic bug? Don't jump to conclusions. Still don't get what's going on? Step through the code, breakpoints. Understand what's going wrong. Breakpoints and stepping is fine, inspecting variables, etc.
Step 5 is fix it. This should be fairly simple now. This is also the time to think about the future. Is this part of the code a constant headache? Is it buggy because it's over-complicated? It might need a bit of a refactor to make sure this doesn't happen again. This is a judgement call, unnecessary refactors are expensive, might even introduce new bugs, but over-complicated code will lead to more bugs. It not being written in your favourite style is generally not a legitimate reason to refactor.
Step 6 is testing the fix, recreating the conditions from step 1. Don't rely on QA if you have them. Ensure you've done your job.
(Step 7 get rid of your logging statements + test it again! You just changed the code)
Now you can skip steps, be far less formal about it. Sometimes it's obvious. Sometimes it's hard and you may need to iterate over 4/5/6 a few times. But never, ever skip steps 1 + 6. Recreate it. Test your fix. Because if you skip those steps, and you "fix" it, you don't actually know it's fixed.