I’ve recently been doing a lot of problem solving on a personal and professional level the past couple of weeks. A pattern began to emerge. A pattern which should be obvious but often isn’t. There’s a problem you’re trying to solve, say getting the result of 2+2. You come up with a solution to count sticks and they don’t add up to 4.
So, you decide to count stones instead. Now you’re getting 6. Then you decide to use color markers, and the new result is 17. You decide to pay professionals to build you a pair of 6-feet stature so you can add it up. But then, you’re now getting 223. At some point, you realize that you’ve drifted far off from trying to solve the problem to trying to make your current solution work.
This rudimentary example happens a lot in life. I believe the wise people of old call it sunk cost fallacy.The sunk cost fallacy is a cognitive bias that causes people to continue an endeavor, or continue consuming or pursuing an option, because of previously invested resources (time, money, effort), even when it’s no longer rational to do so.
Imagine you’ve bought a $100 ticket to a concert. On the day of the concert, it’s storming, and you’re feeling sick. Going to the concert would make you miserable. Rational decision-making would say: “I should stay home and rest.” But because you spent $100, you might feel compelled to go anyway, even though the buses are on strike and your car broke down — that’s the sunk cost fallacy.
As an engineer/designer, you come up with a solution. But then it keeps breaking and missing several edge cases. And you keep patching it. At some point, you must realize that maybe the solution is the problem. I’ve built solutions that took hours, several thousands of lines of code and I keep adding more and more to make it more efficient, which kept introducing new problems to solve. It’s exciting at first.
But after a while, you must realize that the effort being put into making that solution work is better invested in coming up with a more elegant solution.
Recently, I had to implement a solution that logs events happening within a system. The important thing is that we don’t want that logging to degrade user experience. After the typical brain racking, the solution I came up with was to use a queue. Smart right? When events occur, just send to a queue and move on.
Then we have another system that processes the queue items as fast as possible. However, a small stress test quickly revealed 2 issues. First was that the queue grew very quickly, and secondly, that some queue items need to be processed sequentially for correctness, but our queue processor can’t guarantee that. To solve the first problem, I decided to add more resources to the queue processor.
Instead of 1 queue processor, why not 5 or 10? Processing should be faster, no? Unfortunately not. Each processor tries to grab tasks from the same queue, so they often end up competing over who gets to process what.
Even after processing, they all need to put their results to the same place, which causes more delays. Essentially, these queue processors will spend a lot of their time waiting and contending, instead of doing actual processing.
After hours of work trying to make the queue processors more efficient, it occurred to me “I don’t need to make this queue work. I just need to solve the original problem a better”. What was the original problem? Log event but don’t make the user wait for you.
Eventually, I came up with another solution, the same entity that attends to the user should log the event, just do it after responding to the user. I ran the same stress test on this and voilà! it worked more efficiently than I expected, without having to commit more resources to it.
The lesson is not to give up on any problem you’re trying to solve. It’s that after trying to solve it a particular way for so long, it’s probably time to give up on that solution and re-assess.
If a solution consistently introduces new problems or does not consistently solve the original problem you’re trying to solve, it’s probably not the most elegant solution for that problem.
And if you continue to try to make that solution work, you’re almost guaranteed to spend the rest of your life still trying to make it work.
For example, your personal finance, you realize you’re constantly broke before the month ends, despite being a salary earner. One way to solve that is to take a loan to stay afloat till the next salary comes in.
However, months later, you’ll realize you’ve been accumulating and rolling over debt, despite having finite resources. Then you decide to just start taking stuff on credit. And when that introduces new problems, you start asking everyone you know for money.
You see where I’m going with this? You’ve left the original problem and are now trying to “make the solution work”.
Eventually, you must take a step back to re-assess the original problem i.e money finishes before the next salary comes in. A less complicated solution could be to rebase your monthly expenses so it fits inside your monthly earning.
Lastly, if you’re trying to get a door open, you can try different keys and different key sizes. But you must consider at some point, that maybe that’s not a door.