Question
Why is theoretical computer science so dry in jobs, except for the academia? Though every company is faced with challenges, there is no hard-fought war against tough problems, and people tend to choose the easy way around every problem.
Answer
This is like asking, "I really wanted to specialize in solving crossword puzzles, but everybody in the publishing industry is just focused on the 'easy' problems of writing articles and books."
Theoretical computer science has very little to do with practical computing in the s/w industry per se. It is a branch of mathematics that, like number theory, sometimes has relevance to practical problems, and at other times goes off and does its own thing. I also disagree with Ghalib: practice does not lag behind theory by a few years. That is simplistic and assumes they are going in the same direction and that one is "better" than the other. Neither is true. There are areas where practice is far more sophisticated, and areas where theory is more refined. There are also plenty of areas where the two are simply not relevant to each other.
Practice can inform theory and vice-versa, and the lead-lag on overlap fields can be anywhere from months to years and decades. I know some problem areas in industry that could see improvement by applying the 60 year old theoretical CS technology of linear programming. There are other areas, like UX design, where industry pioneers are leaps and bounds ahead of anything the theoreticians have codified and formalized.
So why don't companies prioritize applying theory where there IS an overlap and practice DOES lag?
Your question contains the assumption that you know where the hard problems are. Possibly that's true. Your mistake lies in assuming that those are also the IMPORTANT problems.
Industry isn't doing what you think it should, because in industry, you don't solve any problem beyond the level where it translates to better overall performance. You don't solve the hard problems. You solve the important ones. You stop at "good enough" on every problem EXCEPT the most important one, because that's the one that constrains overall performance. When people with your approach start to run industries, companies start to go bankrupt. I don't mean that in a bad way. Just that people who are addicted to using their favorite tools often end up solving irrelevant problems just to use them.
Even if you could (say) improve the performance of a logistics system 10x by applying some simple theoretical CS algorithm, there is absolutely no point in doing so if it doesn't affect the bottom line. This is known as the "moving bottleneck" phenomenon. There is no point optimizing your supply chain for faster delivery if your marketing isn't actually producing the demand that requires ANY delivery. Wasting money on the supply chain at that point is more than useless, it is counterproductive, since you'd be piling up inventory FASTER. Read Eli Goldratt's The Goal to understand how this works.
If you've ever actually worked on industrial problems, you'd realize that computational complexity or the lack of a good algorithm is only one of the gazillion places where the bottleneck in a business may lie at any given time. The main issue could be falling market share due to a new competitor, unreliable suppliers, breakdowns in equipment etc.
You either become a generalist and help resolve bottlenecks wherever they are, or wait around until your skills become relevant when the bottleneck moves into your area of expertise.
Even within the computing industry, the bottleneck is only occasionally in the foundational CS base. Search quality is NOT always the #1 problem at Google. Recommendation quality is NOT always the #1 problem at Netflix.
If you REALLY want to make an impact using theoretical CS, look around at different industries and identify the bottleneck problems each is struggling with. If any of them look like theory could provide the necessary breakthrough, go for it. You might wait a long time.
Theoretical computer science has very little to do with practical computing in the s/w industry per se. It is a branch of mathematics that, like number theory, sometimes has relevance to practical problems, and at other times goes off and does its own thing. I also disagree with Ghalib: practice does not lag behind theory by a few years. That is simplistic and assumes they are going in the same direction and that one is "better" than the other. Neither is true. There are areas where practice is far more sophisticated, and areas where theory is more refined. There are also plenty of areas where the two are simply not relevant to each other.
Practice can inform theory and vice-versa, and the lead-lag on overlap fields can be anywhere from months to years and decades. I know some problem areas in industry that could see improvement by applying the 60 year old theoretical CS technology of linear programming. There are other areas, like UX design, where industry pioneers are leaps and bounds ahead of anything the theoreticians have codified and formalized.
So why don't companies prioritize applying theory where there IS an overlap and practice DOES lag?
Your question contains the assumption that you know where the hard problems are. Possibly that's true. Your mistake lies in assuming that those are also the IMPORTANT problems.
Industry isn't doing what you think it should, because in industry, you don't solve any problem beyond the level where it translates to better overall performance. You don't solve the hard problems. You solve the important ones. You stop at "good enough" on every problem EXCEPT the most important one, because that's the one that constrains overall performance. When people with your approach start to run industries, companies start to go bankrupt. I don't mean that in a bad way. Just that people who are addicted to using their favorite tools often end up solving irrelevant problems just to use them.
Even if you could (say) improve the performance of a logistics system 10x by applying some simple theoretical CS algorithm, there is absolutely no point in doing so if it doesn't affect the bottom line. This is known as the "moving bottleneck" phenomenon. There is no point optimizing your supply chain for faster delivery if your marketing isn't actually producing the demand that requires ANY delivery. Wasting money on the supply chain at that point is more than useless, it is counterproductive, since you'd be piling up inventory FASTER. Read Eli Goldratt's The Goal to understand how this works.
If you've ever actually worked on industrial problems, you'd realize that computational complexity or the lack of a good algorithm is only one of the gazillion places where the bottleneck in a business may lie at any given time. The main issue could be falling market share due to a new competitor, unreliable suppliers, breakdowns in equipment etc.
You either become a generalist and help resolve bottlenecks wherever they are, or wait around until your skills become relevant when the bottleneck moves into your area of expertise.
Even within the computing industry, the bottleneck is only occasionally in the foundational CS base. Search quality is NOT always the #1 problem at Google. Recommendation quality is NOT always the #1 problem at Netflix.
If you REALLY want to make an impact using theoretical CS, look around at different industries and identify the bottleneck problems each is struggling with. If any of them look like theory could provide the necessary breakthrough, go for it. You might wait a long time.