Is Data Science Dying?

February 2023

I often get told by aspiring data scientists: “I want to be a data scientist. What’s the best way to do it?”. On one hand, data science is the perfect field to go for the naturally curious and analytically rigorous, plus the compensation ain’t bad. On the other I fight the urge to shake them, and beg, for the future of their careers, to not do it.

I wish I could be helpful, truly, but my experience breaking into the field five years ago does not apply now. Data science has become commoditized, saturated, and highly competitive especially for entry-level talent. Look at all of the generic data science courses, guaranteeing jobs if you write a stack-overflow rehash on Medium (which I was guilty of). Compare that to what companies actually want – several years of experience, data engineering, and an unnatural passion for driving revenue.

And in the midst of a white collar recession, it seems like the immediate career prospects for data science in general aren’t too great. Data science doesn’t feel safe anymore, even if you’re experienced. There is a constant battle to prove the upside to our oft-times arcane experiments.

This post isn’t meant to gatekeep aspiring data scientist away from my precious profession. If anything, I want there to be more analytically savvy data practitioners. Join us! We need data fluency to propagate across the workforce, for analytics to be valued and be valuable.

Just be very careful on what prescription to follow, including and especially this one. The data science job market, skills in demand, and tech stack have shifted rapidly recently and will continue to as companies look to tighten the belt in the near future. Before we look into the future, let’s look at how we got here.

The Big Oil and Big Data Boom

It’s the 2010’s. Data was the new oil. Every company raced to acquire Big Data [tm]. Fortune awaited for those who harness it. HBR infamously called the data scientist the sexiest job of the 21st century and doubled down on its prestige a decade later (with caveats).

And it was a great field to get into when I broke into it. Data scientist had quite a weight to its title, at least in tech. I’d often joke that difference between an analyst and a data scientist is that an analyst tests a hypothesis and uses data to test that hypothesis, while a data scientist does the same thing but gets paid six figures. I even insisted on the title searching on my first job search out of college. The data science title could theoretically carry forward into future prospects – it didn’t matter if I was functionally an underpaid analyst at the time (I was).

Nowadays I often question the title. Its meaning, how the market values it, and where its career ladder leads to. And sure enough, there’s data to back up some doubts.

The Data Bust?

The once sexiest job of the 21st century is looking like the least sexy job of 2023.

Employment data in 2022 showed that data science job openings went down 27%, worse than piping hot professions such as software engineers and DEI professionals. Companies are mostly looking to keep headcount stable for the first half of 2023, as pessimistic economic projections drive employers into cutting costs. And this data doesn’t even include the tech layoffs from this year.

Was data science just a low interest rate phenomena?

Companies who’ve gone through recent layoffs grew too much too fast. Likewise, the hype bubble of data science seemed to pop, the casualties of which are mostly just new to the field. The more experienced and fortunate appear to remain entrenched, further fortifying their career moat across from the entry-level chasm.

The data is damning. It doesn’t take a data scientist to tell you that.

The Evolution of Data Science

I’d also be a terrible data scientist if I didn’t tell a compelling spin behind the data.

Data isn’t getting more scarce. On the contrary, it’s still exponentially becoming abundant. The need to process, analyze, and harvest into something valuable is more present than ever. And companies, not just in tech, need skilled practitioners to harness that resource.

But much like a maturing startup shifting its workforce from generalists to specialists, data science has become increasingly specialized. This shift in data science identities is articulated well as Type A (analyst) vs. Type B (builder) data scientists a few years ago. Taking it further, new professions have spawned from the data science tree: data/analytic engineers, machine learning scientists, data-driven product managers, and even data executives to address the increasingly complex data operations of each company. Unicorn data scientists are becoming extinct, mostly working for the smallest and earliest stage companies. They were forged from a bygone time, but they likely won’t be churned out by bootcamps.

In sum: we need data science skills more than ever, but they need not be data scientists in name.

Imagine the data engineer who understands the customer while being able to assess the technical tradeoffs of expedient delivery and future efficiency. A machine learning scientist who happens to have deep domain expertise, able to adeptly design the appropriate experiment to directly address the business hypothesis. The analytically savvy product manager who’s customer-first but has developed the empathy to empower their developers. Each persona is doing data science without being a data scientist.

Data science is often at the intersection between the who, the how and the why. Data scientists have traditionally operated close enough to the money spigots to understand the business problem while being in the trenches enough to know the bounds of the design space. The ability to relay analytical fluency to different tongues, from the business folks to the software engineers, is still highly relevant regardless. It just so happened that data-scientists had a headstart on playing that role.

Behind the buzzwords and fancy algorithms is the need to make better decisions. That’s it. That’s what data science is for. And for as long as data holds sway in those decisions, there will be a need for data science, regardless of whether it comes from a product manager or data engineer.

Concluding Thoughts

The demand for data science will go where data flows. Data science opportunities will manifest itself in mysterious forms, but it won’t look like the point-blank obvious forms of yesteryear.

The most important adaptation for any aspiring or current data scientist is to develop an identity and lean into it. Gather deep domain knowledge. Develop a business intuition. Learn to code. Leverage whatever human capital you have to extract max value.

You either die a data scientist or live long enough to see yourself as a data engineer (or product manager).

The data scientist is dying. Long live data science.

Bonus: Data Scientists and AI

I’d be remiss if I didn’t talk about AI and how that might affect data science. A fair question. It’s like asking how nukes might affect the military back in the 1940’s. Humans still need to be in the loop to serve its purpose and save ourselves from extinctions, lest we ask robots to answer trolley problems.

Or just learn dbt if you’d rather avoid being stuck in the crossfire of the AI ethics debate.

*Sigh*. I almost miss the days when AI was just logistic regression.