When I was in college, I always envied the people who decided to NOT do engineering or data science.
They had cool profs. And cool classes.
I had professors whose accents I could barely understand. In a major that should have been called something like “data science”, or at least “decision engineering”, but was called “systems engineering”, which confused the heck out of me four four years. There was not a single class on systems building or networking in that major at all, let me tell you.
I remember, though, one class. The professor was teaching us about basic prediction systems. He called them “forecasters.” He was from Eastern Europe and every time he said the word, he never bothered to define it, and I had to really struggle, for the life of me, to understand why he was talking about the weather.
It was rough. The classes were rough, the professors were rough, and in one class the average grade (I kid you not) was in the 60s.
But the content was so interesting.
Had they presented it better they could have gotten a major drop on the 2010s, the one time in life where being a data scientist was cool.
Predicting stuff
A lot of the classes in this major focused on using data to predict things in the future.
That was always fascinating to me. I don’t know why. It’s just cool. It continues to be cool.
I remember that one of the first times I was really awed was when we were analyzing data from NASA’s shuttle launches and the professors helped us find an association between O-ring integrity and shuttle stability.
Really fascinating stuff, to me at least.
When I graduated, I didn’t do any data science for about 7 years. It was all pure and simple software engineering. Then I eventually started using data for a few things related to work.
As of today, the summer of 2026, I finally had the chance to get back into it.
It had been a minute (really, about 4 years) since my last real, professional data science project. I used to do this every day at Apteo. Our core product was a machine learning model that could predict what products each individual person was most likely to buy next from an online store.
When we wound down that company, I didn’t have the need to do any more data science projects until about 6 weeks ago.
It’s been great.
Stocks and inventory
I’ve been working on one personal project and one consulting project in which I’m using machine learning to forecast things into the future.
The first project is a rehash of Apteo’s first product - a stock performance predictor. I’m taking loads of data about stocks, including text data, and using it to create a system that can forecast stock performance across various time frames (1 month out, 3 months out, all the way up to 5 years out).
The second is a demand forecasting system for a large retailer. I’m forecasting what the demand levels are likely to be for each individual variant of their catalog, which will, hopefully, help them optimize their product acquisition process going forward.
Each project has its own nuances, but each is very similar in its own right, and the process I’m taking for each project is very different than the process I would have taken 4 years ago.
Today, life is all about telling an AI system what you’re trying to do and course correcting it as it miserably fails its way to success.
Whereas in the past, I would have meticulously set up a data structure, a feature store, done all the engineering, and set up a tournament system to do a grid search across all featuresets and permutations, now I just tell an AI what I’m trying to do and it sets it all up for me.
It runs the systems on my machine, comes up with the champions, and keeps me posted. Inevitably, it makes an error or it assumes something that it shouldn’t, but I make sure to keep a tight eye on how it’s performing, and course correct it from there.
I eventually get an output that includes error rates, some sort of backtesting, and a list of forward looking numbers that I then either use myself (to make stock trades) or build a dashboard around for the team (in the case of inventory).
In a certain sense, some of the joy of the data science has now been removed. It was always interesting to be able to create the actual code that could transform and model data. All of that is now automated.
But in a much larger sense, there is a lot of newfound enjoyment at the creativity that comes from being able to conceptualize a new feature or a new combination of models to try and then going off and telling someone else (or something else in this case) to do it.
It gives me a lot of freedom to be able to ensure that we get the best possible results we can without getting bogged down in code. With both the stock forecaster and the inventory forecaster, I’ve already tried about 200 different combinations of models, features, and model combinations to find the best possible setup. All I have to do is keep pushing the system to try new and more creative things.
That being said, a lot of the fundamentals haven’t changed:
Data engineering is still the foundation
Getting data from where it is to where you need it can require a lot of new systems and automated processes, and, if you’re a big company, a team of engineers (us small guys just use our AI agents to do it all for us). There are data failures and pipeline issues and uptime concerns and data modeling issues and everything that goes into all the devops requirements of a full-fledged data engineering system. It’s ugly and it’s hard, and it’s also the thing that everything else is built on and you absolutely need to get it right.
Data is still messy
It’s hard to get data from one place to another, and it’s even harder to know if a number in a table means the same thing you think it does. It’s important to clean up the data and understand how it’s structured whenever you can.
In the case of the stock price forecaster, I had complete control over that. I knew what the data was and how it was structured and could come up with a clean and comprehensive feature store for it (working on getting there for the inventory forecaster).
Very few systems are fully predictable down to the individual example
Individual instances of something that needs to be predicted have a lot of error. A stock that’s forecasted to go up by 100% can actually go down by 50%. An individual shirt size that’s forecasted to sell 2 units out of a channel might in fact sell 20. It’s important to understand that and it’s important to plan around that.
One of the best ways to plan around that is to understand that as you aggregate, zoom out, and rollup, errors tend to go down. You could invest in 10, or 100 stocks, rather than 1, and your overall error rate will trend more towards what you saw in the backtest. You might buy 20 different families of a jacket rather than 1 and your total purchase rates will get much closer to what you expected than if you only had one.
While most phenomena have patterns that can be discovered and leveraged, those patterns may, in fact, be weak, which means you need to take this into account when you use them to make decisions.
In many ways, data science has both changed a lot and almost not at all in the past four years. AI, like everything else, has made the tedious, repetitive parts of it easy(ier). But, like with software development, AI is not good enough (yet) to do a lot of the strategic planning and oversight work that must be put in place for a large, complex project. Today, it’s possible to move 10x faster than we could even 2-3 years ago, but the fundamentals are still important to keep in mind and keep in place.
Ok, that’s all for today, two articles in one week! I guess I like writing.
Thanks for making it this far. Let me know if I can ever help with anything.
-Shanif