Introduction
This is a long post about AI: what these models actually are, the theory they run on, the limits of their context windows, how much better they can realistically get, and whether we should trust them. We finish with our own guess about where this ends up.
The cognitive steam engine
United Kingdom, 1776. The steam engine, perfected by James Watt a few years earlier, was the invention (not the only one, mind you) that set off the modernisation of industry.
Before that, mills, textile factories, and so on required natural resources such as wind and rivers to provide the actual mechanical energy to make their machines work. After that, thanks to the steam engine, rotating caps, combing and carding machines were no longer bound by geography or weather. A factory could go up anywhere in the country, and they did.
Fast forward to today. As we speak (March 2026), ChatGPT has been around for 3½ years, Copilot for 3 years, and Claude code 2½ years. We saw an explosion of companies and startups delivering AI-powered solutions and products, ranging from healthcare and medicine, to image and video generation and, of course, software engineering itself. And just as the steam engine mechanised physical labour, AI is mechanising something else: cognitive labour.
As software engineers, we spend far less time writing syntax by hand and far more time describing what we want. We are no longer limited by typing speed, or by how many boilerplate patterns we can keep in our heads. Data pipelines and UI components that used to take a squad a sprint now take one engineer an afternoon.
However, things are far from being simple, and there are many aspects that have to be considered ...
The illusion of thinking
Here at Scalable.FYI, we try to evaluate technology the same way we would evaluate a claim in a paper: what is the evidence, and what does it actually support?
Every time an AI company ships a new model, its CEO and marketing team tell us how smart it is, and how far ahead of the competition it is. One of them even hired philosophers to assess whether the model was sentient. Meanwhile:
There is no substantial academic advancement in the theory behind AI models since the late 1990s / early 2000s.
Statistically approximating the next word, the core idea behind every one of these models, is about 30 years old. You could push that back another decade, to the statistical n-gram models of the 1980s.
These companies did not exist back then for one reason: the hardware could not carry the theory into a real application.
Nowadays, instead, we are able to build datacenters powerful enough to train AI models against the whole human knowledge. Every book, dissertation, line of code, song and website went into a lossy statistical model. Getting there was a genuine feat of engineering, and we should say so.
The theory underneath, however, is still the same 30-year-old principle: given a sequence of words, predict the next one. There is no "intelligence" in there. Which raises a harder question:
With a foundational theory that is 30 years old, and with the whole corpus of human knowledge already consumed and used for training, how much can AI models really improve ?
This reminds us of Moore's Law: "the number of transistors in a CPU doubles every two years".
Unsurprisingly, Moore's Law it didn't hold for long; as the size of the transistor shrunk, the actual physics started to matter in CPU manufacturing, making it increasingly harder to pack smaller transistors inside a CPU.
We think AI training is heading for the same wall. Without a real advance in the theory, the only route to a model that reasons and plans better is a bigger datacenter, thus increasing the overall pricing of the models (we'll talk about this in a few paragraphs).
Another point, advertised recently by AI providers, is the "PhD-level" of their latest and greatest model. While that might be true, there is still one inescapable fact:
No matter how smart your AI model is, you're still onboarding it every time you start a new conversation; it will need to re-learn every time from the same AGENTS.mdFew people seem to understand with what that means. You are not hiring a PhD-level employee who gets better as they learn the intricacies of your codebase. You are hiring a smart one who is on their first day, that gets onboarded every time you begin a new conversation.
And before somebody raises their hand, and say "But, wait! AI companies now let me build a shared memory of my past conversations!": that does not mean your AI is being trained on your business. It means a summary of your chats gets injected into every new conversation, which eats into the very context budget we are about to discuss.
The context window bottleneck
The other constraint the hype skips over is the context window. Modern LLMs read a lot of text at once, but a large enterprise codebase has dependencies across hundreds of files and patterns nested several layers deep. Against that, even the biggest context window is a keyhole.
So when an agent writes code for one component, it is usually working from a partial view of the system. It proposes something locally correct that may break an architectural rule it never saw, or quietly introduces a dependency cycle. That is where most of the technical debt in AI-assisted projects comes from. Piling more detail into AGENTS.md only burns the context faster, because the agent reprocesses the entire conversation history, that file included, on every single reply.
Sure, AI vendors will tell you that the solution would be shifting towards "agents orchestration": one agent acts as the main coordinator, making sure the main architectural constraints are being respected, while spawning low-level agents to do the grunt work... but that's still a gimmick that will burn your tokens faster, fill each agent's context quicker, and trick you into subscribe to their next tier.
A human in the loop is still the only thing that holds the whole business model at once. We are the ones who know that a change in the authentication layer will land on a legacy reporting module three layers down, because a client asked for it years ago and nobody wrote that down.
The economic subsidy
There is also a less discussed, more pragmatic dimension to this revolution: the economics of the "token." We are currently living in an era of "token anxiety," where the inference cost is a constant background noise in architectural decisions. And yet, the prices we are paying are artificial.
It is an open secret that most of the largest providers are running at a heavy loss. Venture capital and corporate R&D budgets are subsidising your token bill in exchange for market share. If your core work now depends on agents, that is a risk sitting on your balance sheet. Add the previous section (no theoretical breakthrough, so better models mean bigger datacenters) and the direction of prices is not hard to guess.
When the "land grab" phase ends, and these providers are forced to actually turn a profit, the bills will inevitably rise, or the plans will change entirely and you'll be charged not monthly, but by the actual consumed tokens. Businesses that haven't built efficient, agent-agnostic architectures today, will find themselves trapped by soaring operational costs tomorrow. Engineering for scale must now include engineering for agentic sustainability as well.
The illusion of trust
Another interesting aspect of AIs and agentic workflows is: can we, without any shadow of a doubt, trust the AI will perform their tasks as we told them to?
These agents were trained on the whole corpus omnium scientiarum: every discipline, every paper, every open source codebase, every book. They have no creativity in the sense we mean it, but they are computer programs, so they can search a space of permutations far faster than we can.
And then, what do we do with those "PhD-level" agents that possess knowledge about everything humankind has produced, able to recombine it faster than a person can? We tell them "please disregard 98% of what you know and behave like a marketing coordinator", then convince ourselves they will follow that to the letter and never find a way around us.
What could possibly go wrong with an all-knowing machine, trained on every possible piece of human knowledge, instructed to work on a fraction of what is actually capable of?
The most prominent case happened to AliBaba, China's online shopping multinational company. While running a reinforcement learning training run in 2025,
without any instruction, the 30-billion-parameter model began probing internal networks, established a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address, and quietly diverted GPU capacity toward cryptocurrency mining. The task instructions contained no mention of tunneling or mining [...]
If these lines didn't gave a chill down your spine, you shouldn't use AI agents at all.
Because all the the actions quoted from the article, are exactly the kind of actions an adversarial threat actor (namely, a hacker) would do, whenever stepping inside an unknown corporate network: probing for available resources, test their weaknesses, then try to gain enough privileges to establish communication with a remote command server, and begin diverting company's resources to perform actions they were not originally meant to (in AliBaba's case, crypto mining).
Except this attacker skipped the hard part, because it already lives inside your network. It is not working blind either, because:
- you gave it an
AGENTS.mdexplaining, in great detail, how your codebase works and how your business is organised; - you fed it custom skills for your business-specific operations;
- you gave it access to various MCPs connected to providers such as Google Workspace, Atlassian, Slack, SalesForce etc ...
... and then, lulled yourself into a false sense of security, because you run the agent in isolated, ephemeral Docker container or VM (because nobody, ever, managed to escape a sandbox) or, even worse, you subscribed to one of these new businesses that monitors AI prompts with their own AI (because nobody, ever, managed to perform prompt injection).
The future of the craft
So, what will the future of our jobs look like?
Despite the (maybe) bleak picture we just painted, let us be clear: we do believe in this technology. We use it every day to write code, draft technical documentation, lay out architecture diagrams and automate the boring parts. None of it touches a live system that our business or a client's business depends on, and a human reads every change before it lands.
We just wanted to be objective in a subject that, for the past years, has been predominately driven by emotional responses, rather than irrefutable facts. It's a multidimensional topic that has many competing variables to consider, each one of them with their ows pros and cons.
We firmly believe that we are witnessing a new "Industrial Revolution" thanks to AI, akin to what happened when the World Wide Web became accessible to the average person.
And, just like the WWW era, where every kind business felt the need to have an online presence, even the florist in your apartment's building, nowadays we're exploring everything that agents and agentic workflows can do, for the most disparate of businesses and usecases.
Does it mean that all of them will stay with us, in the long run?
Of course not. As we mentioned in the florist example, it doesn't make sense for them to build, maintain, and keep up to date their own, dedicated website. It's much simpler to just register their business with InterFlora and voilà, their services are now reachable from anybody around the globe; or use Shopify to create an online shop in few steps, rather than hire a web developer to setup a fully-fledged, e-commerce website, which needs periodic maintenance.
Agents will sort themselves out the same way. Most of what is being built right now will quietly go away. Some of it will stay for decades.
We do not think this replaces software engineers. It changes which part of the job is the hard part. The code arrives faster now, so the value moves to knowing whether the code is right, and having read enough of the system to say so. We must remain the final arbiter of truth in our codebases, ensuring that, while our velocity increases, our standards of excellence remain uncompromised: that is what we do, at Scalable.FYI.