Introduction

When I was in high school, I gravitated toward physics, eventually enrolling in a specialized program at a prominent school in my hometown. I competed in the provincial olympiad, the 30/4 Traditional Olympiad at Le Hong Phong High School, and the Vietnam National Physics Olympiad (VPhO).

Physics taught me how general principles govern behavior, i.e., how a ball falls given its velocity and angle, how acceleration affects motion. More importantly, it built mental models for problem-solving that extend beyond any single discipline. What captivated me most was the continuity of calculus: tracking change through derivatives, with differential equations as the language of a dynamic world.

When I encountered maths in university, things became discrete, i.e., Karnaugh maps, Boolean algebra, everything forced into true or false. Honestly, I had neither interest nor knack for these structures after spending years with physics, where everything was continuous.

I struggled through my first semesters, learning things I didn’t care about. Only when I heard that AI and deep learning are built on calculus and optimization did I begin to feel I’d found my direction, something I had longed for since my early undergrad years.

I started learning machine learning, i.e., bias-variance tradeoff, loss functions, first and second-order optimizations. Deep learning was the most captivating part: moving from decision trees to neural networks with thousands of layers. Like everyone else, my first deep learning project was training on a curated dataset, Supervised Learning (SL).

But I quickly felt bored with this paradigm. Learning from data is straightforward when you have good examples to mimic, but preparing datasets consumes unfathomable time and effort. I wanted something more universal.

That’s when I discovered Reinforcement Learning (RL), a paradigm where an agent trains itself through interaction with its environment.

In this blog, I want to share why I found RL much more fascinating than traditional SL and why I believe it will be the next cornerstone of AI frontiers, the paradigm that some researchers believe could lead us toward AGI.

Fitting a model vs Training an agent

This section compares these two training paradigms based on my intuition and experience up to the time of writing. Let’s start with supervised learning.

Fitting a model

In supervised learning, we fit a model to a fixed training dataset and evaluate it on a held-out test set. The pipeline looks roughly like this:

• First, gather enough data. Deep learning with billion-parameter models demands massive datasets, either collected from the real world, synthesized, or generated by another model.
• Second, explore the data. Understanding your dataset’s traits, especially for temporal or financial data, directly impacts model selection. Skip this step and you’re choosing a model blind.
• Third, make assumptions about the data distribution and pick a statistical model accordingly. The No Free Lunch Theorem says no model works for every problem, so you must tailor your approach to the data.
• Fourth, fit the model. Loss functions, optimizers, learning rate schedules, feature engineering, this is the step people usually mean when they talk about “training,” though it’s no more important than the steps before it.
• Finally, evaluate on the test set. Watch for underfitting, overfitting, and distribution mismatch between test and real-world data.

Now let’s see how RL differs from this pipeline.

Training an agent

Now to the superstar in my heart, Reinforcement Learning. Here, data and model engage in a continuous, never-ending interaction.

• Data Gathering: in SL, we collect existing data from the internet, i.e., articles, images, whatever’s already there. For RL, the story is entirely different. Since our agent learns through interaction with its environment, data can only come from that specific environment. We cannot use data from an energy management simulation to train a traffic light controller.

  We must generate every data point ourselves by having the agent act in its target environment, which drives us to parallelize data collection as much as possible. For example, to train Bob (an agent from AI Warehouse), the author ran 200 simultaneous experience collection scenarios.

• Non-stationarity: unlike SL, where the data distribution is fixed, RL data depends on the agent’s own policy, which changes as the agent learns. I’ll cover this in more detail in the next section.

Reinforcement learning as a natural learning process

When we talk about something natural, we usually mean something that emerges without human intervention. Evolution gives us thousands of hints about what is natural: a dog knows how to bark from birth; a butterfly knows how to fly the moment it escapes its cocoon. Every species shares the same reaction to danger: they run away. Running from what we fear is one of the most natural phenomena we know.

So what about learning? If running is the natural response to danger, what would natural learning look like? Supervised learning gives us an incredibly powerful framework, i.e., train on a dataset that describes what we want the model to learn. This paradigm has long been a cornerstone of AI because of its statistical foundations: maximum likelihood estimation, maximum a posteriori estimation, divergence minimization, concepts established decades ago with immense mathematical rigor.

But what if we don’t have data at all? What if our model needs to interact with the very environment we want to deploy it in, and learn from scratch? This is where RL comes in. The most interesting property of RL is that we don’t need to prepare any dataset beforehand. Our agent interacts directly with its environment and learns from that experience. Let’s illustrate with a simple example: how a child discovers that fire is dangerous, without any RL jargon. Let’s call our curious child Bob.

• Bob sees the fire is sparkling at the corner of the house and has no clue what that brightness is.

• Bob tries to approach the fire out of pure curiosity and wants to touch it to feel what it is like.

• The heat from the fire immediately inflicts insufferable pain on Bob’s hand. This is the first time Bob experiences something so harsh in his young life, he’s barely 2 or 3 years old.

• Bob cannot stop himself from panicking and crying out loud from immense pain. He immediately deems fire something that would cause him such pain when he touches it.

• From this moment onwards, fire is deemed dangerous by Bob, and he never dares to come close to it again.

The lesson: we don’t need a dataset to teach Bob that fire is dangerous. We just watch what he does and how he reacts to the consequences. (Of course, we’d ensure his safety—not everything gives us a second chance.)

Learning through trial and error is the most natural solution when an agent must figure out how a system works. The same applies to riding a bike, using a game controller, or fixing a leak. Thousands of real-world skills can be unlocked this way. But this power comes with a dark side: instability, which we’ll look at next.

Instability as a Service

In cloud computing we have Software as a Service, Infrastructure as a Service, Platform as a Service. In RL, the defining feature is something I’d call instability as a service: it’s the heart of the field’s complexity.

Overfitting & underfitting

In machine learning, we often talk about overfitting and underfitting. We want our model to fit the true distribution of the world, but since we can’t access that, we fit to the data distribution as a proxy.

In SL, the data distribution cannot be affected by the model parameters at any time. The information flow is one-way: from the data to the model. The model updates from the data, not the other way around. Not so in RL. Here, the data doesn’t stand still; it moves along with the model parameters!

Non-stationary data

Here’s the magic of RL: the data used to train the agent is generated by the agent itself. At the start, with random weights, the model explores all actions equally, data is roughly uniform across states and actions. But as the agent learns through Bellman equations or other frameworks, it begins preferring some actions over others.

Its policy distribution shifts, and so does the data distribution. States following preferred actions accumulate probability mass while others atrophy. The data distribution at step 1000 looks nothing like it did at step 1. Since this non-stationarity is intrinsic to RL, all algorithms must contend with it, leading to the fundamental divide between on-policy and off-policy methods.

On-policy and off-policy

Two major representatives illustrate the on-policy vs off-policy divide: Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC). On-policy (PPO): uses only the current policy to generate action probabilities. After each update, old data must be discarded; it reflects a stale distribution. This means continual data loss. When data collection is expensive (robotics, for example), that’s a significant drawback.

Off-policy (SAC): retains all past data throughout training. This solves the data waste problem but introduces a challenge: old data belongs to old distributions. We reduce this bias by sampling actions from a different distribution (say, $\epsilon$-greedy) and using Importance Sampling to correct the mismatch, re-weighting samples so their expectation matches the current policy distribution.

In summary, reinforcement learning introduces new challenges but also new opportunities for us to think about what learning actually is. Depending on the situation at hand, we might prefer on-policy approach over the other or vice versa.

Actor-Critic and the push-pull of RL

Unlike SL, where we minimize a fixed loss function against ground truth labels, RL optimizes expected cumulative reward, a moving target that depends on the agent’s own choices. Since no ground truth values exist for the optimal reward an agent should expect from a given state, the problem becomes circular: good actions depend on accurate state values, and accurate state values depend on seeing good actions.

This is where the Actor-Critic framework comes in.

The Actor decides which action to take given the current state. The Critic estimates the value of that state, i.e., how much reward the agent can expect from it. The two train together: the Actor improves by following the Critic’s guidance, and the Critic improves by observing actual rewards.

Errors in either component compound through bootstrapping, since state values propagate to neighboring states. Monte Carlo methods can reduce this bias by sampling complete trajectories, though at higher variance.

One interesting hybrid is Imitation Learning, notably Behavioral Cloning, where an agent pre-trains on expert demonstrations before exploring on its own. It’s a supervised warm-start for an RL problem, borrowing the stability of fixed targets before graduating to the dynamic uncertainty of true RL.

Challenges

All the fascinating properties of RL come with frontier-level challenges. Here are two of the most important ones.

Credit Assignment

The Credit Assignment Problem is perhaps RL’s most famous fundamental challenge, and I believe no mature mathematical framework fully solves it yet. Imagine training a robot to walk. It performs 100 joint movements and then falls over. Was it the last move at step 100 that caused the fall, or a poor decision at step 20? If the robot succeeds, which actions in its trajectory actually contributed? Determining the real factors and assigning them proper credit is the heart of this problem.

The problem has two forms:

• Temporal assignment: which past action in a long trajectory caused a future result?
• Spatial assignment (multi-agent): which agent in a collaborative/competitive system caused a particular outcome? When both spatial and temporal dimensions interact, it becomes a much harder space-time assignment problem.

Exploration-Exploitation trade-off

The Exploration-Exploitation trade-off is another frontier challenge. In SL, the model fits the data it’s given. In RL, the agent faces a dilemma: stick with the best-known action for guaranteed reward (Exploitation), or try something random to discover a potentially better path (Exploration)?

We often use a simple strategy like $\epsilon$-greedy to handle this:

This reminds me of thermal noise in physics, sometimes you need a bit of heat or randomness in the system to jump out of a local trap and find the true global optimum. For someone who likes the smooth shapes and continuous representations of physics, watching an agent navigate these high-dimensional policy spaces and solve these hurdles is much more satisfying than just fitting a curve to a static cloud of points.

This is the end of my first technical post. If you’re interested in RL and diffusion models, feel free to reach out via my email or GitHub. Below are some resources I found valuable on my learning journey.

Resources

• UC Berkeley BAIR Lab , A gold mine for robotics and RL, with prominent names like Sergey Levine.
• Lilian Weng’s Blog (Lil’Log) , Expert summaries of massive research areas. Lilian was at OpenAI and now works at Thinking Machines Lab alongside John Schulman (the creator of TRPO and PPO).

Endings

This post reflects my personal fascination with reinforcement learning, not as a rejection of supervised learning, but as an argument that it’s not the final answer. Learning is more complex than optimizing parameters against static ground truth. That’s just one form of it.

RL has seen remarkable progress: from DeepBlue to AlphaGo, AlphaFold, and AlphaTensor. These milestones hint at what’s possible when we move beyond fitting curves to static datasets and embrace the messy, dynamic reality of agents learning through interaction. The fact that this discipline speaks the language of calculus and dynamical systems, the same language I fell in love with through physics, makes the journey feel like coming home.

Perhaps one day I’ll contribute to these advancements myself. Until then, I’ll keep learning, writing, and sharing what I discover along the way.