First day at Uni

I’d read countless blogs by researchers and writers across the internet, always finding something valuable in them. But I had never written anything myself. Not a single post. Somehow, writing always felt like something for “someday”, until a conversation with a friend changed that.

It was the first lecture of Introduction to Programming at university. I rushed in to find most seats taken; I had arrived about 15 minutes late, my biological clock still adjusting after moving from the countryside to the city. Exhausted after sprinting across campus, I spotted one empty seat beside a guy and slid into it.

After a few exchanges, I thought he seemed friendly enough, though I didn’t think much of it. Minutes later, the teacher, an easy-going gentleman, asked us to introduce ourselves. We had plenty of time, since the first day was mostly about regulations and basic course information. When my seatmate spoke, I discovered we had similar backgrounds: we had both enrolled in this university with awards from national olympiad competitions, mine in physics, his in math.

By some turn of events, we ended up whispering not about code or why we chose this university, but about quantum physics and relativity. Fortunately, he was a nerd like me, so these topics fit a first meeting between friends. If he had been an ordinary guy with no interest in scientific stuff, this would have been the most awkward conversation of my life. Despite his background in mathematics, especially combinatorics and number theory, he knew quite a bit about physics and its prominent figures, citing Einstein, Heisenberg, and Schrödinger.

He was brilliant, a graduate of a top high school in the city, with a mind for maths that both impressed and intimidated me. Yet, we found common ground in nerdy, big ideas. Our interests were like two sides of a coin: he was drawn deeply into the human world, things like religion, society, history, since he had been devouring books like Sapiens and other major works about humanity, belief systems, and imaginary orders. On the other hand, I was pulled toward the abstract and objective aspects of nature. Everything that sounds universal and mysterious, i.e., consciousness, spacetime, and finally, the so-called metaphysics. Metaphysics is one of the biggest pillars of general philosophy, which focuses on looking at the world from a higher perspective, using abstract reasoning to draw out some of the most universal principles from the physical world instead of dissecting each phenomenon individually, hence the name meta-physics.

During those chats, he’d share his wild, often startling theories on society and ethics. Some were so out-of-this-world that I’d joke he needed solid insurance for his thoughts, or society (and all of its traditional prejudices) might confront him on a beautiful evening. One afternoon in the library, chatting about everything to relax after finishing our last examinations of the second semester in our first year of undergraduate, half-amused and half-serious, I told him, “Why don’t you start a blog? These ideas aren’t the kind of things people hear in daily life. Perhaps you’ll find someone who is like-minded with you.” Despite saying it as if my thoughts were nowhere near his, I strongly believed that at that time, I was the one who could understand most of what he thought in the university we were studying.

I didn’t think much of it, but that sentence seemed to light a spark in him. He got distracted for days and began pouring his thoughts into Google Docs. When I asked why he didn’t just build a website already, he said something quite impressive, though still predictable enough: if he focused on building the site first, he’d lose the motivation to write. The ideas had to come first, starting from their infancy and gradually becoming more mature over time before any websites came into being. Actually, I think the real reason is he hadn’t built anything web-related before, so building a website out of the blue without any experience would be challenging (later I learned it’s actually a nightmare for him to use these kinds of modern technologies; “challenging” just doesn’t capture it).

Several months later, he had already written dozens of blogs on his Google Docs. Each one was noted clearly with the published date, updated date, author, citation, and even references to books he had read before. They were separated into different folders serving different styles, i.e., philosophy for serious people who would like to experience some strange views, dummy folders for small thoughts that only appear once in a while and then immediately disappear. Now the tables had turned. He kept nudging me: “When are you starting a blog?” I always refused. It sounded tedious, and I didn’t feel I had anything impressive enough to say. Why would anyone care? At that time, I also didn’t have any specific passion to start a blog. When you blog about something, you at least need something in mind, right? At the time, I was just struggling with all the coursework and exams in class; I didn’t really have time for anything else. Blogging then didn’t seem suitable; perhaps I might look into it another time later.

Two years later

A lot has changed since that conversation. I was no longer a freshman in my first year; I was now a third-year undergraduate, with a clearer mind and a stronger mental model, and, most importantly, I’d figured out what I wanted to do and research.

Surviving dozens of projects across many subjects from my first year through my second year, I realized that my formal coursework hadn’t led me to the technical frontiers I found most compelling. While my school’s curriculum hadn’t yet delved deep into the AI and deep learning topics that sparked my curiosity, I decided to take my own steps rather than relying too much on my university curriculum. For over a year now, starting sometime in the second semester of my second year at university, I’ve been on a self-directed journey down the rabbit holes of reinforcement learning and deep learning foundations. There are a few reasons why deep learning caught my attention at this time:

• First, it uses many more derivatives, integrals, differentiation, and concepts from calculus and continuous, analytical functions and optimizations. Those were what I dealt with a thousand times back when I was competing in physics olympiads, where I had to solve roughly hundreds of them to build my own mental model of how differential equations work.
• Second, it draws inspiration from nature: deep learning borrows the neural architecture inspired by the biological structure of the human brain, while reinforcement learning embraces the idea of trial and error in the learning process of nearly every life form on Earth, including us humans.
• And finally, I just wanted to escape my boring daily life at the time, to break away from the normal workload of school and find something out there that I felt passionate about, where my intrinsic motivation is the single source of power that drives me into this discipline with no brakes and no expectation about the rewards or benefits I can get from it. Just the experience of understanding something and being able to write it down neatly and creatively is as satisfying as finally cracking a hard problem.

After a long period of self-teaching the foundational knowledge of deep learning and reinforcement learning, I found something that, once again, captivated my mind with its beauty and mystery: the generative power of diffusion models. My background is in physics, so anything that deep learning brings from physics to AI to enhance its power is always captivating to me. I still remember learning about the dynamics of particles in dynamical systems: we have heat transfer between particles in the same material (say, a long tube with one cold end and one hot end, where heat gradually transfers from the hot one to the other); the internal friction between layers of fluid flowing through an environment, where layers slide against each other, creating friction that slows some layers and speeds up others; and the final phenomenon, which you might guess from my talk, is the diffusion process. In that process, particles scatter throughout a container or environment, increasing the system’s entropy until the particle distribution is nearly uniform everywhere. Somehow, computer scientists have successfully brought the idea of diffusion processes into generative models, creating state-of-the-art models in image generation, and now they’re extensively used for other generation tasks, including text in natural language processing and trajectories/rewards in robotics learning. In the near future, I might write some blogs about how these fascinating topics from my old friends from the old days of preparing for physics competitions blended their ways into my current pathways to deep learning and give me such excitement.

This process of teaching myself complex topics has taught me something crucial: the best way to solidify your understanding is to articulate it clearly for someone else, especially if we can draw hidden connections between different seemingly unrelated concepts and unify them into a single tree of abstract structures where nearly everything is correlated. The idea of this self-teaching technique is easily found on the internet. There are a few keywords when talking about these techniques:

• Rubber duck debugging: The idea is simple, i.e., explain your code to a rubber duck, articulating every detail clearly enough that a literal duck could understand, and you’ll often spot the bug within minutes. Writing works the same way: teaching an idea to an audience is just explaining it to a rubber duck at scale.
• The Feynman Technique: Named after Richard Feynman, the Nobel-winning physicist famous for his ability to communicate quantum electrodynamics with clarity and humor. The technique is disarmingly simple: explain an idea in its simplest form so that a child could understand it. If you can’t explain it simply, you don’t understand it well enough.

Ultimately, you don’t truly know a concept until you can explain it. That’s the first reason I’m here now.

This blog, then, is my new tool for thinking. It’s a place where I plan to break down the ideas I’m wrestling with, from the gradients of data distributions of generative modelling to the exploration-exploitation trade-off in deep reinforcement learning. Talking about learning, I have quite a strange opinion on this topic: what we strive to learn is what actually drives us to the top and stays with us for the rest of our days. Any knowledge bestowed upon us that we take for granted will never be a true companion and will soon be replaced or abandoned. This observation comes from some of my past experience, and I strongly believe that lifelong learning and self-learning are the best combination for our human race for the next era. I might write a blog presenting my ideas on this opinion later.

Speaking about blog writing, my interests aren’t confined to neural networks and loss functions. I also want to explore a different kind of model here: models of thought. I had a long-standing interest in philosophy, particularly epistemology (how we know what we know, the philosophy of knowledge, mind, wisdom, and language) and metaphysics (the nature of reality, existence, objective and subjective matter, etc.). Sometimes, the questions in AI and the questions in philosophy feel surprisingly adjacent. Their connections feel so natural yet subtle that not many people may realize how these disciplines are tightly connected. What does it mean for a machine to “learn”? What is the nature of the “world model” an agent builds? What actually is consciousness, a concept we talk about as if we knew it, but which is ready to be replaced or redefined at any time? Writing will allow me to express some of my ideas better and surely it would help me relieve some stress I cope with during my studies, both at university and through self-study.

Eventually, my friend’s long journey of persuading me to write a blog turned out to be more successful than he thought. This space is my commitment to the process of learning, thinking, and sharing.

Looking back, that first conversation about quantum physics in a programming lecture set something in motion that neither of us could have predicted. He found his voice in Google Docs, and I’m finding mine here. This blog will be a learning journal, a place for technical deep-dives into deep learning and AI, reflections on philosophy and the nature of knowledge, and occasional explorations of whatever else captures my curiosity. It’s a record of the ideas I’m wrestling with, written not because I have all the answers, but because writing is how I find them.

Dasvidaniya!