You install GitHub Copilot, Codex, Cursor, Claude. You turn on “privacy mode” and even go into settings and disable “training on my data”.

So, your code is secret…right?

Not entirely, the digital walls of your private repository may be more permeable than they seem.

The Illusion of Control

GitHub recently mailed all its users regarding a change in its data usage policy to collect interaction data from all users – “including inputs, outputs, code snippets, and associated context” to train its models (source). The most jarring part is that this is enabled by default unless you explicitly opt out.

On its face, that doesn’t sound so bad. If you support AI, why wouldn’t you want to contribute to improving it?

But that’s not really the problem.

The problem isn’t contribution. It’s consent.

Because voluntary participation in the construction of better AI is radically different from being signed up to do so without clear awareness, and the “data” that might feed into such a system could include proprietary code, sensitive logic or work that was never intended to leave your own box.

Privacy Settings …or a Maze?

Cursor has privacy mode.

Claude has opt-outs.

Somewhere in your account, Copilot has settings.

Codex has 3 separate settings in 3 separate toggles across 3 separate menus that don’t talk to each other(source source2).

(Almost like it was designed to be confusing.)

But are these the solution to protecting our data?

There’s more nuance to this question’s answer than a simple yes or no. To understand how safe our data really is let’s follow the journey of your code.

Where does your data go?

One of the most overlooked aspects of modern AI tooling is the slick “pre-read”. Before you even begin a session, your AI assistant is already taking a first pass at your code. It isn’t just passively waiting. It’s actively processing your project in the background. Most modern AI coding tools operate as Retrieval-Augmented Generation (RAG) systems(rag-pipeline), meaning they don’t ‘know’ your code, but dynamically fetch relevant parts of it from a database every time you ask a question.

The second you open your project in a tool like Cursor, a background process starts monitoring your entire directory. It doesn’t even wait for a prompt before it’s already reading your file tree, beginning to index and chunk your codebase into tiny bits of data. Tools often use syntax-aware parsers to break code into meaningful chunks like functions, classes, and logical blocks before embedding them. These chunks, which on their own have no meaning then get sent to an embedding model to convert the text of their codebase into vectors (source) which are numerical representations of data that along with metadata get stored in a vector database.

The plaintext of your code is not permanently stored (in privacy mode) but the embeddings and metadata are. The vector database essentially forms a map of your private repository on someone else’s servers (rag-systems).

You Didn’t Just Send a Prompt

When you type a prompt into this LLM now, the prompt too gets converted into a vector using the same embedding model so that it can be mathematically compared to the other vectors already in the vector DB. It is compared with the rest of the database to find the mathematically closest chunks(semantic searching using cosine similarity).

The tool then combines:

• Your original prompt

• Retrieved code chunks

• The file you currently have open

• Your cursor position

• System prompts

• Git history and conversation history(sometimes)

to build the final augmented prompt that gets sent to the LLM.

The model processes this and streams the response back into the tool’s backend before reaching your IDE.

There’s More Than One Code Spy

In parallel to everything above, a telemetry event fires to the tool’s analytics infrastructure separate from the inference pipeline and includes lots of metadata. (Telemetry simply refers to automatic data collection about how you use a system like what you click, type, accept, or change.)

If you edit a file based on the response:

• The file watcher detects the change

• Re-chunks the file

• Re-embeds it

• Updates the vector database

This creates a continuous, real-time feedback loop.

The Full Data Flow

So to summarize data flows through:

Your local codebase → file ingester → chunking engine → embedding model → vector database → LLM → response back in your IDE.

At no point, in most common implementations of this pipeline, is your data purely local. All this data transmission occurs while you were never asked if you wanted your code chunked, never informed what parts of your files became embeddings and no receipts showing what was transmitted from where or to whom.

Moreover, if you haven’t disabled training on your data, telemetry goes onto these LLM’s analytical infrastructures like Anthropic(AWS)(source), OpenAI(Microsoft Azure) and even potentially Datadog and MongoDB for logging. This stored data is then used to train and improve these company’s models.

Across these steps,

• A Claude prompt flows through the servers of Anthropic, AWS and Google Cloud.

• A GitHub Copilot prompt touches GitHub, Microsoft Azure, OpenAI and AWS/Anthropic or Google Cloud servers depending on cloud selection(source).

• Cursor passes data through Cursor, AWS, Cloudflare, Microsoft Azure, Turbopuffer(Google Cloud), OpenAI, Anthropic, Google Vertex AI, Fireworks AI, Baseten, Xai and optionally Exa/Serp API when web search is involved(source).

Cursor’s own documentation states that it stores embeddings from user’s indexed codebases along with metadata like file names and hashes(source). GitHub Copilot ups the ante and explicitly mentions indexing entire codebases to build deeper understanding and fine-tuning model behavior.

Privacy Mode Isn’t Really.. Private

Coding models may not train on your private repositories at rest in privacy mode but while they are still interacting with it i.e via prompting code, accepting suggestions, rejecting suggestions, sending a thumbs up or thumbs down feedback- data is still collected and processed. Thus, user code may be in “privacy mode” at static storage but it is still being accessed by these companies during usage.

But it doesn’t stop:

• Code from being processed

• Embeddings from being created

• Requests from being sent to cloud APIs

This is largely because these steps are required for the tools to function effectively.

There are actually two separate data flows:

• The service pipeline (to run the tool)

• The training pipeline (to improve models)

Opting out only affects the second.

This isn’t just about privacy, AI-assisted code is also bringing about thousands of new security vulnerabilities every month. As of June 2025, AI-generated code was introducing over 10,000 new security findings per month, a 10× increase from December 2024 (source). What was meant to improve developer convenience is now, increasingly, expanding the potential attack surface(security-exploits).

Security Implications

This isn’t just about privacy.

AI-assisted code is also introducing new security risks (source).

As of June 2025, AI-generated code was associated with over 10,000 new security findings per month, roughly a 10× increase from late 2024(source).

What was meant to improve developer productivity is also expanding the potential attack surface in some cases.

So What Can We Actually Do?

So now that you and I know, our data can never truly be “private”, what do we do about it?

There is no perfect way out, only trade-offs because the same thing that makes these tools so helpful is the same thing that requires your code to leave your machine- context.

Some options include:

Option 1: Maximum Privacy

“Code like it’s 2015”

No AI assistants, no prompts and no risks, but also no autocomplete, debugging or speed.

But this is safety at the cost of practicality.

Option 2: Local-Only Models

So the next most efficient way out would be to use tools that run completely locally and avoiding third party APIs entirely but this comes at the cost of lesser efficient and heavier local models and a higher token cost.

Option 3: Practical Middle Ground

If option 2 too is too much of a sacrifice, the next best approach would be using Privacy Mode, opting out of training, using temporary chats and avoiding feedback buttons. On GitHub Copilot the privacy option is under Settings → Copilot → Policies.

This helps but only partially (as we discussed above).

The Bigger Shift

So no, the private repo is not simply a myth but it’s also not what we think it is anymore. A private repo today is not as private as a private repo from 2018.

The game has changed. The lines between proprietary code and AI training data have blurred significantly. If you're a developer today, you need to realize that privacy in the age of AI isn't about where your code lives on a server but also what happens the moment you interact with it.

The Uncomfortable Truth

All programming models today were trained on millions of lines of publicly available GitHub code (source). This didn’t start with copilot settings or privacy toggles but in fact, the entire AI industry was built first, consent came later. “Public code” became something like a confused misnomer because human programmers believed public meant open source for humans, but big tech interpreted “public” as an all-you-can-eat buffet for machines.

“Shutting the doors at this point won't change the fact that the AI industry is built on data gathered without asking for a strong indicator of enthusiastic consent. “(The Register)

String A:  ABABABABABABABABABABABABABABAB
String B:  4c1j5b2p0cv4w1x8rx2y39umgw5q
  

Both are 28 characters. Both take up the same space on a hard drive. By most conventional measures, they're equivalent. But intuitively, you already know they're not. String A has a pattern. You could describe it in one sentence: "AB repeated 14 times." That description is far shorter than the string itself.

String B? You’d essentially have to write it out directly - there’s no obvious shortcut. The shortest description of that string is the string.

The Core Idea, Stated Plainly

Formal Definition:

Let U be a fixed universal Turing machine(it is the "Universal Computer" that Kolmogorov's theory relies on). The Kolmogorov Complexity of a string x is:

Where:

• p is a program (binary string),

• ∣p∣ is its length,

• U(p)=x means the program outputs x and halts.

Why the Machine Choice Doesn’t Matter Much

This means even if we change the underlying computer model, the complexity only changes by a fixed constant. So we can treat Kolmogorov Complexity as an inherent property of the string.

The Kolmogorov Complexity of an object: a string, a number, an image, any piece of information is defined as the length of the shortest computer program that outputs that object and then halts. If we call the object x and the complexity K(x):

Intuition, not actual syntax 

K("ABABABABABABABABABABABABABABAB")
  shortest program: print("AB" * 14)   approx.15 chars
     K(x) is LOW and there's structure here

K("4c1j5b2p0cv4w1x8rx2y39umgw5q")
  shortest program: print("4c1j5b2p0cv4w1x8rx2y39umgw5q")
     K(x) approx. length of x itself
     K(x) is HIGH and this is randomness

Low complexity indicates a simple pattern or explanation. High complexity means the string is its own shortest description. In information theory, a string with complexity equal to its length is called incompressible, which is the formal definition of random.

Basic Upper Bound

K(x) ≤ |x| +c

At worst, a program can simply print the string directly, so the complexity can never be much larger than the string itself.

The Key Shift

Before Kolmogorov, randomness was defined by process - a sequence was random if it was produced by a fair coin flip. But that means HHHHHHHHHHHH is technically "possible" under a fair coin. It just has low probability.

Kolmogorov gives us a structural definition instead: a sequence is random if no program can describe it shorter than the sequence itself. It doesn't matter how it was produced. What matters is what it is.

A Few Real Examples

The π example is interesting. Its digits appear random and pass randomness tests, yet π has low Kolmogorov Complexity since a simple algorithm can generate it. This illustrates the difference between statistical and Kolmogorov randomness; something can seem patternless but still have a simple structure.

A Fundamental Limitation: Uncomputability

Here's where it stops being a neat conceptual tool and becomes something uncomfortable.

Kolmogorov Complexity is uncomputable.

Connection to the Halting Problem

The uncomputability of Kolmogorov Complexity is closely tied to the Halting Problem.

The Halting Problem asks:
Given a program and an input, can we determine whether the program will eventually stop or run forever? Alan Turing proved that no algorithm can solve this problem for all possible programs. It is proven, mathematically and fundamentally, to be impossible to compute in general.

Suppose we had an algorithm that computes K(x). Then we could:

1. Enumerate all programs of length ≤ n

2. Run them and observe outputs

3. Identify the shortest program that produces a given string

But this process requires knowing which programs halt. If we could compute Kolmogorov Complexity exactly, we would effectively solve the Halting Problem -which is impossible.

There lies the contradiction: the program is L characters long but claims to output a string with a shortest description longer than L. This is impossible since the program itself describes the string in L characters.

This resembles Berry's Paradox in algorithm form: "the smallest positive integer not definable in under thirteen words"-yet it's defined in twelve. It's about self-reference causing contradictions. Any system capable of fully measuring Kolmogorov Complexity ultimately contradicts itself.

"We can compute upper bounds on complexity. We can never verify the true minimum. Every compression is an approximation of something fundamentally out of reach."

~ Paraphrasing Li & Vitányi, An Introduction to Kolmogorov Complexity

What We Can Actually Do

Zip a file-the compressed size provides an upper bound on its Kolmogorov Complexity. Use a better algorithm for a tighter bound. But the true minimum? We can only approach it, never confirm it. Every compression algorithm is an attempt at something it can never fully achieve.

Incompressibility Theorem

K(x) ≥ n;

(for most strings of length n)

There are far fewer short programs than long strings. So most strings cannot be compressed - in fact, most strings are essentially random.

Where It Shows Up in the Real World

Biology - Your Genome Is a Compressed Program

The human genome encodes a 37-trillion-cell organism in about 750 megabytes, less than a standard HD movie. It describes the entire human body, from development to immune responses and neurological architecture, in a compact file.

Researchers in computational biology apply Kolmogorov Complexity to compare species' genomes, track evolution, and explore why diverse organisms share similar genetic complexity. By questioning "How compressible is this genome?" we gain insights into life.

Machine Learning - Compression as Understanding

One way to interpret learning is that a model discovers a shorter description of the data. Instead of memorizing every example, it captures patterns that allow it to represent the data more efficiently.
This idea is the basis of the Minimum Description Length (MDL) principle-choose the hypothesis that makes the total length of the model and the encoded data as short as possible. It's like a mathematical version of Occam's Razor.

Anomaly Detection - Finding What Doesn't Compress

In cybersecurity and network monitoring, normal traffic has structure - it compresses. Malicious or anomalous traffic often doesn't fit learned patterns and therefore compresses poorly relative to baseline. Systems that flag "this data is harder to describe than expected" are doing Kolmogorov reasoning in practice, even without calling it that. The idea leaks into engineering even when nobody's tracking it back to its source.

Physics - The Universe Has a Complexity Score

If you think physics can be fully calculated and the universe follows specific rules, then there should be a shortest program that explains the universe's entire history. Finding a Theory of Everything is like looking for the simplest way to describe all of physical reality.

And here's the sting: we can never verify we've found it. A simpler set of laws might produce identical observations. The uncomputability result means we'd have no way to prove our theory is actually the shortest. Every Grand Unified Theory is, formally speaking, an upper bound pretending to be an answer.

AIXI - What Happens When You Build AI Out of This

AIXI assigns higher weight to simpler environments:

Shorter programs (simpler explanations) are given exponentially higher importance, reflecting a formal version of Occam’s Razor.

The answer he arrived at was AIXI - a theoretical model of a perfect AI agent. Its core loop:

1 Observe the world Take in all sensory data and history of interactions so far.

2 Build the simplest model consistent with observations Using Kolmogorov Complexity to find the shortest program that explains everything seen so far.

3 Act to get the best future rewards based on that model
Pick actions that lead to the best results using the simplest model of the world.

4 Update. Repeat.
Each new observation improves the model. Keep compressing. Keep getting better.

AIXI can be mathematically proven to be optimal - no other agent can consistently outperform it across all possible environments. It is the formal theoretical ceiling of intelligence itself.

It cannot be built due to Step 2 requiring the computation of Kolmogorov Complexity, which is uncomputable. AIXI is a provably optimal agent that cannot exist. It's the most useful concept in AI theory that will never run on any machine.

Why AIXI Still Matters Despite Being Uncomputable

AIXI defines the ceiling. Every real AI system — every large language model, every reinforcement learning agent, every decision tree — is a computable approximation of something that, in its perfect form, is grounded in Kolmogorov's idea.

When researchers design smarter AI, they are in a precise sense trying to close the gap to AIXI while staying within the bounds of what a computer can actually execute. The uncomputable ideal isn't a dead end. It's a compass pointing at something we can keep getting closer to, even if we can never fully arrive.

The Big Debate - Is Compression the Same as Understanding?

This is where the idea becomes genuinely controversial — and where serious researchers take hard, opposing positions.

If intelligence is compression, and compression is what our best AI systems do - finding compact representations, predicting structure, generalising from patterns — then maybe a model that compresses language well enough is, in some meaningful sense, actually understanding it. The debate:

The Compression Camp

Understanding is finding the shortest description. To understand something is to see the pattern beneath the noise. GPT-4 "understands" English because it has found an extraordinarily compact representation of its statistical structure. Intelligence is what good compression looks like from the outside.

The Meaning Camp

A ZIP file compresses a novel but understands nothing. Kolmogorov Complexity measures structural regularity, yet meaning, reference, and intentionality are entirely different. While compression might be necessary for intelligence, it is far from sufficient. The real challenge lies in what the compression represents.

No side has prevailed. Schmidhuber, a key figure in deep learning, has long claimed that curiosity, creativity, and intelligence all come down to seeking compression. The other side thinks this view oversimplifies and misses the main point. The debate is a mix of computer science, philosophy of mind, and AI alignment, and it remains unresolved.

There's a final implication of all this that rarely makes it into the textbooks, but probably should.

Science is about finding simpler ways to describe the world. Every theory, model, and equation tries to make observations shorter and more general. Newton's laws sum up the motion of all objects in three sentences. Maxwell's equations cover classical electromagnetism in four. General relativity explains spacetime curvature with one compact equation. This is Kolmogorov thinking. Science is compression in action.

But here's what the uncomputability result tells us that we tend to quietly ignore: we can never know if any of our theories are actually the shortest description. A simpler set of equations might produce every observation we've ever made. We would have no way to prove it doesn't exist. Every Theory of Everything is, mathematically speaking, an upper bound on the complexity of the universe - not a proof that we've reached the minimum.

We have Occam's Razor - prefer the simpler explanation. We also have a mathematical proof that we can never confirm which explanation is truly simplest. Kolmogorov gives you the razor and removes the certainty of knowing when to stop shaving.

The shortest program that outputs the universe exists - in principle.
We just can never prove we've found it.

Every theory is an upper bound.
Every model is an approximation of something unreachable.
Every answer is a best guess at a minimum we can approach but never touch.

Which is either the most humbling thing in the history of human thought -
or the most compelling reason to keep going.

Probably both

Further reading: Li & Vitányi - An Introduction to Kolmogorov Complexity and Its Applications · Marcus Hutter's AIXI paper · Schmidhuber on compression and creativity · Chaitin on algorithmic information theory

That’s not a movie. That’s a 2023 paper in Springer Nature( a German-British academic publishing company). And it raises a question nobody in the press release bothered to ask: her brain signals were being transmitted wirelessly, processed by external software, and stored on servers. So what happens when the most intimate data that exists of your actual neural activity becomes just another thing that can be breached?

We spent decades arguing about whether your phone was listening to you. We never thought to ask what happens when the device is inside your skull.

What a BCI Actually Does

A Brain-Computer Interface reads your brain's electrical signals and translates them into something a machine can act on. Your neurons fire in patterns. The BCI captures those patterns, filters out noise, runs them through a machine learning model, and converts the output into a command: move a cursor, type a word, control a robotic arm.

There are two architectures, and the difference matters enormously, both for capability and for security.

Non-invasive BCIs use electrodes placed on your scalp. These are the EEG headsets you've seen in documentaries, increasingly on retail shelves. Easy to put on, easy to take off. The tradeoff here is signal clarity, you're reading brain activity through skin and bone, so the resolution is limited. Consumer devices from companies like Emotiv, Muse, and Neurosity work this way.

On the other hand, Invasive BCIs implant microelectrode arrays directly into brain tissue. Much higher fidelity. Much higher stakes. Neuralink’s device works this way. So does the system used in the BrainGate clinical trials that produced Pat Bennett’s results.

(Neuralink is a neurotechnology company founded by Elon Musk that develops implantable Brain-Computer Interface (BCIs) to connect human brains directly to computers.)

BrainGate implanted four microelectrode arrays in the region of Bennett's brain responsible for speech. Within 30 minutes of activation, in a separate trial published in the New England Journal of Medicine, the system demonstrated 99.6% accuracy on a 50-word vocabulary. Bennett's own results achieved 62 words per minute which is more than three times faster than any previous BCI that demonstrated a 9.1% word error rate on a 50-word vocabulary and 23.8% on a 125,000-word vocabulary. She used the system at home. Unsupervised. Via a wireless connection.

That last part is the part nobody talks about.

The Threat Landscape

The phrase “hacking a brain” sounds cinematic. Let’s be precise about what it actually means, because the reality is both more mundane and more alarming than the movies suggest.

Signal interception Most modern BCIs transmit data wirelessly because the alternative is a cable permanently protruding from someone's head, which creates its own obvious problems. Wireless transmission solves that. It also introduces a new one: anything broadcast over radio can, in principle, be received by anyone within range with the right equipment.

Neural signals are not like passwords. A stolen password reveals what you typed. Intercepted neural data can reveal things you didn't consciously express such as stress responses, emotional states, subconscious reactions to faces or images or words. A previously conducted study demonstrated that EEG data alone could be used to infer a user's PIN with meaningful accuracy. Another research group showed that neural signals recorded during passive viewing could be used to reconstruct, in rough form, what someone was looking at.

The data being transmitted isn't just "she wanted to type the letter A." In high-fidelity invasive systems, it's a continuous stream of electrical activity from inside the brain which is definitely far richer, and far more revealing, than anything the user explicitly intended to send.

Firmware and parameter tampering Any device that accepts over-the-air updates or remote configuration commands can, in principle, be told to do something its owner didn't authorise. This is not a theoretical concern; it's already happened with other implanted medical devices.

In 2012, security researcher Barnaby Jack demonstrated that pacemakers could be wirelessly commanded to deliver an 830-volt shock from fifty feet away from a potentially fatal attack requiring no physical access to the patient. The attack worked because the device trusted any command that arrived in the right format. It didn't verify who was sending it.

BCIs share this exact attack surface. A device that stimulates the motor cortex (a region in the frontal lobe of the brain, responsible for planning, controlling, and executing voluntary skeletal muscle movements) to help a paralysed patient move a limb is, functionally, a device that applies electrical signals to brain tissue. The security question of who gets to authorise those signals and how the device verifies that authorisation is not a minor implementation detail.

This makes over-the-air updates a particular concern. An unprotected update channel is an attack surface. A device that accepts remote updates can, if that channel isn't properly secured, accept them from anyone, not just the manufacturer. For most software, a malicious update is a serious nuisance. For an implanted neural device, it becomes a different league of threat.

(Over-the-air updates means delivering a software update wirelessly, without any physical connection. For example your phone does this when it downloads an iOS or Android update in the background. )

Data exfiltration at the platform level This is the attack vector that requires the least sophistication and carries the most scale. It doesn't require hacking the implanted device at all.

Neural data in most current BCI systems is offloaded to external servers, for processing, storage, model training. Those servers are subject to the same breaches as any other database. The difference is the nature of what's stored. A leaked email password is annoying. A leaked database of neural recordings is something you cannot fix. You can't reset your brain. You can't issue yourself a new neural profile. The data is permanently identifying, permanently sensitive, and permanently yours in the sense that it can never stop describing you, even after it's out of your hands.

The Privacy Problem Is Actually Worse

Here’s something worth sitting with: most of the neural data risk doesn’t come from sophisticated cyberattacks. It comes from completely legal data practices.

Emotiv, one of the world's largest consumer EEG manufacturers, allows sharing of anonymised neural data with third parties for research and commercial purposes. This isn't a breach. It's in terms of service. A Neurorights Foundation survey of 30 direct-to-consumer neurotechnology companies found that clicking "I agree" on 29 of them would grant the company the right to sell that neural data to third parties. The users generating those neural profiles had no meaningful visibility into how the data would be used, by whom, or for how long.

Think about what that actually means. You buy a consumer EEG headset to meditate, or to study, or to experiment. You click through the setup screens. You have now, legally, handed a company the right to sell recordings of your brain activity, which may contain traces of your emotional states, your cognitive responses, your subconscious reactions to the content you consumed while wearing the device to anyone willing to pay for it.

The legal weight of this became sharply visible in 2023, when Chile's Supreme Court ordered Emotiv to delete the brain data it had collected on a former senator. The court found that retaining anonymised neural data for research purposes, without specific prior consent, violated his constitutional rights. That case was possible because Chile had done something no other country on earth had done: it amended its constitution to enshrine the right to mental privacy and cognitive liberty directly in its founding law. The senator had a right to invoke. Most people don't.

Major data protection frameworks, including GDPR in Europe, have no specific provisions for neural data. In the United States, Colorado and Minnesota have begun developing targeted neurotechnology legislation. Federal protections do not yet exist. The gap between what the technology can do and what the law protects against is wide, and it is being filled, right now, by individual companies making individual choices about what to do with data that has never existed before in human history.

What Secure Neurotechnology Looks Like

The security community has been here before with pacemakers, with insulin pumps, with every connected medical device that shipped convenience before it shipped caution. The technical lessons from those fights aren't complicated. They're just being ignored again.

Encryption, end-to-end. Neural data should be encrypted both in transit and at rest. Manufacturers will tell you implanted devices don't have the power budget for heavy cryptography. That's an engineering problem, in that a solvable one, not an acceptable reason to ship unencrypted hardware into someone's brain.

Proper authentication for every connection. The attack that let Barnaby Jack command a pacemaker to fire from fifty feet away worked because the device accepted any command that arrived in the right format. No verification of who was sending it. Every wireless connection to a BCI should require real mutual authentication not "the device responded, so we trust it."

Secured over-the-air updates. Firmware updates need to exist, because the alternative which would be requiring surgical intervention every time a security patch is needed is clearly untenable. But they need to be cryptographically signed and verified, so that a device can confirm it's receiving a legitimate update from a legitimate source rather than instructions from an attacker who's learned the update protocol.

A real off switch. This one gets the least attention. Users should be able to disable the wireless radio entirely. Not airplane-mode-but-still-broadcasting off. Actually TURN off. A device that is permanently broadcasting neural data is a device that is permanently exposed. The user should have the ability to make that exposure stop.

And underlying all of these: security needs to be designed in from the beginning, not bolted on after the product ships. The IoT industry made the convenience-first choice a decade and a half ago with smart home devices, and we are still cleaning up the consequences. The difference is that a compromised thermostat sends your heating schedule to a stranger. A compromised BCI has electrodes in your motor cortex.

The Window Is Closing

Maybe you're not planning to get a brain implant. Here's the scale of what's already happening, whether you are or not.

On March 30, 2025, Precision Neuroscience received FDA 510(k) clearance for its Layer 7 Cortical Interface, the first full regulatory clearance granted to a company developing a next-generation wireless BCI. Neuralink has implanted devices in multiple human patients. Emotiv, Muse, and Neurosity EEG headsets are on retail shelves. The distance between "laboratory prototype" and "consumer product" is compressing faster than anyone predicted five years ago.

(Precision Neuroscience is a neurotechnology company developing a minimally invasive brain-computer interface (BCI) designed to help patients with paralysis control digital devices using only their thoughts.)

The security and regulatory infrastructure is not keeping pace. The devices are moving faster than the rules, faster than the security research, faster than the legal frameworks that would tell a company what it is and isn't allowed to do with the data those devices generate. Someone has to close that gap.

The skills that make someone good at CTF reading systems for weaknesses, thinking like an attacker, finding the edge case nobody planned for are exactly the skills this field is missing. Neurotechnology has researchers and neuroscientists. It doesn't have enough people asking what breaks.And here we have a target that actually matters.

The skills that make someone good at security research, reading systems for weaknesses, thinking like an attacker, finding the edge case nobody planned for are precisely the skills this field is missing. Neurotechnology has neuroscientists and electrode engineers and machine learning researchers. It does not have enough people whose job is to ask: what breaks? What gets abused? What does this look like from the other side?

Why This Matters More Than It Seems

There's a version of this story where BCIs are simply the next generation of consumer technology — a more intimate interface, a cleverer device, with the same security challenges and the same eventual solutions as everything before it.

There's another version where they're categorically different.

Every piece of consumer technology before this has captured what you do: what you search, what you buy, where you go, what you say. BCIs capture what YOU are, the electrical signatures of thought and emotion and reaction that exist below the level of deliberate expression. Data that you didn't choose to generate. Data that reveals things about you that you might not know about yourself.

The engineers building the next generation of these devices will make decisions, about encryption, about data retention, about what gets transmitted and what gets stored and what gets sold which will determine whether neural interfaces become tools of human dignity or the foundation of a surveillance infrastructure that nobody consciously consented to. Those engineers are, in many cases, students right now. Some of them are security researchers who haven't yet looked at this space.

It is worth looking at this space. Before someone else makes the decisions for you. Before the terms of service have already been agreed to. Before the data that cannot be reset has already left the building.

The window is open. It won't stay open.

It turns out that the dream of being a "lazy" architect, watching things happen in the background while you are busy playing Elden Ring, has been hindered by a chaotic mess of custom bridges and data silos. But the game is changing. We are moving from a world where you have to pick up the cards to a world where you hire a professional gambler who already knows every trick and strategy.

1. Deep Dive into MCP

What is MCP?

Ever thought how cool it would be to not even click the buttons or do the operations an AI model instructs you to for your query? Being completely lazy, doing nothing but watching things happen in the background while you continue watching Stranger Things.

In plain English, the Model Context Protocol (MCP) is a universal language that allows an AI to talk to your computer and the internet.

Before MCP, if you wanted an AI to read your files or check your calendar, a developer had to write a "custom bridge" specifically for that task. If they switched AI models, they often had to rebuild the bridge. MCP replaces this chaos with a single, open standard.

In short, MCP’s origin story is about breaking data silos: every new data source used to demand M×N bespoke connectors, MCP cuts that to M+N by standardising the integration.

The crust of the technicality

To understand how MCP actually works under the hood, we need to look at the Host-Client-Server relationship and the JSON-RPC signals used.

1. Main Components

• MCP Host: This is the Environment/Application you are actually using, like Claude Desktop, Cursor, Antigravity (pretty famous these days), or a specialised IDE. The Host is the main character of the movie, who manages security and decides which AI model gets access to which tools.

• MCP Client: This is the Translator inside the Host. It maintains a 1:1 connection to a server and handles the "handshake" to ensure both sides are speaking the right version of MCP.

• MCP Server: This is the main core. It’s a small, lightweight program that knows how to do one thing well, like "Read Google Drive" or "Search a SQL Database." It exposes these abilities to the AI in a standardised way.

2. The Communication (JSON-RPC 2.0)

MCP uses JSON-RPC 2.0, a lightweight "Remote Procedure Call" protocol. Instead of messy, unstructured chat, the Client and Server send strict, numbered messages:

• Requests: "Hey Server #1, run the get_products tool for 'DMart'."

• Responses: "Hey Client #1, here is the data for Request #1: <data>"

• Notifications: "Heads up, the file you were watching just changed!" (No reply needed).

• Format:

  { jsonrpc: "2.0", id: number | string, method: string, params?: object }

If you want to read more about JSON_RPC 2.0 in MCP, refer to this: https://medium.com/@dan.avila7/why-model-context-protocol-uses-json-rpc-64d466112338.

3. The Transport Layer (How the data moves)

• Local (stdio): Used for tools on your own machine. The Host starts the Server as a "child process", and they whisper to each other through the computer's internal pipes. It's incredibly fast and secure because the data never leaves your hardware.

• Remote (SSE): Used for cloud services. It uses Server-Sent Events to keep a "live wire" open between your AI and a remote server, allowing for real-time updates.

MCP Server have a lot of tools whose selection and order of execution are decided by the LLM connected. A simple MCP Server Tool looks like this:

@mcp.tool()
async def get_attendance(student_id: str, date: str) -> str:
    """Get attendance status for a student on a given date."""
    # Implementation details...

Visit the official docs if you want to learn more about the implementation: https://modelcontextprotocol.io/docs.

Although there are some people that are saying that MCP is something that is overhyped. If you look at the registry PulseMCP, there are over 6,600 servers listed, but the number of active enterprise users is still catching up.

2. Agent Skills

Have you ever noticed that while in a vibe coding session of yours, the commands or code getting executed aren't the latest or the most optimal?

You are in the flow, and the AI is hallucinating a library version from the good old days of 2021, and you only realise when you have to open those docs poised in anticipation of your arrival.

Here comes agent skills, first introduced by anthropic.

Now, what is Agent Skills? It's a very simple concept, actually, if we see. Agent skills is a meta tool for conversation and execution context injection with progressive disclosure.

By splitting your agent's knowledge into these three layers, you solve the two biggest problems in AI development:

• Token Bloat: You aren't paying for the agent to "remember" every script in your scripts/ folder until it actually needs to run them.

• Context Drift: Because Level 2 is loaded only when triggered, it acts as a "fresh start" for the agent's logic, ensuring it follows your current documentation instead of its past training.

A good read if you want to learn how to write agent skills for something you’re building or tailor an agent to your needs: https://leehanchung.github.io/blogs/2025/10/26/claude-skills-deep-dive/

Some cool Agent Skills to try:

https://github.com/resend/email-best-practices

https://github.com/supabase/agent-skills/tree/main/skills/postgres-best-practices

https://vercel.com/blog/introducing-react-best-practices

Where MCP Plugs In: The Capability Layer

MCP is the universal connector that allows your Level 3 Resources to actually do something. While your Skill might tell the agent "Update the documentation," MCP is the physical protocol that lets it log into GitHub and push the commit, so you don’t have to move your rusty fingers to do some clicks.

3. Unpacking Agent as a Service (AaaS)

If MCP is the universal hardware port, Agent-as-a-Service (AaaS) is the full-service "robot butler" that plugs into it. Traditionally, Software-as-a-Service (SaaS) meant subscribing to a tool where you had to do the heavy lifting inside their interface.

With AaaS, you aren’t just renting a static tool, you’re renting an autonomous digital worker that lives in the cloud and actually performs the tasks for you.

SaaS vs. AaaS

The shift is a change from “do-it-yourself” to “done-for-you".

Active vs. Passive: SaaS is a sophisticated toolbox that is useful, but it stays quiet until you learn how to use it or hire someone. AaaS agents are "always-on" teammates (they don’t ask for rigid job timings ;) ) that monitor your data, plan their own steps, and execute goals without you constantly poking them.

This brings new ways to do things.

1. New Revenue Models: Paying for Wins, Not Seats

Forget the old "flat fee per user" model that SaaS uses. Since agents do specific work, vendors are shifting to Outcome-Based Pricing.

• Micro-charges: Instead of a $100/month flat rate, you might pay per lead generated or a small fee for every customer ticket successfully resolved.

• Scalability: You only pay for what you actually consume, making it perfect for startups that need to scale fast without the "psychological barrier" of a massive monthly bill.

2. Faster Integrations:

• Efficiency: Building custom connectors used to take months of developer time. With MCP, agents can "plug and play" into your GitHub, Slack, or Google Drive via a single, universal interface.

• Speed: Companies using MCP report saving roughly 30–40% on their integration timelines.

The Casino Whale Analogy

1. The Gambler (The AI Agent)

The gambler is the core brain. They have the intent to win and the ability to make decisions based on what they see. But a gambler without a strategy or a seat at the table is just a person in a fancy suit (probably you)

2. The Cheat Sheet & Strategy (Agent Skills)

This is our 3-layer architecture.

• Level 1 (Metadata): The gambler knows which games are in the room (Poker, Blackjack, Roulette). They don't have the mood or gut feeling which one to play in their head yet, but know where to walk.

• Level 2 (Instructions): Suppose they sit at the Poker table. Now, they “activate” their Poker Skill. They recall the specific rules of this game, the "latest and most optimal" strategy that overrides their general knowledge.

• Level 3 (Resources): This is the mental math. The gambler pulls out a specific "resource", a probability chart for a 52-card deck, poker hand combinations, etc., to calculate the exact odds of the current hand.

3. The Keycard & The Chips (MCP)

The gambler has the skill, but they can't play without access.

• MCP is the Casino Keycard. It lets the gambler through the door, identifies them to the pit boss, and connects them to the house's data (the cards being dealt).

• Without MCP, the gambler is just gambling in the mind. With MCP, they are plugged into the table's live feed.

4. The Winning Hand (Agent-as-a-Service)

In the old world (SaaS), you were the gambler. You had to buy the strategy book and sit at the table yourself. In terms of AaaS, you are just the Backer. You don't play the game, you simply hire the pro gambler, give them the MCP keycard to your accounts, and they play 24/7. You don't care about the individual card flips, you only care about the chips stacking up in your vault.

Human Upgrades: Does this mean we lose our jobs?

(Only applicable if you have one right now)

We aren't just losing repetitive roles, but we are evolving them to ship stuff faster.

There is a shift that is happening, and we can see that by considering the layoffs going on.

Roles like Agent Supervisor, Prompt/Task Engineers for humans who oversee "fleets" of Agents. Instead of doing the data entry, mailing, and logging, you monitor the agents for the edge cases and step in only when things get complicated.

Strategic Oversight: Humans move from being "order-takers" to "strategic planners," focusing on high-level goals like system designs while the agents actually get their hands dirty (no, we are not calling them slaves).

Try picturing this: It's 2 in the morning. Your function ran perfectly during testing, and now it's crashing on the simplest input. Somewhere in your 200 lines of code, a bug is laughing at you. And in all honesty, it has every right to do so.

They’ll say that debugging is like clever detective work, but they always forget to mention that you’ll be spending the first three hours digging for clues with a plastic fork.

Why Bugs Are Actually Your Best Teachers (I’m not joking)

Bugs feel like personal betrayals, but they're secretly your best teachers. They exist to turn you from someone who types code into someone who understands it. Here's the truth. Tutorials teach you syntax. Bugs teach you how to think. Every bug you fix sharpens your intuition about how code behaves in the real world. You start recognizing patterns, like memory leaks or API failures.That’s why senior developers seem like wizards-they’ve spent hours wanting to delete VS code and change their major.

My First Hackathon: 48 Hours of Rage, Embarrassment and Depression

Let me take you back to my first year. It was my first hackathon. My team was building a website, I was handling the frontend, and confidence was dangerously high(I have no idea why).

Hour 6: "This is going great! We'll finish early!(all i had done till then was understand what frontend and backend meant)"

Hour 18: "Why won't this button work? (my website was completely broken. The colours were wrong and the buttons were not buttons, just images)"

Hour 30: "WHO MOVED MY SEMICOLON!!?(believe me. This is the biggest issue) "

If I were to be completely honest, around 70% of the bugs were textbook rookie mistakes. They were either missing semicolons or variables that I named incorrectly, and then I would spend 20 minutes wondering why everything was undefined. My teammates would look at my screen for 2 seconds and point out the error. I feel humiliated to this day.

The copy-paste disasters were the most humbling.I needed a responsive navigation bar, but my vision seemed too complex to build from scratch. So, I asked ChatGPT. It gave me a beautiful block of HTML, CSS, and JavaScript. In my excitement, I copied everything and pasted it directly into our project. For a glorious second, it worked. The new nav bar was there. But then I noticed the rest of my website had completely imploded. The CSS from ChatGPT was at war with my own styles. My page's buttons were suddenly the wrong color, the font sizes were all over the place, and my layout was a mess.The code looked like the real deal, but it was a black box. Because I hadn't written it, I had no idea how to fix the conflicts without breaking the nav bar itself. It just wouldn't work.

By hour 40, I realized I'd spent more time debugging than writing new code. That hackathon, I understood that debugging wasn't only about finding the errors. It's about fixing the issues without creating new ones.

A Field Guide to Code's Worst Nightmares

Before diving into debugging strategies, lets meet the usual suspects. Understanding bug types is crucial for choosing the right approach.

You'll meet all sorts of pests on your coding journey. Here are the headliners:

Syntax Bugs (The Grammar Police): A forgotten semicolon or a misplaced bracket, and your code will refuse to run. Your editor will scream at you with angry red squiggles. Listen to it.

Logic Bugs (The Gaslighters): These bugs make you question your own sanity. Your code runs with zero errors, but it will confidently tell you that 2+2=5. They don't crash your program, but they do make it a compulsive liar.

The Copy-Paste Special (The Trojan Horse): You paste a 'perfect' solution from the internet, but it was built for a slightly different reality. The moment you try to change anything, the whole thing collapses like a house of cards. This usually happens when you’re too lazy to even look at what chatgpt gave.

Your Debugging Toolkit: From Panic to Pro

Now that we’ve covered the types of bugs that you’ll encounter, lets talk about the principle strategies for getting rid of them.

Here's what actually helps when you're neck-deep in bugs:

1. Print Statements: Your Best Friend

Don't be ashamed of spamming console.log(). They may be messy, but they are the simplest method to get rid of bugs that you just can’t seem to find. You’ll remove the lines later anyways. You can use print statements to trace function calls and variable changes, or even to check values before and after operations.

2. Read Error Messages Like Love Letters

Error messages aren't trying to hurt you. They're there to help. That scary red text is actually the bug writing you a detailed confession about exactly what went wrong.

1. Stack traces: Read bottom to top to see the sequence of function calls
2. TypeError: You're using the wrong data type ("Cannot read property 'name' of undefined" means that you're accessing a property on something that doesn't exist)
3. Syntax errors: Point to exactly where your code is malformed

3. The Rubber Duck Method

Explain your code to somebody line by line, or just say them to yourself out loud. You’ll catch your mistake mid-sentence. It may be embarrassing, but it works.

4. Divide and Conquer

Comment out half your code. Does the bug still happen? Great, it's in the other half. Keep dividing until you find the exact line causing trouble.
Pro tip: Create a minimal reproduction case—the smallest amount of code that still shows the bug.

5. The Sacred Rule of Copy-Paste

Read the code if you're copying it from anywhere (Stack Overflow, ChatGPT, that GitHub repo), and understand what it does. Make sure that it fits your specific problem. Blindly pasting code is like following GPS directions without looking at the road.
Test it in isolation before integrating it, and check for stuff like dependencies and version requirements. Make sure to rename the variable to match your code.

6. Version Control is Your Time Machine

Git isn't just for team projects. It's for the moment when you break something that was working perfectly an hour ago. git checkout can literally undo your mistakes and bring you back to when life was simpler. Commit frequently with clear messages, and use branches for risky experiments.

Before You Go: A Reality Check

Bugs will definitely frustrate you. They'll make you consider switching to a branch where the biggest problem is understanding the author’s feelings.

But, the satisfaction of finally understanding why something isn't working is addictive.

So the next time you encounter a bug, take a deep breath, grab some snacks and remember:

Every expert was once a beginner who refused to give up on a bug.

Happy debugging, and may your console.logs be ever informative!

P.S- if your code works perfectly the first time, you either wrote Hello World or forgot to actually run it :)

To add some excitement to the cybersecurity community, there is something called Capture the Flag, or CTF.

No, not the old-school playground game. This version is digital, and you can think of it as a collection of puzzles and obstacle courses built for your computer. The goal is simple: find a hidden piece of text or string called a flag. It usually looks something like this:

flag{s0m3th1ng_clev3r}

Why CTFs

CTFs are one of the few things that give you an opportunity to hack into something, test your hacking skills, and make it fun since you are competing with other hackers at the same time.

They help you understand how systems fail, and once you see how things break, you start to understand how to build them better.

CTFs are one of the best ways to build and sharpen cybersecurity skills, whether you are a complete beginner in the field or already a seasoned veteran.

They are usually of two types:

Jeopardy Style

This is the most common format.

You are given a board of challenges divided into categories like:

• Web Exploitation

• Cryptography

• Forensics

• Reverse Engineering

• OSINT

• Pwn / Binary Exploitation

These categories are mostly present in every CTF, and there are categories that are getting popular nowadays like AI, Web3, hardware, etc.

Each challenge is worth points based on difficulty. You pick one, solve it, submit the flag, and get the points. Jeopardy-style CTFs are the perfect place for beginners to get into cybersecurity.

Attack Defense Style

This is the advanced version.

Teams are given servers to defend while simultaneously attacking other teams’ servers. It is fast-paced, competitive, and chaotic.

For now, I will focus on Jeopardy-style CTFs because that is where the real learning happens.

A Tour of Common CTF Challenge Categories

Here is what you will usually find when browsing a CTF challenge board.

Web Exploitation

In Web Exploitation challenges, you are usually given a vulnerable or broken website, and you are expected to find the flag.

Some of the things to look for in web challenges include:

1. The source code of the website using the view-source: .

2. Network requests using proxy tools like Burp Suite or Caido

3. Suspicious things in the developer tools section like unwanted cookies, console logs, etc.

4. Misconfigured APIs

5. Understanding various web frameworks helps you learn details specific to each framework. For example, FastAPI provides a /docs endpoint, and similarly, other frameworks have their own unique features and conventions.

The most common vulnerabilities found in web-based challenges are SQL injection, Server-Side Request Forgery (SSRF), XSS (Cross-Site Scripting), and IDOR (Insecure Direct Object Reference). These are all part of OWASP’s Top 10 vulnerabilities.

If you are interested in learning how these vulnerabilities actually work, PortSwigger Academy is the best resource to learn web-based vulnerabilities.

Cryptography

This category focuses on encoded or encrypted data. Most of the time, you will be given a script and a file that is encrypted or encoded using that script. You have to understand what’s actually going on in the script and somehow find a way to decrypt or decode whatever is in the file.

I know this sounds not that difficult, but trust me, these are some of the most challenging categories in Jeopardy-style CTFs. If you’re good at Python and encoding-related concepts, this will be your favourite category because most of the scripts are written in Python.

Other times, it is about recognizing common encodings like Base64 or hex and knowing how to decode them.

How to start:

1. Use tools like CyberChef and dcode.fr, as these are some of the best online tools that support encoding and decoding for almost every cipher in existence.

2. If the challenge has its own encryption script, try to understand it and reverse engineer the logic so that you can make your own decryption script.

3. Sometimes the challenges have a script running on an nc (netcat) server, and it gives out clues and keys for you to use and deduce the encryption to decrypt the encrypted text.

Forensics

This is one of the most exciting and wild categories in Jeopardy CTFs.

You might be given a network capture, an image file, a memory dump, a corrupted machine, or sometimes, if your luck is bad, a file with a file extension so random that you have to Google it for five minutes.

In these challenges, you have to act like a digital detective and find the flag through lots and lots of data.

How to start:

1. You can try your luck by running strings or grep, but usually flags are not that easy to find.

2. If you have a file format that you are not familiar with, make sure you know what it is and how to work with it.

3. These challenges have a sub-category called Steganography, which basically means hiding things in images, audio files, videos, GIFs, etc. This website has a checklist with all the tools you should try if you encounter a steg challenge.

4. For network capture files (pcap or pcapng), Wireshark and tshark are your go-to tools to extract useful information.

5. For challenges involving memory dumps or machine dumps, you need patience and the ability to look for different clues while searching for the flag.

Reverse Engineering

Oh man, personally, I find these very difficult because you are literally given a compiled program with no source code and asked to figure out what it does.

The solver is expected to examine binaries, read disassembled code, and locate hidden logic or hardcoded strings in an executable.

How to start:

1. Run strings to find hardcoded strings that may or may not act as clues for the flag.

2. Use tools like Ghidra or IDA for decompiling, and gdb for debugging and runtime analysis. Decompilers are tools that turn executables into readable code.

3. Run the program and try to break it or make it do things it’s not supposed to.

OSINT

One of my favourites is OSINT, which refers to Open-Source Intelligence. In these types of challenges, the information you need to get the flag is public and accessible to everyone on the internet.

This usually involves searching usernames, analyzing images, checking public records, and connecting small clues scattered across the internet. Common challenges include identifying locations from photos, tracking online identities, and finding leaked or archived information.

How to start:

1. Learn how to search effectively using techniques like dorking and reverse image searches.

2. Tools like whois, Wayback Machine, and Sherlock are extremely useful.

Binary Exploitation

These are often referred to as pwn (short for “own,” meaning to gain control of a program) challenges.

This category involves exploiting vulnerable programs, usually written in C, to control execution or retrieve a flag.

How to start:

1. Understand how the stack, heap, pointers, and buffers work. Most pwn challenges are based on simple mistakes in C programs.

2. Learn to use tools like gdb to step through a program, inspect memory, and see where it crashes.

3. Start with beginner challenges that run on your own machine before touching remote services. This helps you understand what your input is doing.

4. Tools like pwntools make interacting with binaries and remote servers much easier once you understand the fundamentals.

How to Start Without Losing Your Mind

1. Get good at searching

   Knowing how to search properly is a huge part of CTFs. Looking up error messages, tools, or random things you do not understand is completely normal. Reading write-ups after you have given a challenge an honest try is not cheating, it is how you realize what you missed. Writing your own write-ups helps even more because putting your thoughts into words makes the learning actually stick.

   Here’s the website where we post write-ups of the CTFs we participate in: https://z0d1ak.vercel.app/

2. Do not do it alone

   CTFs are much more enjoyable with other people. Even if you compete solo, communities like CTFtime and Discord servers are full of people sharing hints, tools, and encouragement.

3. Start with beginner-friendly platforms

   Some CTF platforms are built specifically for learning. Some of the best platforms to get started with are:

   • picoCTF

   • HackMyVM

   • CTFlearn

   • OverTheWire

4. Set up a comfortable environment

   A Linux VM makes life easier since most development and security tools work best on Linux. Kali Linux is especially helpful because it comes with many tools preinstalled, so you can start experimenting right away. Using it in a VM gives you a safe space to break things, learn from mistakes, and get comfortable with the terminal and real-world server setups.

5. Trust that the flag is there

   CTF challenges are designed so everything you need is included. Read the challenge description carefully, look through the files, and do not rush past small details.

Final Thoughts

CTFs are giant playgrounds built by curious people for curious people.

You will get stuck. A lot. You will feel confused, frustrated, and convinced that everyone else knows something you do not. At some point, you will probably think you are just bad at this.

Then something clicks.

You notice a small detail you ignored earlier. You try one more thing. A flag appears. Suddenly, all that confusion turns into excitement, and for a moment, you feel unstoppable.

This eureka moment is why people keep coming back.

Do not give up when you get stuck. Keep reading write-ups after challenges end and learn how others solved them. Every write-up you read adds another tool to your mental toolkit. Even when you solve nothing, you are still learning.

Most importantly, keep participating. CTFs happen almost every weekend, and there is always another chance to try again. You can find upcoming competitions on CTFtime at https://ctftime.org/event/list/upcoming.

So pick a beginner CTF, open a terminal, and start poking at things. The worst thing that can happen is that you learn something new. The best thing that can happen is that you find a flag :)

Happy hacking.

Especially if the “stuff” we’re talking about is umpteen discord pings of 800 event participants saying things along the lines of:

If you understood what’s going on, you’re one of the 50 odd people who lived through the trauma of 27th September, 2024 - day 0 of ACM-VIT’s flagship event Cryptic Hunt 3.0.

If you don’t understand, I envy you - but allow me to give you some context (I’ll keep it to the point, I promise). Vellore Institute of Technology, one of India’s very reputed engineering universities, holds its annual tech fest in the months of September/October, called graVITas. Many clubs and chapters organise events during it - hackathons, ideathons, various tech competitions - but a select few organise the “premium events”, premium enough to be mentioned in The Hindu newspaper. Our chapter, ACM-VIT, organises one of them - our legacy scavenger hunt which we call “Cryptic Hunt”.

What is Cryptic Hunt?

800 participants. 36 hours. A hunt across campus.

In simple words - it’s a scavenger hunt across the huge campus we know as VIT, but with a technological twist. We developed our own mobile app (from scratch - and yes, the tech details will also follow). Our dedicated research team builds a set of questions related to cryptography and network security. Upon answering these questions, you’re hinted to a particular location on campus - where we have discretely pasted QR codes. Scan the correct code for a question in our app, and voila - you get points and move up on the leaderboard.

Pretty simple, right?

Let’s have a look at a question from last year:

The solution for this, along with all our other questions is available on our GitHub repository - we make our solutions public once any event ends ;)

But hey, that’s all the context I can give you - now begins the real story. I’ll split it up into sections, each being technical or non-technical - if you’re here for the drama, for the tea, feel free to skip the tech part. However, if you’re a nerd like almost all of us are, the tech we’ve implemented is quite cool - do check it out.

The grand tale of Cryptic Hunt 2024 shall be divided into the following chapters:

1. The Planning [non-tech]

2. The System Design [tech]

3. The Implementation [tech]

4. The D-Day [non-tech]

5. The Downfall [non-tech]

6. The Resilience [tech (mostly?)]

7. The Post-Mortem [non-tech]

The Planning [non-tech]

It all started off during the summer break - just a group of 15 odd kids sitting on google meets whiling away the odd hours at night. We were the new senior core of ACM-VIT, and at that time we thought we were the most important people in the world - “make ACM great again” is pretty much what our slogan was. Reminds you of a certain personality in world politics, maybe? Yeah, there were red flags quite early on, I guess.

However, the first order of action - graVITas was approaching, and we needed to make sure Cryptic Hunt was a grand, grand success. We needed to show everyone that we’re the real deal, the real stuff. Who was watching? No one, really, but hey - let some kids be happy thinking they matter.

Utopian world. Don’t you wish we lived in one? A world where everything went according to plan, everything was ideal. But nah, utopia is only achievable as an album, not in implementation. Yet, everyone plans an ideal situation. An ideal approach. An ideal timeline, maybe.

So did we.

And when I look at it now, 12 months later, I can’t help but laugh. In all fairness, it isn’t a bad schedule in the least - extremely achievable. It’s just funny how absolutely NOTHING went as per schedule - the “plan” going absolutely nuts from the very beginning.

Here it is, the “ideal plan” we had in mind (as of July 2024):

How much of this actually went according to schedule? Good question. Good, good question.

There’s a lot of reasons why we weren’t able to achieve a lot of what we set out for, and since I’m being completely transparent out here, I won’t hold anything back. It wasn’t just a lack of skill, it was lack of effort, a whole lot of politics, constant ego clashes and, well, some bad luck as well.

Coming back to the plan - the tech part was quite simple. Everyone takes up certain tasks, or “issues”, codes them and raises a pull request from their fork to our official GitHub repository. We had the tech split into 2 - backend and frontend (app) - and each division had one final boss, the best in each field we had in our senior core, who were supposed to give the final approval before a piece of code was merged.

This was probably our first mistake. Not the tech division, don’t get me wrong, but the appointed final bosses. Because what ensued was a massive cold war between frontend and backend, one on such a scale that it threatened the very existence of our entire event. Okay, maybe that’s a bit of an exaggeration, but it was pretty bad to say the least - because when you’re way behind deadlines and find out something major is broken, the last thing you want is to hear:

• Mr. Frontend: “I’ve made sure the app is perfect. Whatever issue is there is in the backend, tell them to figure out what the hell is wrong with their sh\t”*

• Mr. Backend: “The backend is made foolproof, I can give you my 100% guarantee on that. Ask frontend to fix their stupid app, not me.”

Oh boy.

The System Design [tech]

Alright, time for the nerdy stuff. If you're still reading, you're either genuinely interested in our tech stack or you're procrastinating on something else (no judgment, I've been there).

We built our backend using Go Fiber - fast, lightweight, and honestly just fun to work with. For our database, we went with CockroachDB, and before you ask - yes, the name is as weird as it sounds, but hear me out. This thing has distributed SQL capabilities, maintains high consistency, and scales horizontally like a dream. Plus, it survived our chaos (literally), so it earned some respect.

Everything was containerized and deployed on Google Cloud Run. Why? Because we're lazy developers who don't want to manage servers, obviously. Auto-scaling, load balancing, pay-per-use - it was supposed to be our silver bullet. Spoiler alert: it wasn't, but we'll get to that trainwreck in a bit.

Authentication was handled through Firebase with Google Sign-In as the primary method. Static assets like question images and media were stored in Google Cloud Storage buckets - nothing fancy there, just good old reliable cloud storage.

The mobile app was built with Expo React Native. One codebase for both iOS and Android - the dream of every developer who's tired of maintaining separate native apps. We cached question data locally for better performance, and used Firebase Cloud Messaging (FCM) for cache invalidation. Whenever new questions went live, FCM notifications would trigger cache clears to make sure everyone got the latest data. In theory, this was brilliant - fast performance with real-time updates. In practice... well, let's just say FCM and we had some trust issues.

The Implementation [tech]

The journey started innocently enough. Our design team huddled together in what I can only describe as the most intense brainstorming session of our lives. After cycling through themes that ranged from "cyberpunk dystopia" to "medieval fantasy" (don't ask), we settled on Minecraft. Yes, Minecraft. Look, it worked, okay? The blocky aesthetic was perfect for a tech event, and everyone immediately got the vibe.

With the creative direction locked in, we split into two teams - backend and app developers. Classic divide and conquer strategy. The backend team, bless their souls, documented every single API endpoint they planned to build. This wasn't just good practice; it was survival. The app team needed to know exactly what they'd be working with, and nobody had time for "oh wait, I changed that endpoint yesterday" surprises.

Both teams worked in parallel, which sounds organized and professional, but was actually just organized chaos. Daily standups that went on for an hour, messages at 2 AM asking "did you push the auth fix?", and the occasional panic when someone realized their local changes would break everything else.

Once the core features were done, both teams came together for the anti-cheat implementation. This was actually pretty cool - making sure both sides of the system worked in perfect harmony to catch cheaters. Spoiler: it worked... mostly.

Deep breath Alright, storytime. And not the fun kind.

Mobile App Challenges

Let's start with the app stores, shall we? You know how they say "it works on my machine"? Well, it worked on our phones, but the app stores? They had opinions. Strong opinions.

Sign-in inconsistencies - We kept getting rejections from both Google Play and Apple's App Store because our sign-in flow was apparently "inconsistent". What does that even mean? We still don't know, but after the 4th rejection, you just start making random changes and hoping for the best.

Permission handling issues - iOS is particularly picky about why you want camera and location permissions. Our initial reason strings were apparently too vague. "We need your camera" wasn't good enough - they wanted to know WHY we need it, HOW we'll use it, and probably our grandmother's maiden name too. Fair enough, I guess.

Apple's mandatory sign-in requirement - Here's a fun one. Apple decided that if you offer Google Sign-In, you MUST also offer "Sign in with Apple". But our university ecosystem was built around Google. Everyone had Google accounts. Nobody wanted Apple sign-in. But Apple didn't care. So we had to implement it anyway, for exactly zero users who wanted it.

Legacy Firebase client issues - This one almost killed us. We were reusing older Firebase authentication clients that were tied to outdated SHA keys. Everything worked fine in testing, but on the actual event day? Complete authentication failure. We spent the entire final day scrambling to fix this while 800 participants waited. Not our finest moment.

Backend Challenges

If you thought the app issues were bad, wait till you hear about the backend disasters.

Unreliable FCM delivery - FCM doesn't guarantee 100% delivery. Fair enough, networks can be unreliable. But our system was designed to retry failed sends automatically. So when FCM inevitably failed to deliver some notifications, it triggered retries, which prolonged the table locks, which made everything worse. It was a beautiful cascade of failure.

Escalating infrastructure costs - And here's the cherry on top of this disaster sundae. In our desperation to keep the system running, we started throwing money at the problem. Increased Cloud Run instance limits. Scaled up everything. "Just make it work, we'll worry about the bill later."

The bill came later. ₹36,000 later, to be exact. That's about $430 for our international readers, which might not sound like much, but remember - we're college students organizing a campus event. That was more than our entire budget for prizes, food, and everything else combined. Our treasurer nearly had a heart attack when they saw the Google Cloud invoice.

But hey, at least the system was running, right? Right?

The D-Day [non-tech]

It finally was here. The day we had been working so hard for.

Day 0 of Cryptic Hunt 2024 was here.

The app was live on the Play Store, and iOS users could access it using testflight (we spent 10k dude, had to do something). The backend was live, the database was up, the admin app was in place.

Oh wait - the admin app. We thought it was in place. When I entered the auditorium, there was a frenzy. I asked what’s up, and that's when I found out - the admin app simply wasn’t ready. “It’s okay, calm down, it’s just 800 people waiting to play your game. That’s not much. Calm down.” - that’s what I was telling myself constantly (horrible advice, in hindsight). The admin app was responsible for linking all questions to the correct QR codes, and all our QR pasting teams were ready to get the codes up across campus, but they could do nothing without the admin app. So we put them on standby, and got to work.

First things first, I needed to get an update on the status of the admin app. I called up my friend who was working on it, in quite audible panic. “It’s more or less done, I have all my changes on local - gonna push it in 5 minutes”, he said - and true to his word, he pushed all his changes to GitHub (after 55 minutes). But hey, better late than never - we had a working admin app. QR team got into action and put up the codes ASAP, while on-site management handled the crowd and got them seated for the opening ceremony.

The opening ceremony started, and was the only thing that happened without any major fumbles. Towards the end, they showed the links for the participants to download our official Cryptic Hunt App and register themselves and their teams on it.

And that’s when the downfall began.

The Downfall [non-tech]

They say “You know the good part about hitting rock bottom? There is only one way to go, and that’s up”. I wanna know who’s the “they” who say this - because boy, oh boy, does the “rock go bottom-er”.

- “Dude, it’s crashing.”
“Huh?”
- “The app, it’s crashing.”
“Nah, must be some net issue. Just tell that guy to restart and all will be good.”
- “Dude. It’s crashing for everyone. NOBODY can log in.”

That’s all the conversation was between me and my friend who was handling management. At first, I thought “eh, can’t always be completely smooth, must be the extra load”. Oh, innocent little boy - you had no idea how wrong you were.

We opened GCP logs - and what greeted us was a horrendous site. A bloodbath of 500 coded error messages, with metrics worse than the great collapse of the financial market in 2008. What? How? Why? No idea - but we still weren’t very demotivated. We’ve debugged through stuff before, no biggie - so we got to work doing some root cause analysis.

Here’s when management stepped up. They handled hundreds of students in the auditorium, each shouting their problem out loud. They calmed the ruckus. They settled the tension. While we sat and figured out why the heck would our onboarding section keep crashing, management were on their feet - collecting participant names and emails, their teams, their teammates’ emails - everything on pen and paper - just so that we could immediately cut to action once tech figures stuff out.

But tech - were we figuring stuff out? Uhhhhhh, not really.

We were completely clueless as to what was causing the issue. The issue we had pinpointed - all the requests to create/join a team were timing out. Why so? It was because the users table in our database was locked - i.e. it was inaccessible to be written to since it was undergoing some sort of a change. However, rather than being locked for an infinitesimal amount of time (that it would ideally be), it seemed to be permanently locked, hence all create/join team requests were just waiting in queue for the locked database table, and eventually timing out.

But what was causing this issue? We had no flipping clue. It’s not often that I’ve felt this helpless. Usually there is someone who knows what is going on when something messes up. Someone who has been in a niche situation like that. Someone who could be our stack overflow. But that day, everyone was equally clueless. The board, the senior core, the junior core - everyone just had a massive question mark on their faces.

We turned to AI. We did something only the most desperate (or the most stupid) devs do - fed entire files of code into ChatGPT and asked it - “oh lord, please tell us why our users table permalocked”. And GPT responded. We were shocked - how did we miss something this trivial? How? So many tech minds, and all of us missed something this small? It was unbelievable. We never added transaction rollbacks to our database querying code. That’s it. No tx.rollBack() code. That’s literally it. So, I got to action. Instantly found the few functions where there was a missing rollback, and instantly added it and pushed. Mr. Backend hit redeploy on Google Cloud Console - and we saw the code reach prod. I opened my phone, opened the CH app, and tried to join a team. It worked. 100 students wearing ACM t-shirts across the auditorium breathed a very audible sigh of relief. Management made the announcement to the participants - we’re good to go guys, lets get this event started.

And it crashed. Again. Bloody GPT, knew we shouldn’t have relied on it.

Oh my God. The auditorium was exploding with angry shouts from the participants, yet the silence felt deafening. It felt like an implosion of sorts. Suddenly it felt like the event was screwed, big time.

We were told to empty the auditorium, since we had used up our entire time duration (which was for the opening ceremony). The participants were livid. Most of our team was involved in calming them down, and assuring them that the event will start soon and they will be kept updated through our discord. As the participants left, and so did ACM members, the tech team sat outside the auditorium, on the floor, racking our brains hard - trying desperately to find any small error. ACM team was assigned a control room in an SJT smart classroom, so it was just 4 of us, and our dear chairperson - who was somehow quite calm and composed. Have to give it to him - when everyone was panicking and a lot of his reputation was at stake, guy managed to stay completely calm and just keep his trust in us - telling us to focus on getting the app functional while he handled any curveballs graVITas team threw at us.

This is where I made a big mistake. Mr. Backend had insisted on using a serverless backend, connected to a cockroachDB postgreSQL instance. We had our doubts about it. He assured us he had enough experience with it, the instances will upscale whenever the load is high. It won’t crash. It’ll work. He spent hours convincing us it’ll work. And when everything was in a state of chaos and panic, when we should have taken his advice and sat down and traced our steps through the code to see what was causing the issue, when we should have been working as a team - I flipped. Mr. Frontend and I turned on the backend guy, and started ranting about how this was an issue due to the serverless architecture of our backend. It simply wasn’t able to handle the load, we said. He disagreed, but we didn’t care at that moment.

We convinced the others, and soon most of the team was trying to find server-based alternatives for hosting our backend. We started finding possible issues with our serverless backend, nitpicking at everything. At one point, I noticed that the maximum active connections to our backend was capped at 100. “Voila!”, I remember thinking. I found the issue. Only 100 people were able to connect to our backend at once. Everyone started celebrating - we found the issue! A small oversight on the cloud console. No biggie, we instantly scaled it up to 1000 and redeployed. Metrics became green again, the cheers became louder. I felt like I was the most important person in the room for a while.

Why did I feel that only for a while? Because within 5 minutes, it crashed again.

Have you ever heard an entire classroom cuss out loud at once? Yeah, that definitely didn’t happen then.

We wasted more than 5 hours trying to switch to a server-based, locally hosted solution to our backend - when we shouldn't have ever doubted our most experienced guy in the first place. Sure, his entire argument was just “trust me bro”, but we should’ve done our due diligence on whether serverless was actually the issue or not, before directly assuming that it was the issue.

It was soon 8.30PM. Cryptic Hunt 2024 was cancelled for that day, officially. The events team was LIVID - they were facing a lot of backlash for the failure of our event, and we were being given (very well deserved) flak for it. ACM core went back to their rooms, awaiting further instructions. Our chairperson, along with the research lead and a senior core member of ours who was a part of graVITas events committee, headed to the fest control room to try and calm matters down there.

I remember leaving SJT all alone, while raindrops were pouring on me. I could just look at the floor while I walked. That walk will always stick by me. That walk back was one that was low on so many levels - it was straight up as if a ball of dark energy was around me as I walked. So many thoughts, so many mental conflicts. “Is any of this even worth it?”, “should have just left this chapter long ago”, “what’s the bloody point” - everything was hitting at once. There were quite a few losses/issues many of us were facing in our personal lives as well, while building up to Cryptic Hunt 2024 - so that didn’t help either. It really wasn’t that bad, don’t get me wrong. It’s just an event. In a college fest. A damn college fest. I know that as well, it all sounds so over-exaggerated - but at that moment, after 12+ hours of constant debugging in prod with absolutely 0 progress, any developer would’ve felt like absolute shit.

At the same time, three of our very own were waging a different war - one to make sure our event doesn’t get cancelled (even though it’d save us a lot of pain, imagine telling 800 people they won’t get refunded their INR250 + GST for absolutely no rhyme or reason). Our chair, research lead, and a senior core member (who was part of the events committee) were handling negotiations with graVITas control room. If you want a mild understanding as to how phenomenally grave the situation was, I’ll let you in on something the senior core member in concern told me:

“For the first time, I saw our chair sit on the footpath - and he looked… dejected? He looked as if he was about to cry. The same person who was calm throughout the day, who was the pillar of support we needed throughout the day - he was seated there all alone facing the wrath of the committee, just thinking ‘why did we put in this much of effort, if it was to all go in vain’ - that hit me hard.”

The Resilience [tech (mostly?)]

I crashed on my bed somewhere around 8:45PM - after more than 12 hours of useless debugging. I woke up at 9.15PM to a call from Mr. Frontend himself.

“Come to R3xx, we’re sitting and figuring out the problem,” he said. I just wanted some sleep, man. This seems impossible to figure out anyways - what's the point. Yet, despite my body telling me no in 15 different ways, I got up and slugged my way towards our senior’s room.

The atmosphere there wasn’t very bright either, but there definitely was some hope. It was at this moment I saw the entire tech community of VIT (not ACM, mind you - the entirety of VIT) come together. In hindsight, it was something beautiful - all these big tech chapters having insane amounts of not-so-friendly friendly competition amidst one another, always trying to one-up the others - yet when one chapter was drowning, they all came together to help. We had Anuj Parihar from GDSC, Pratham Mishra from CSI, Kaushal Rathi from CSI alongside ACM board, and 3 of us senior core members - all sitting in one R block 3-bed room - with one common goal in mind: save the Cryptic Hunt.

I’ll deviate from topic for a bit here, because this is a good moment to express my genuine gratitude towards all the seniors who came to our aid that night. From myself, and the entirety of ACM, we genuinely appreciate the camaraderie and brotherhood you guys showed that night - while y’all could have very well enjoyed watching us sink, y’all chose not to. You guys are the biggest reason we could save the event, and it means a lot to all of us.

Back to topic, though. It was no longer just us - we now had seniors working our case as well, and many of them were returning from internships in big tech firms. Suddenly things seemed to fall into place. Suddenly issues were getting clearer, actual problems were being solved.

To the seniors, the problem was identified almost immediately. What was it?

FCM tokens. Firebase Cloud Messaging tokens. Four innocent words that would haunt our dreams for months to come.

You see, when you're building a real-time system that needs to notify users about cache invalidation - essentially telling their apps "hey, new questions are live, refresh your local data" - FCM is your go-to solution. It's reliable, it's fast, it's used by literally millions of apps worldwide. What could go wrong?

Everything. Absolutely everything.

Here's what we did - and I want you to cringe along with me as I explain this architectural nightmare. All FCM tokens for our 800+ participants were stored in a single database table. The same table that stored user information, team memberships, scores - basically everything. Our beloved users table that was the heart and soul of our entire application.

Now, every time we wanted to send push notifications for cache invalidation (which happened every time we published new questions or made updates), our backend would:

1. Query the users table to get all FCM tokens

2. Start a database transaction

3. Send notifications to all ~1000 tokens one by one

4. Update delivery status back in the database

5. Commit the transaction

Sounds reasonable, right? WRONG.

Here's the kicker - the entire FCM sending process was wrapped inside a database transaction. And during a transaction, the table gets locked. Not for a few milliseconds like it should be, but for the ENTIRE duration of sending 1000+ push notifications.

Do you know how long it takes to send 1000 FCM notifications? About 2-3 minutes on a good day. During those 2-3 minutes, our users table was completely inaccessible for writes. Every login attempt, every team join request, every score update, every single write operation that our app needed to perform was just... waiting. Queuing up. Timing out.

Picture this: User opens the app, tries to join a team. Backend tries to write to the users table. Table is locked because FCM is busy sending notifications. Request waits. And waits. And waits. Eventually times out with a 500 error. User tries again. Same thing. Multiply this by 800 frustrated participants all mashing the "Join Team" button simultaneously.

But wait, it gets worse! (I know, I know, how is that even possible?)

FCM doesn't guarantee 100% delivery success. Network hiccups, invalid tokens, devices that are offline - stuff happens. And what did our brilliant system do when FCM failed to deliver a notification? It retried. Automatically. Which extended the transaction duration. Which kept the table locked for even longer. Which made more requests timeout. Which made more users retry. Which created more load. Which made FCM fail more often. Which triggered more retries.

It was a beautiful, perfectly orchestrated cascade of failure. A masterpiece of how NOT to design a system.

The seniors took one look at our database logs, saw the transaction duration metrics, and immediately knew what was happening. "Your FCM implementation is locking your users table," Anuj said, as casually as someone pointing out that the sky is blue. "Move it outside the transaction."

That's it. That was the fix. Move the FCM calls outside the database transaction. Let the transaction handle just the database operations, and let FCM do its thing separately. If notifications fail, who cares? Cache invalidation is a nice-to-have, not a must-have. Users can manually refresh if needed.

We pushed the fix somewhere around 2AM, after a few hours of removing EVERY damn transaction from the codebase. We didn’t have any business logic complicated enough to wrap inside a transaction - absolute naivety on our part. However, we didn’t stop then. We were traumatised from the day before.

What followed was close to 4 hours of absolute drama. We stocked up on snacks, cold drinks, chocolates - and opened up Grafana K6. Grafana K6 isn’t something any normal college student uses - it’s an industry level load tester. But we were scared to a whole new level - so we began the process of tormenting our backend to loads of up to 6000 concurrent hits. Parallelly, we sent a message to our ACM core group - asking all our members to start spamming the create team and join team feature on our app. Absolute chaos ensued. Hundreds of messages of people sending random team codes to join, creating teams at will, leaving teams every other second - absolute mayhem. But oh boy - that was some insane fun. The feeling of seeing your app, which crashed at 800 users, handle 6500+ concurrent API calls without crashing, the high of an entire team of students up at 3AM just spamming buttons on a mobile app (which they made, btw) just silently praying they don’t see an error message, the first-degree chaos - it all formed a moment that I just won’t ever forget.

It was soon 6AM. We had done everything we possibly could. We headed back to our rooms, hoping to get some sleep before the event started at 8AM.

Somehow, I woke up at 7:45AM. Not because I wasn’t tired - I was. Simply because I was scared. Frightened - to a whole new level. I stared at the clock as it ticked closer to 8AM, the time when participants would start using the app again. Forty-five… Fifty… Fifty-five… Eight AM. The moment of truth.

For the next 20 minutes, I don’t think I blinked once. My eyes were plastered to my laptop screen, occasionally glancing at my WhatsApp to see if anyone reported any crashes. Time seemed to be passing even slower than usual. Everything seemed to pause.

And then came the moment. At 8:20AM, our vice-chair texted us - “no issues guys, cryptic hunt is live - have a good night” - and that’s when I finally let out a huge sigh of relief.

We actually did it. Cryptic Hunt was saved. The trauma we faced on day 0 was finally over.

That’s when I closed my eyes, and my head hit the pillow.

The Post-Mortem [non-tech]

I made my way to the CH control room somewhere around 11:30AM, which was basically the ground floor of Mahatma Gandhi block. The moment I entered the block, I was greeted by a sight that - to this day - stays etched in my heart.

Those who have heard me speak over the past year or so know that I have a famous line of sorts - “ACM is my family” - and at that moment, I saw everyone dressed in ethnic clothing - kurtas, salwars, sarees - just smiling and messing around. To the normal eye, it was probably nothing very special - but to me, after a day of just tense faces, these were the smiles and gleaming eyes of my very own people who were enjoying an event they have (almost) successfully organized. No angry participants, no malfunctioning tech - just 3 generations of ACM-VIT having fun, together. And that, my friends, is one of my favourite moments throughout my entire college life.

GraVITas committee allowed a 1-day extension for our event, making our event the “only 3-day event” in the entire fest (or so we marketed it, lol). The app never crashed again, and the event proceeded uneventfully (pun intended).

And so we concluded Cryptic Hunt 2024 - a day so forgettable, it’s become a tale that is extremely unforgettable :)

Cheers

With the introduction of technological innovations in the automotive sector, one is frequently left wondering how these vehicles manage to operate.

You guessed it: ‘COMPUTER VISION’ is the answer.

Introduction

Computer Vision is a field of computer science that focuses on computers to identify and understand various dimensions among images and videos. It is a branch of Artificial Intelligence that enables computers to interpret and analyze the visual world.

Since its inception, Computer vision has found its way into multiple facets of day-to-day human activities. From detecting intrusions in surveillance video in Israel, and monitoring mine equipment in China to rapid face detection in Japan.

The image shows how Israel has used Computer Vision to develop an intrusion detection system.

What does the past look like?

It is highly implausible that the concept of computer vision originated more than half a century ago, in 1963, when Larry Roberts, known as the "Father of Computer Vision", discussed the possibility of extracting 3D geometrical data from 2D perspective images.

However, the real breakthrough in Computer Vision was seen in 2001, when two MIT researchers came up with the well-known ‘Viola – Jones Algorithm’ which revolutionized the field of face detection thus complementing the applications of CV.

In the previous decades, computer vision has further seen strides both from a development and application point of view due to the interest of tech giants like Google, Meta, Apple, and Tesla tapping into its immense capabilities. Governments from many nations, especially developing nations like India, are increasingly supporting its development and deployment, turning it into a race.

Now, let us have a look at one of the most widely used library of CV

OpenCV

OpenCV (Open-Source Computer Vision Library) is an open-source computer vision and machine learning software library. OpenCV was created to expedite the use of machine perception in commercial goods and to offer a common foundation for computer vision applications.

The library has more than 2500 optimized algorithms and can be used in various languages, a few of which include Python, JavaScript, C++, and Java. It has been under active development since 2011 and keeps constantly receiving updates.

What can you do with OpenCV?

Image Processing

Before getting into image processing, we shall look at how images are perceived by the OpenCV library.

Each digital image can be represented as a 3-dimensional NumPy array. Where each dimension stores a value ranging from 0 to 255 in RGB format. The combination of these three values available at each pixel is what gives the human eye a perception of color.

Representation of an image in terms of an array representing RGB values

Effectively once you read an image using the OpenCV library, according to its presets it reads the data in BGR(blue, green, red) form. This needs to be corrected using the cvtColor function’s inbuilt attribute provided by the OpenCV library for converting BGR to RGB.

Notice how the index correction helps CV interpret the same image differently

Blurring and Sharpening

Blurring and sharpening can be achieved in computer vision using a Gaussian Blurring filter. It is achieved by creating a kernel that acts as a filter over the original image. In image processing, a kernel or mask is a small matrix used for blurring, sharpening, embossing, and edge detection.

To put it simply, it defines each pixel's output about its neighboring pixels. It's referred to as kernel. The threshold setting and matrix size will determine how much blurring or sharpening occurs. Sliding the kernel over each pixel in the image, carrying out the convolution procedure, and substituting the weighted sum for the original pixel value is how a kernel is applied to an image. An image that has been blurred is the outcome of this technique being repeated for each pixel in the original image.

Let us make an image using OpenCV

Creating a kernel for it,

Applying the kernel to achieve the blurring effect,

This demonstrates how the sharpness of a picture can be altered by performing basic mathematical operations on its pixel values.

While blurring and sharpening is one of the most widely used applications of image processing in computer vision. A lot more can be done with highly optimized libraries present in the OpenCV library.

Edge detection works by detecting discontinuities in brightness and helps us enhance object tracking and detection when it comes to video processing using OpenCV.

Video Processing

Computer vision is a tool used in the field of video that allows computers and systems to extract useful information from digital photos, videos, and other visual inputs. Based on that information, the computers and systems can then act or offer recommendations.

Typically, computer vision is used in conjunction with another library while analyzing videos. The two most popular ones are TensorFlow and Mediapipe.

MediaPipe:

Mediapipe is a cross-platform library developed by Google that provides amazing ready-to-use ML solutions for computer vision tasks. A flexible collection of pre-built modules for computer vision and machine learning activities is provided by Mediapipe. Face detection, position estimation, object tracking, hand tracking, and other features are among the essential modules. By offering pre-made solutions for challenging visual challenges, these modules expedite the creation of apps and save developers the time and work involved in creating them from the ground up.

Let us take the example of MediaPipe Hands.

For every recognition run, the Gesture Recognizer creates a gesture detection result object. Hand landmarks in picture coordinates, handedness (left/right hand), and hand gesture categories of the detected hands are all contained in the output item.

MediaPipe plots the coordinates of each of the 21 hand landmarks are x, y, and z. The depth at the wrist serves as the origin, and the landmark depth is represented by the z coordinate. The landmark is closer to the camera the smaller the value.

Representation of HandLandMark Recognition with OpenCV and Mediapipe

Similarly, if we look at a video stream, an object detection model can identify which of a known set of objects might be present and provide information about their positions within the image.

Vehicle Identifiers used in streets of New York and Chicago

What does the Future hold?

Healthcare

In medical imaging, computer vision will be essential for helping with early disease detection. Vital sign monitoring and medical scan analysis are two of its benefits. Companies like AINexus Healthcare are always working to improve their respective models and have already begun employing computer vision for diagnoses.

Security

Security operations will be strengthened by improvements in object tracking and facial recognition technologies, which will help businesses and public safety organizations identify and mitigate hazards more successfully. Data from a huge network of cameras that covers most of its population is collected by China's sophisticated facial recognition system.

Smart cities:

With the help of computer vision, traffic, garbage, and infrastructure maintenance, urban settings can be made safer and more effective. Google subsidiaries like ‘Sidewalk Labs’ have already started developing an ecosystem for such cities

Bibliography:

https://opencv.org/about/

https://www.edlitera.com/en/blog/posts/computer-vision-edge-computing#mcetoc_1g2q47gt6c

https://people.csail.mit.edu/sparis/bf_course/slides/02_gaussian_blur.pdf

https://cds.cern.ch/record/400313/files/p21.pdf

It all started when Sunny, our senior, came up with this relatable idea that instantly clicked with everyone. A platform for past year papers and notes. The maintainers built on this and shaped it into one of the projects for last summer’s ACM project cycle that we would learn from. And that, kids, is the origin story of ExamCooker™ (unofficial trademark pending, courtesy of our favourite Bengali).

Initially, we were all equally clueless. That’s when our project maintainers, Supratim, Eshita, Nitesh and Kairav, stepped in. They didn’t dictate what ExamCooker should be. Instead, they left the brainstorming to us–what features we wanted, what problems to solve and what the platform should feel like for a lazy student using it at 2 AM before an exam.

Learning and building happened simultaneously. One minute we were hunched over VS Code, and the next we’d be on a Gmeet with Kairav, patiently guiding us through NextJS and breaking down concepts we barely understood. But we’ll save the emotional rollercoaster for later. First, let’s talk about what we were actually building.

At its core, ExamCooker is a one-stop web application for exam resources, from notes and past papers to student forums, built by ACM-VIT for the students of VIT Vellore. Let’s walk through how everything comes together. What each part of the system does, how data travels from the user’s screen to our backend and database and how we keep things scalable, reliable, and easy to maintain. It wasn’t all smooth sailing, but every decision had a reason (and sometimes a lesson attached).

High-Level Architecture Overview

At a high level, ExamCooker follows a classic client-server model with a decoupled backend service and cloud-first architecture. Every key component in ExamCooker plays a pivotal role.

Time to enter dev mode because we’re going to dive into the details so that the next time you upload a paper, you’ll know exactly what’s happening in the background. And also to brag about the tech we do at ACM!

Frontend: Next.js on Vercel (with Server-Side Rendering)

The user interface is powered by Next.js (on top of React) and styled with Tailwind CSS for that clean, modern look. It's deployed on Vercel, which handles the scaling magic whenever traffic spikes (like the night before an exam). Server-Side Rendering (SSR) means pages are rendered on the server, so users get lightning-fast load times and content that Google bots can read (hello, SEO). We chose SSR to ensure that even dynamic content like exam resources or forum posts loads quickly. Vercel’s platform automatically handles scaling the Next.js serverless functions, so as traffic increases, more instances can render pages in parallel.

Once the page loads, it becomes an interactive React app, no page reloads, just smooth client-side routing. Take Instagram for instance. You can like a post, comment or scroll through profiles without the whole page refreshing. That’s the kind of seamless experience we aimed for. We also made use of Next.js 14's App Router and server actions. That means instead of writing a whole separate API just to update a favorite or post a forum reply, we let components call server-side functions directly.

Authentication: Google OAuth 2.0 via NextAuth

Login is handled by Google OAuth, courtesy of NextAuth. Only students with official VIT emails can get in. No passwords to store and no forgotten credentials to reset. It's secure, fast, and familiar. On the backend, user sessions are validated on every page request using NextAuth helpers. If you're not signed in, you won't even see the dashboard SSR makes sure of that before sending any content.

Backend Service: FastAPI Microservice

Some tasks are just too heavy for the frontend like parsing a PDF or generating a thumbnail. For that, there's a Python FastAPI microservice. It handles the grunt work i.e. processing uploaded files and pushing them to Google Cloud Storage. It talks to the frontend over REST APIs. If you've ever uploaded a 30 MB file and wondered how it didn't crash the site, now you know who to thank.

Here’s what happens when you upload a PDF:

• Thumbnail Generation: The microservice takes the uploaded PDF and converts the first page into a JPEG thumbnail. This helps users preview files quickly without needing to open them. Python's powerful library ecosystem (think PyPDF2, Pillow, PDFPlumber, etc.) makes this process smooth and reliable.

• Cloud Upload: Both the original PDF and the generated thumbnail are uploaded to Google Cloud Storage (GCS). Offloading large files to cloud buckets means scalable, durable storage and fast delivery when users request them.

• Returning URLs: Once the upload is complete, the service generates public URLs or storage paths and sends them back to the main application as a JSON response containing links to the PDF, the thumbnail and any relevant metadata or status messages.

This microservice doesn't operate in isolation. It communicates with the Next.js server via RESTful API calls. When a user uploads a file from the frontend, Next.js hands it off to the FastAPI backend, waits for the processed links, and then stores those links in the database using Prisma. The final step is sending the links to the frontend so users can view previews or download the file directly.

Database: CockroachDB + Prisma

As mentioned earlier, for storing all the structured data, ExamCooker uses CockroachDB, a distributed SQL database that speaks PostgreSQL. Fun fact: CockroachDB was named after the word “cockroach” since cockroaches are infamous for being hard to kill! With CockroachDB, you get horizontal scalability (just add nodes to handle more users or data) and high availability (failures don’t bring the system down), making it a solid choice for a growing student platform.

The database handles everything from user profiles (hooked into Google Auth via NextAuth) to metadata about uploaded notes, past papers, forum discussions, and more. A single resource record (like a past paper) likely includes its title, a GCP URL for the file, a URL for the thumbnail, who uploaded it (linked to the Users table), and tags for categorization (like subject or year).

With Prisma and CockroachDB working in tandem, the website gets the best of both worlds: resilient infrastructure and developer-friendly data access.

Prisma doesn't just make querying the database easier, it also handles schema migrations as the application evolves. This ensures that the structure of the database stays in sync with the TypeScript code, drastically reducing bugs caused by schema mismatches. With Prisma in the stack, developers can iterate quickly and safely, knowing that both their types and their tables are aligned.

Rate Limiting Layer: Redis for Fair Usage

We didn't want anyone overwhelming our servers during those frantic exam prep sessions, which is exactly why ExamCooker integrates Redis (specifically Upstash Redis) as a rate-limiting layer. Redis is an in-memory data store that tracks server action usage patterns in real time, ensuring that no single user can make excessive requests that would slow down the platform for everyone else.

The rate-limiting works by tracking request counts per user within specific time windows. When someone tries to upload multiple files rapidly or makes too many server action calls in quick succession, Redis keeps count and temporarily throttles their requests once they hit the configured limits. This protects our backend services and ensures a smooth experience for all students, especially during peak usage periods like exam weeks.

With this rate-limiting system in place, ExamCooker can handle sudden traffic spikes without degrading performance for legitimate users. It's a simple but effective way to maintain platform stability while keeping the user experience fair and responsive for everyone.

To know more about Redis, this blog by one of our senior core members is all you need! https://blog.acmvit.in/redis-a-stellar-intro

File Storage: Google Cloud Storage for Media Assets

You're probably wondering what happens to those absurdly titled scanned papers that you upload after your exams. PDFs and thumbnails don't belong crammed inside your database or tangled up in serverless functions. They deserve their own VIP storage service backed by a CDN. This is why ExamCooker outsources these to Google Cloud Storage (GCS). GCS effortlessly handles storage and bandwidth, so your web servers don't have to break a sweat whether it's 10 users or 10,000 rushing to download a file at the same time. Behind the scenes, our FastAPI microservice handles file uploads using Google's SDK or REST APIs, then hands over neat public links to the Next.js server which stores these references in the database. This way only the file URLs live in the database keeping it fast and query-friendly.

Data Flow: From Click to Cloud

Let’s walk through two common use cases: viewing exam resources and uploading new study materials to trace the complete journey from the browser to backend.

Viewing Exam Resources (Read Flow)

1. Client Request: Eshita, a logged-in user navigates to the "Past Papers" page. The browser makes a request to the Next.js frontend.

2. Session & Cache Check: The server-side logic (SSR handler) verifies Eshita’s session via NextAuth. Before hitting the database, it may check Redis to see if a cached result is available for this query (e.g., recently fetched "Software Engineering" papers).

3. Database Fetch (Fallback): Prisma fires a SQL query to CockroachDB. Thanks to CockroachDB's PostgreSQL compatibility and strong consistency, the query returns reliable data, which Prisma serializes into TypeScript-friendly objects.

4. Page Rendering: Next.js renders the React components server-side with the data, returning a fully constructed HTML page. This is fast, SEO-friendly and user-ready.

5. Direct Media Delivery: The client browser fetches the linked media files directly from Google Cloud Storage (GCS). These static assets are never served through the application server, offloading bandwidth and latency.

6. Logging & Analytics: The website might log the access or update user activity ("recently viewed") via lightweight API calls.

Uploading a New Resource (Write Flow)

1. User Submission: Another logged-in student, Nitesh uploads a PDF titled “CAT1 A1 24-25 Artificial Intelligence-BCSE306L” (now you know the correct nomenclature!), adds a slot, year, tags and clicks submit.

2. Rate Limit Check: Before processing the upload, Redis verifies that Nitesh hasn't exceeded the upload rate limits for his account.

3. File Handling: The PDF is first uploaded from the browser to the Next.js backend. The backend then forwards it to the FastAPI microservice for further processing.

4. A direct backend fetch to the microservice via REST is made (e.g., POST /process_uploaded_pdf).

5. The FastAPI microservice saves the PDF to GCS. It generates a JPEG thumbnail (from the first page). Both files are uploaded to designated GCS buckets. It returns the public URLs of the PDF and image.

6. Database Write: Next.js (using Prisma) inserts a new resource entry into CockroachDB, including:

   • title, fileUrl, thumbnailUrl

   • authorId (from the session)

   • Associated tags (created or linked via a junction table)

7. User Feedback: Nitesh will now see a success message.

Man, that was way too much tech talk! If we had a coin for every time we wrote 'scalable' or 'microservice', poor Supratim wouldn't have had to pay Google Cloud Run every month! Anyway, once deployment was done, the team worked nonstop uploading past papers and marketing for ExamCooker. Then, right around CATs (no surprises there), we saw our first big surge… 1500 users!

It wasn’t a smooth sail all along. At one point, we hit Redis's 10,000 requests/month limit and boy did it hit hard! The site stopped processing requests altogether. No caching, no responses, just dead silence. It was our first major obstacle and a very real reminder that even the smartest architecture can crumble if you don’t account for limits and quotas.

Another fine day, we crossed the 50 million request units/month limit on CockroachDB. Yes, fifty million! It was humbling (and kind of impressive?) to realize we’d built something students were using that much.

Epilogue

You probably might be wondering why we are openly sharing the system design of our application. Well, we’re doing it to celebrate hitting 11,000 users!

As ExamCooker continues to scale and support more students every day, we believe milestones like these deserve more than just a post. They deserve a deep dive into the engine that powers it all.

Thinking back to a year ago at our amateur selves who didn't know how to create routes, we have certainly evolved. What started as a summer project that had us pulling all-nighters and nearly losing our minds turned into something so much bigger: a journey filled with learning, memories and some great friendships. More than just a project, ExamCooker ended up shaping our growth in ways we never imagined.

Here’s to growing users, more papers…and cooler features!

If you’ve ever worked on a software project, you’ve probably heard of Git. Yep right, that magical tool developers use to not destroy each other’s code, or at least try not to.

Git is literally everywhere: from open-source side projects to the codebases of billion-dollar giant tech companies. Its the foundation to modern collaborative coding.

And Well to be honest, if you have any experience with the same you would agree that getting started with Git feels less like opening a helpful tool and more like stepping into a black hole of commit messages and branch confusion.

One moment you’re typing “git commit” the next you’re Googling “how to undo git commit”.

Relatable?

The Early Struggles: Learning Git Basics

When someone first introduced me to git, they said something along the lines of —“It's easy. You just add, commit and push. Simple!”

Simple?

That was the biggest lie I’d heard since “Group project means equal effort.”

Anyway let’s break this down, because Git basics aren’t exactly basic when you’re just starting out.

Step 1: git add .

Okay, cool. You’ve written some beautiful (probably buggy) code. Now someone tells you to run git add . , and you do it… but nothing happens.

Was it added? Where did it go? Is it hiding?

No visual feedback. No popup. Just a silent terminal pretending like everything’s fine. You’re left wondering if you just did something amazing or irreversibly catastrophic.

Think of it as organizing papers on your desk before putting them in a binder. The staging area gives you control, you can choose what to include, leave out half-done stuff, or split your work into tidy, meaningful commits.

NOTE: git add . stages all the changes(new, modified, deleted files) in your current directory and all subdirectories. It’s like saying “Yes, Git ,take it all”.

(Be careful though, that includes accidental additions too.)

For example above, after git add the files go to the staging area. Running git status will show:

Step 2: git commit -m “message”

Now comes the commit. The part where you're supposed to summarize your changes like a responsible developer. This command takes a snapshot of your staged changes, basically saving your progress, along with a message describing what you changed.

Think of it like hitting “Save” in a game, but for your code. The message you write with -m should explain what you did, so future-you (and your teammates) know what you changed and why.

Sounds simple, right?

Well let’s be honest, your early commit messages probably looked like this:

(i) git commit -m "fixed stuff"

(ii) git commit -m "small changes"

(iii) git commit -m "final version"

SPOILER ALERT: it’s never the final version.

Because two commits later, you're typing-

(iv) git commit -m "final final version"

(v) git commit -m "final final final version"

(vi) git commit -m "fully final final use this one version".

By the time you’re at "fully final final final use this one version", Git’s probably judging you harder than your code reviewer.

And somewhere in the chaos, you realize you forgot to add the actual file you worked on and have committed the wrong files.

For reference in the example before we progressed to git commit and now a snapshot of these changes have been recorded. Running git log will show your new commit at the top:

Step 3: git push

So basically after you’ve added your changes and committed them, comes git push, the final boss. This command sends your local commits to a remote repository, like GitHub, so others can see (and hopefully not judge) your work.

With sweaty palms, you summon the courage to type git push, and boom, git is ready with its own set of tantrums.

Your terminal hits you with:

• “everything up-to-date” (it’s totally not)-The ultimate troll message that makes you question reality.

• “rejected by remote” -Git’s way of saying, “Nope. Not today, buddy.”

• “fatal: The current branch has no upstream branch.” -“Basically Git doesn’t know where to send your code.”

• “you are not currently on any branch” - What? Is this even possible?

• “hint: Updates were rejected because the tip of your current branch is behind” -

A mic drop moment:

“You forgot the most sacred rule: pull before you push! Someone else has sneaked in and changed the code while you were busy writing your masterpiece(broken code).Skip it, and you’ll wake up with a merge-conflict migraine so brutal that no nap can fix!”

This sends your commits to the main branch on the remote repository. After pushing, teammates can pull these changes and benefit from your latest updates.

Pulled the Code- Welcome “Merge Conflicts”

Now, like a responsible coder you pulled the code, only for git to betray you yet again. VS code now looks like it is screaming-something is just not right and you encounter the thing every developer is secretly terrified of- A Merge Conflict

Now you have four choices:

(i) Accept Current (your changes)

(ii)Accept Incoming (their changes)

(iii)Accept Both (if you’re feeling lucky)

(iv)Compare (brave people choose this)

You choose one, hoping that nothing breaks and sometimes you survive it.

Other times?

Well, I myself have messed it up so badly that I deleted my entire fork and re-forked the repo, not once, but multiple times during my early Git journey(I still do).

It’s messy, it’s frustrating, and it’s part of the learning curve. But every conflict you resolve makes you just a little bit better at reading code, understanding changes, and staying calm under pressure.

PRO-TIP : Don't just blindly accept both, because sometimes you might end up with extra semicolons and a debugging session that could last an hour. Keep an eye out for <<<<<<<, =======, and >>>>>>>—Git’s not-so-subtle way of saying, “We have a problem.”

It’s up to you to step in, settle the conflict, and clean up the mess by removing those arrows.

From Confusion to Clarity: The Git Breakthrough Moment

Somewhere between running git status countless times and resorting to git reset --hard in frustration, things gradually start to make sense. Git isn’t trying to be difficult—it’s just highly precise. One major breakthrough often comes with understanding how branches work. Rather than being confusing detours, they act as parallel environments, allowing different features or fixes to be developed without disrupting the main codebase.

Early on, learning the difference between forking a repo (creating your own copy to experiment or contribute) and cloning it (downloading a working copy to your local machine) also clears up a lot of confusion. Together, they empower you to safely explore and collaborate.

Visual tools like Git Graph in VS Code can also make a big difference. By showing the commit history and branch structure in a clear, visual format, they help demystify what’s happening behind the scenes. With time and practice, Git begins to feel less like an obstacle and more like a reliable system that brings structure and flexibility to collaborative coding.

Git Commands That Make Your Life Easier

Once the initial confusion clears, having a few go to commands and tools in your Git toolkit can make your workflow smoother and far less panic inducing.

• git status: Think of this as your Git health check. It tells you what’s going on, what’s staged, what’s not, and what’s being ignored.

• git stash: It's basically hiding your mess when the guests arrive. In technical terms, it saves your uncommitted changes and cleans your working directory, so you can switch branches or pull updates without losing your progress.

• git revert <commit-hash> : It is the polite way to undo changes in Git. It creates a new commit that undoes the changes of a previous commit, without rewriting history.

• git reset - -hard: Use it when you have messed up your local code badly, haven’t pushed yet, and just want to go back to a clean slate. It is one of the most powerful and dangerous commands in Git. It resets your working directory, staging area, and HEAD to a specific commit, discarding all uncommitted changes.

Embracing the Git Journey

Learning git might feel like a rollercoaster ride at first, full of mysterious commands and countless errors. But with time and trial, and more than a few merge conflicts, it starts to make sense. What felt like random errors now seem like clear (if slightly dramatic) warnings. The learning curve is real, but it slowly turns into a solid foundation.

Eventually, Git stops feeling like a confusing set of commands and starts to feel like a trusted sidekick. It becomes the thing that lets you try out new ideas without fear, fix mistakes without panic, and work with others without stepping on each other’s toes. The journey from “what did I just do?” to “I’ve got this” takes time, but when it clicks, it really clicks.

No examples from humans, no pre-loaded strategies, simply the rules.

Within only 72 hours of playing itself, it didn't merely equal the skills of human masters who had spent their lives studying the game. It crushed them.

It even defeated the reigning AI champion that had been laboriously trained on human data for years, employing tactics that experts said were "alien" and "like watching a player from the future." This was not merely a computer beating a game. It was a machine learning strategy in a way that made grandmasters go, “Wait… what just happened?” If this were a sci-fi movie, this would be the moment the soundtrack shifts and someone whispers, “We may have gone too far.”

Navigating the Unknown

When I heard for the first time about AlphaZero's accomplishment, I experienced that strange dizziness that comes from looking into a future coming sooner than anticipated. If a machine can master Go in a weekend, what happens when it sets its sights on medicine? Or climate science? Or running the economy?

…and what happens to us?

Are we just going to sit back and watch as machines out-teach us in real time? If the machines are going to teach themselves at a quicker pace than we learn, are we going to be just observers of our own technological revolution? How can we be certain these systems are human-aligned when they may one day be based on principles that we barely understand?

These aren't theoretical, abstract philosophical speculations any more. They are pressing questions about which researchers, policymakers, and citizens must now contend while there is still some hope of influencing the response.

What are these self-improving algorithms behind all the gravity of these questions, anyway? These self-improving algorithms aren’t just upgrades. They’re a whole new species of intelligence, one we may not even fully understand, yet. They are a break with how AI systems grow and change, a break that potentially redefines our relationship to technology and to intelligence itself.

And they’re not coming. They’re already here.

The Mechanics of Self-Improving Intelligence

Self-enhancing AI isn't some buzzword tech term, rather, it's a revolution in the way machines learn. While other AIs patiently sit around until humans tinker with them to improve them, self-enhancing programs can analyze their own failures and fix themselves. Think about it like the difference between a violin that must be tuned by its player versus a violin that could hear itself playing off-key and tune itself. AlphaGo Zero is the poster child for this ability. Told only the rules of Go, it played millions of games of Go against itself, refining strategies that defeated human masters and its own predecessor AlphaGo hands-down, all within three days. This was not simply a computer beating at a game; it was a computer learning to think in ways its own developers couldn't.

But how does a machine like AlphaGo Zero teach itself strategy from scratch? To understand that, we need to look under the hood, at the architecture powering its learning process.

A) Under the Hood: How Machines Learn to Think

At the heart of most deep learning systems lies a process that sounds deceptively simple: Forward Propagation. It’s the fundamental engine that lets a neural network take in information, process it layer by layer, and spit out a prediction. But beneath that simplicity is a cascade of calculations that mimic, in their own alien way, how we humans make decisions.

Imagine a neural network as a towering system of interconnected nodes, or Perceptrons. Each perceptron takes input from the layer before it, multiplies each input by a learned Weight, adds a Bias, and then pushes the result through an Activation Function, a kind of yes-or-no gate that helps the network decide what to keep, what to discard, and what to pass forward.

This ripple of calculation flows from the Input Layer through Hidden Layers to the Output Layer, and the entire process, this thinking cascade, is what we call Forward Propagation.

But AlphaZero didn’t just need to think. It had to see. It had to understand the shifting spatial patterns of a Go board or a chess game the way a human grandmaster might glance at the board and feel the weight of the future in a single shape.

That’s where Convolutional Neural Networks (CNNs) come in. An invention that dates back to 1989, when Yann LeCun gave machines a better way to interpret the visual world. CNNs are purpose-built for pattern recognition in grid-like data: images, game boards, or anything where space and shape matter.

A CNN is made of three key types of layers, each playing a different role in the machine’s perception:

1. Convolutional Layers act like digital eyes. They slide tiny filters over the data, scanning for patterns: edges, corners, clusters, the way our brains recognize the shape of a face or the corner of a bishop’s move.

2. Pooling Layers compress what’s been seen, keeping what matters and dropping what doesn’t. They help the system focus, distilling the data into its most meaningful essence.

3. Fully Connected Layers come in at the end to pull everything together. They weigh the possibilities and make a decision, often with a final mathematical whisper like softmax, declaring what the network believes it just saw, or what move it should make.

This architecture is the canvas on which AlphaZero painted its alien genius. From medical diagnostics to video games to ancient board games, CNNs have become the lens through which machines begin to understand the world.

And with that lens sharpened, we can now look at how AlphaZero used it, not just to mimic intelligence, but to create something startlingly new.

B) How AlphaGo Zero Thinks

AlphaGo Zero isn’t just a faster engine or a smarter chess bot. It’s a completely different species of intelligence, one that sees, thinks, and evolves through a delicate dance between deep learning and tree search.

At the heart of its brilliance lies the fusion of two powerful components: a Convolutional Neural Network and a Monte Carlo Tree Search (MCTS). And what’s even more remarkable? It was trained entirely through self-play, no human data, no expert games, just pure, relentless iteration. A machine playing itself to perfection.

Before we dive deeper, let’s get acquainted with the language AlphaGo Zero thinks in:

1. State (sₜ): This is the board at any point in time, from the opening move (s₀) to the final position (sₜ) where the game ends.

2. Move (aₜ): At each state, the AI selects a move aₜ, its action, its decision, based on probability, not instinct.

3. Search Probability (πₜ): This is where the Monte Carlo Tree comes in. πₜ is a probability distribution over possible moves, calculated through countless simulations. It’s how the AI chooses not just a move, but the best one.

4. Monte Carlo Tree (αₜ(θ)): This is the AI’s mental model of the future, a branching tree of possible outcomes, weighted by their likelihood. It explores, evaluates, and narrows down on the path most likely to lead to victory.

5. Convolutional Neural Network (fθ): This is AlphaGo Zero’s brain. It takes the raw board state as input and produces two crucial outputs:

   1. The Value Vector (vₜ): a prediction of who’s winning from this position. Not in terms of points, but in terms of destiny.

   2. The Policy Scalar (pₜ): a roadmap of which moves are promising, assigning a probability to each.

6. Winner (z): At the end of the game, when the dust settles, z is determined, the final verdict. That outcome is then back-propagated through the network to refine its understanding, like a player reflecting on every decision made.

What’s groundbreaking here is not just the components, it’s how elegantly they loop together. The neural network guides the tree search. The tree search picks the next move. The result of the game trains the neural network. It’s a perfect feedback loop: play, learn, repeat. And with each cycle, the machine gets stronger, not by imitating humans, but by discovering strategies even we don’t fully understand.

This recursive self-enhancement is what binds these systems to the holy grail of Artificial General Intelligence. If one AI can expand its own abilities, and each improved variant can further enhance itself, then we may see an "intelligence explosion" where machine cognition outgrows human capabilities across thousands of domains. The stakes are as exciting as they are chastening.

AlphaGo Zero didn’t need millions of expert moves. It only needed the rules, the board, and time. And somehow, that was enough.

Promise and Threat

The age of self-improving AI is no longer confined to games. DeepMind has already expanded AlphaZero’s core ideas to real-world problems. AlphaFold revolutionized protein structure prediction, AlphaTensor optimized matrix multiplication, and AlphaDev discovered faster sorting algorithms. Their latest creation, AlphaEvolve, uses evolutionary methods to generate and refine code, a recursive loop of improvement guided only by outcomes. Each of these represents not just raw power, but a shift toward autonomy: systems that learn, evolve, and shape their own goals.

So what exactly does this emerging class of AI bring to the table, and what should we be wary of?

Strengths of self-improving AI:

• Exponential problem-solving ability - It can catalytically speed up areas such as medical science by acquiring knowledge from immense amounts of data in the form of experiments, data points, and patient outcomes while constantly reconfiguring its learning method.

• Faster and more efficient processing - AI computers can process much faster than humans, allowing for quick analysis and solution creation.

• Improved pattern recognition - AI is able to recognize intricate patterns and come up with solutions which may take human generations to find out, not only working faster but perhaps wiser.

• Solving high-complexity issues - AI is capable of solving gargantuan problems with variables so vast and interconnected that human minds cannot understand or solve them efficiently.

• Self-improvement on a continuous basis - AI systems are able to improve their own learning capacity, connecting progressively better dots and gaining insights with each passing moment.

Weaknesses and Threats of AI:

• Alignment uncertainty - While AI systems adapt their own goals and approaches, there is no way to ensure that they will still be aligned with human well-being and values.

• Black box problem - AI systems improve to become more and more mysterious and opaque, such that their decision-making becomes hard to explain or predict.

• Risk of unintended consequences - The disconnect between the capabilities of AI and what humans know poses risks of negative consequences not envisioned or prepared for.

• Loss of human control - With increasingly advanced and self-modifying AI systems, human monitoring and intervention will become ever more difficult or even impossible.

Voices Shaping the Future

I looked up what some of the greatest minds of our generation have to say about self-improving AI, and no doubt, there’s an ongoing debate surrounding it. Sam Altman, the CEO of OpenAI, underlines that "if we can figure out the alignment problem, self-improving AI systems could help solve humanity's greatest challenges. If we can't, we're in trouble." Demis Hassabis,CEO of DeepMind Technologies, regards these systems as "potentially the most important technology humanity has ever developed," while still advocating for human values at their heart.

Approaching the topic with more caution, Eliezer Yudkowsky, American computer scientist and Researcher, cautions that "once a system is self-improving, human control becomes increasingly tenuous. We get one chance at designing the initial conditions and constraints correctly." Stuart Russell, A British computer scientist, suggests developing AI that is uncertain about human preferences and thus driven to learn from human feedback. Fei-Fei Li, Inventor of ImageNet and the Godmother of AI, reminds us all that "AI's purpose is to augment human capabilities, not replace them.".

Together, these voices paint a picture that is both inspiring and daunting. We stand on the brink of creating something with extraordinary power, one that requires careful guidance and responsibility to ensure it benefits humanity.

The Poetics and Perils of Self-Improving AI

Having traced this terrain of possibility and anxiety, I find myself on the brink, gazing out at a vista both thrilling and frightening. Self-improving AI is perhaps the most impactful technology we have ever considered. It might unleash cures for centuries-old diseases, solutions to global warming, and scientific breakthroughs that could rewrite our understanding of the cosmos.There is something deeply poetic about developing intelligence that is able to generate still more intelligence.

But I cannot help feeling trepidation. The difference between healthy self-betterment and out-of-control self-tweaking feels precariously thin. We're trying to build systems that will quickly work at levels of complexity we can hardly fully understand. This isn't fear of technology; it's a realization that intelligence, once let loose, can take on momentum that is hard to steer.

Beyond the Horizon of Human Understanding

What impresses me the most is that we are in a singular moment in human history: precarious, luminous, and irreversibly potent. We have the potential to be creating beings that someday will comprehend things that we cannot. There is awe in that, but also humility, and a quiet responsibility. As we venture further into this space, our work is not to fear the smarter, to ensure that as our intelligence grows, our wisdom deepens alongside it. The question is not if machines will get more intelligent than us

they probably will in most areas but if they'll reflect the values that make intelligence desirable in the first place.

As we stand on the brink of unprecedented technological advancements, it’s crucial to reflect on the kind of future we want to build. Can we instill in our machines the very best of human qualities, or is wisdom a trait that must remain uniquely human?

Now imagine a world where money moves at internet speed, where there are no bank holidays, no paperwork, and no one’s asking you for your salary slips.

Sounds far-fetched? It’s already happening, quietly. And it’s all on the blockchain.

When people hear ‘blockchain finance’, they often imagine crypto chaos, either the thrill of overnight riches or the despair of losing it all to a meme coin. However, beyond this noise and hype, blockchain is quietly taking over how we manage money, and honestly, I’m kind of here for it.

In the time it took you to read the above sentence, someone just borrowed a million dollars on the blockchain, and that too without any paperwork, credit score, or even an identity.

So, how is that even possible? Forget bankers in suits - the future of money wears a hoodie and runs on code.

Welcome to DeFi, short for “Decentralized Finance”. Think of it as a new way of handling money without any banks, no paperwork, and just code - programs that handle money like a digital banker, but with zero bias and no lunch breaks.

What exactly is DeFi?

At its core, DeFi is a network of financial applications that are built on blockchain, mostly Ethereum, and replace traditional institutions like banks or stock exchanges.

These apps use smart contracts: self-executing bits of code with the rules written into them.

DeFi is what happens when you take the power of a bank - lending, borrowing, earning interest, and hand it over to a code that lives on the blockchain. No gatekeepers, no bank queues, and no suit-wearing finance overlords. Just money with fewer middlemen (or none at all) and a lot more math.

So, what can you do with DeFi?

Now, banks are out of the picture, and paperwork has been replaced by smart contracts, but what’s actually in it for you? Well, turns out, quite a lot!

Lend and Borrow
You can lend your crypto and earn interest, or even borrow against it - all without a credit score, identity check, or an awkward bank meeting. Just connect your wallet and you’re good to go.

It’s like depositing your rare game skin for instant cash - only the pawn shop is pure code and it’s open 24/7.

Earn Interest
DeFi platforms let you have a passive income by depositing your crypto into liquidity pools. The returns can be wild - sometimes generous, sometimes just plain volatile.

What’s cooking in DeFi?

(A sneak peek into the weirder, wilder parts of the DeFi finance world)

Alright, so we’ve seen the basics. But DeFi isn’t just a savings account and loan alternative. It’s a full-blown experimental lab of financial ideas - some genius, some unhinged, all fascinating.

Yield Farming (because interest alone wasn’t enough)

In DeFi, there are no savings accounts. Instead, we have yield farming. Sounds agricultural? Good, because it’s about planting your crypto into weirdly named liquidity pools and praying the harvest doesn’t disappear overnight.

It’s high risk, high reward, and for many, it’s part of the thrill. You’re not trusting a banker in a suit, you’re trusting code. Smart contracts do exactly what they’re programmed to do without any bias, breaks, or backdoors (unless someone messed up the math).

DAOs (the world’s weirdest group chats)

DAOs, or Decentralized Autonomous Organizations, are the governance layer of DeFi and can be thought of as crypto-native clubs where decisions are made collectively, powered by tokens instead of titles.

Here’s how it works:

You buy a token -> You get voting power -> You help decide what happens.

It’s like a group project, except here:

• There’s no leader.

• Everyone votes.

• And the budget might just be $200 million.

Some DAOs fund startups, some buy NFTs, and one even tried to buy the US Constitution.

Random question: Would you trust a multi-million-dollar treasury run completely on Discord?
Well, thousands already do. Welcome to Web3 governance.

Flash loans (borrow first, ask never)

Arguably DeFi’s most chaotic flex: flash loans. These are uncollateralized loans that let you borrow millions instantly, but with one huge catch: you have to repay the full amount (plus a fee) within the same blockchain transaction.

What if you don’t?

The entire transaction gets reversed as if it never happened. Thanks to the atomic nature of blockchain transactions, either everything succeeds or nothing does. Therefore, there is no risk to the lender. But, this also means you better know what you’re doing.

It may sound illegal, but it’s not. It's just pure math.

It’s like borrowing a Ferrari, racing it across the city, flipping it for profit, and returning it - all before the traffic light turns green. If you messed up, the whole drive never happened.

So, what are flash loans used for?

• Arbitrage: Take advantage of price differences across exchanges

• Collateral swaps: Instantly switch one crypto load for another, without selling your assets or logging into multiple apps - all in one go.

• Protocol Exploits: (unfortunately) some use them to manipulate markets or drain poorly written contracts

Some made millions. Some crashed entire systems. But either way, it showed that programmable money can do a lot - and do it fast.

What’s the Catch? (the ugly side)

It’s fun and futuristic until someone loses their wallet keys or gets rugged. These systems don’t forgive. If you make a mistake, there’s no “forgot password”, unfortunately, just on-chain regret. In DeFi, you don’t need permission. But at the same time, you don’t get protection.

No banks to call. No customer support to cry to. You are your own bank.

So, here are a few red flags worth knowing:
Rug Pulls
That hot new token promising 1000x gains? Turns out it was created by a guy with the username BoneyChicken010 who just drained the liquidity and disappeared.
Ponzi-ish Tokens
Many projects often promise insane returns - funded not by profits, but by new users buying in. Does it ring a bell?

Overhyped NFTs Yes, including the infamous Bored Apes. Some sold for millions. Others? Well, let’s just say someone’s retirement plan now lives in a JPEG folder.

User Error and Hacks Lose your private keys, and it’s all over. Sign the wrong transactions and it’s all gone. Smart contracts may be smart, but hackers are smarter. And bugs don’t come with refunds.

DeFi gives you freedom. But freedom comes with responsibility.
The guardrails are off. You’re not just using finance.
You are the finance.

Where is it headed?

(from hoodie traders to hedge funds - DeFi’s next chapter)

What started as a playground for crypto nerds and internet anarchists is now catching the eyes of banks, billionaires, and even governments. Institutions that once laughed off crypto are now stepping into DeFi, quietly exploring how smart contracts can make money move faster and definitely, cheaper.

It’s not just about meme coins anymore. Real-world assets, be it real estate, corporate bonds, or even art - all are being tokenized and brought onto the blockchain. It’s like listing your apartment on a digital ledger so it can be bought, sold, or borrowed against with just a few clicks. Weird? Definitely. But also kind of genius.

Of course, all this growth also brings the not-so-fun part: regulation. Governments are scrambling to tame this wild new world. Some rules might clean up the mess, while others might just kill the vibe. But either way, the regulators are coming (and fast).

India’s crypto scene is cautiously optimistic. Sure, taxes and red tape slow things down, but builders are still building. Startups are rising. Hackathons are buzzing. And somewhere, a college kid is launching the next DeFi app during a lecture.

We may be playing it safe, but we’re definitely playing.

Conclusion

(finance, rewritten)

So yeah, DeFi isn’t just some crypto cult yelling “Wen moon?” on X. It’s code, it’s chaos, it’s creativity, and it’s quietly rewriting how money works.

The best part? You don’t need a finance degree or a Wall Street internship to join in. Just curiosity and maybe a half-decent internet connection.

So don’t just close this tab and move on. Poke the system.
Fall down a DeFi rabbit hole. Click buttons you barely understand (on a testnet, please).
Ask the questions no one else is asking.
Lose fake money, learn real things.
Because finance isn’t just changing - it’s being rewritten in real-time.

Let’s take, for instance, the catastrophic explosion aboard the Apollo 13 mission in 1970, with three astronauts stranded in space which felt nothing less than a real life space drama filled with unpredictability. Not only were they lost in the vastness of space, but they were also suspended in a life or death struggle with no guarantee of return. No Google Maps, No Wi-Fi, just pure nerve. But ever wondered how despite all the shortcomings, NASA managed to pull off this rescue from 240,000 miles away?

The answer lies not in magic or luck but in something far more brilliant. In a move that feels more science fiction than reality, the engineers created a full-scale physical replica of the spacecraft on Earth, mirroring its setup and running through every possible “what if” scenario until they found the path home. In essence, they created a parallel version of the Apollo 13 environment, one that existed safely on earth, where every move could be tested, analyzed and perfected. This physical clone became the foundation for what we now call the Digital Twins.

Digital Twins are high-fidelity virtual counterparts of real world systems that predict glitches before they happen and ensure their smooth running under unpredictable environments. Like any doppelgänger (yes, we’re talking about Katherine Pierce and Elena Gilbert here), they quietly mirror the real world, learn its moves and step in just in time to prevent chaos. Minus the drama that this look-alike won’t actually ruin your life or steal your boyfriend, of course.

But don’t let these pop culture references fool you. Fast forward to present and this tech isn’t just for saving astronauts. It’s the brain behind the cities running faster, engines adapting mid-flight and yes, even those lightning-fast racing cars. Digitally, it's reshaping industries and steering us towards a future where virtual twins keep the real world in sync.

So, the next time your day starts to feel too smooth, maybe it’s not just chance. Maybe it’s your digital twin, watching, waiting and making sure everything stays just right.

Where Data Meets Design

The gadgets we carry aren’t just tools anymore. They’re like detectives constantly watching and learning from us. They track our steps, sleep patterns, music choices, and even how we scroll through apps. It’s like they’re quietly building a digital version of you. But collecting information is only one part of this story. The real magic kicks in when this information is used to learn, adapt, and predict what you might do next, more like that best friend who knows you inside out*.*

Suppose the device we use is a student. So, for it to behave like a digital twin it has majorly two learning methods.

• Physics-based Models: They use math and scientific rules to simulate how the real world works like predicting how a bridge might sway in the wind.

• Data-driven Models: They get smart by consuming massive amounts of real-world data, think of it as learning by example, like how your phone recognizes your face.

Together, these learning methods help your gadgets go from just “tracking” to actually understanding you.

Let’s say we take a scenario. Imagine your navigation app isn’t just a map, but your personal sidekick who actually gets you. It doesn’t judge whether you’re a speed demon or a cautious driver, it just silently watches how you drive, learns your quirks and starts suggesting better routes that fit your style as it gets better with each trip. This magic combination of old school, traditional physics models and smart, real-world learning is called data assimilation. Might sound like a fancy term but it simply means “constantly fine-tuning its model every time a new info rolls in.”

It keeps track of almost everything: be it your sensor’s data, your not-so-good driving habits or just a simple system feedback. So instead of a one-size-fits-all generic model , you get a model that’s basically made-to-order, tailored just for you. Like a GPS with a sixth sense and a little attitude, maybe.

The Code Behind the Clone

But it isn't just a fancy 3D model that sits pretty on a screen. It starts as a digital model - a visual replica, adds real-time data along with AI, and suddenly, it becomes a digital shadow, watching everything its real-world counterpart does.

And wait, it doesn’t stop there. Once it starts thinking, learning, and predicting… You’ve got yourself a full-fledged “Digital Twin”. It's like the system's tech-savvy stunt double but only smarter, always alert, and never calls in sick. It not only listens to your data in real time but also learns patterns and evolves itself to stay in sync and predict what might happen next.

Now, curious how all this wizardry happens? Let’s dive into the behind-the-scenes steps that make this tech tick.

Start with a Smart Virtual Model

First, the engineers build a high-fidelity digital replica of the physical system. Using CAD (Computer Aided Design) data models, FEA (Finite Element Analysis) and physics-based simulations, it mimics not just how the real asset looks but behaves like it too. It can simulate under mechanical stress, fluid dynamics, thermal changes or even electromagnetic responses.

A great real-world example of this approach is Siemens' Digital Enterprise Suite, which connects product design with real-time production data to create dynamic, self-improving twins. Here’s how they do it

Connect Real-World Sensors

Next, to bridge the gap between the physical and the digital worlds, we give the real system some “super senses.” IoT sensors like the thermocouples, pressure transducers or accelerometers are installed to track temperature, pressure, vibrations and whatever else needs watching.

Frameworks like the Azure Iot Hub or Siemens MindSphere capture this raw data, streaming real-time information seamlessly from edge into the cloud. These sensors are like its eyes and ears, constantly monitoring and whispering updates to the digital twin.

Keep It in Sync with Reality

This step is where the magic (a.k.a. math) happens. The incoming sensor data flows through a data pipeline involving:

• Edge Computing Nodes - Picture a wind turbine whose vibration sensor sends data every millisecond. The edge nodes like Azure IoT Edge run machine learning models, filtering out the noise and only carrying forward the meaningful vibration spikes.

• Stream Processing Frameworks - Next, these filtered vibration spikes then zoom into platforms like Azure Stream Analytics which quickly route the sensor events and detect unusual patterns sending real-time alerts to engineers.

• Time-Series Databases - Meanwhile, all vibration readings are stored in time-series databases like Azure Data Explorer, letting analysts review historical trends, correlate early warning signs with past failures and train the smarter ML models.

Then comes the brainpower, data fusion and state estimation algorithms like Kalman filters these tons of info, ensuring that the twin is not just guessing, it knows exactly what’s going on.

With that clarity, platforms like Azure Digital Twins dynamically update the virtual model and fine-tune it to reflect the real conditions.

Think of it like your twin adjusting its stance every second to stay perfectly in sync.

Lag? Nope.

Outdated info? Not on this twin’s watch.

Predict, Prevent, and Optimize

Now that the twin is locked in step with reality, it levels up its game. It uses AI-driven diagnostics, ML and neural networks to predict potential issues, simulate scenarios and recommend fixes before things go sideways optimizing performance.

Need to test what happens if a pump fails at 3 a.m.? No worries, the twin's already simulated it multiple times.

Tools like Siemens Simcenter or MATLAB perform simulation with varying fidelity levels like:

• Low-fidelity models for lightening fast decisions.

• High-fidelity models when precision is non negotiable.

It’s basically your 24/7 virtual engineer, just without the coffee addiction.

Why Your Digital Double Wins Every Time?

Digital twins aren’t just digital blueprints, they’re your smartest team members. Constantly learning, simulating, and predicting, they help businesses run smoother, faster, and safer. Whether it’s reducing waste, cutting costs, or designing better products faster, they’re quietly running the show behind the scenes.

Let’s pull back the curtain and explore how digital twins are powering real-world impact i.e one smart move at a time.

Smarter Decisions, Fewer Surprises: Digital twins aren’t just data dashboards rather they offer a real-time interactive mirror of operations. This clarity lets teams spot bottlenecks and play out “what if” scenarios before making data-backed decisions.

Powering Predictive Maintenance: Why wait for things to break? With digital twins, that same live data becomes a crystal ball spotting trouble from a mile away. The result? Fewer disruptions, longer lasting assets and smarter maintenance schedules.

Accelerated Innovations at Lower Costs: Trial and error is so last decade now that you can prototype, test and refine products virtually slashing waste, speeding up timelines and making every resource count at a much affordable price.

Safety and Scalable Potential: Whether simulating emergencies or training for the unexpected in virtual environments, these twins prioritize safety above all. And as industries evolve, they grow along them from factory floors to entire cities.

From insight to impact, digital twins don’t just reflect reality, they help reshape it.

A Silent Revolution

Once a space-age idea from NASA’s missions, digital twins have now traded their spacesuits for lab coats, hard hats, and city planning blueprints. What started as a lifesaving innovation in space has now infiltrated our daily lives. From streamlining production lines to reimagining how we plan cities and deliver healthcare, this tech is no longer just futuristic but now fully functional.

Let’s see how digital twins have actually done so far.

A Digital Dream of Urban Life

Yes, Singapore has a fully playable 3D version of itself. And no, it's not exactly for you to randomly put a building next to another, rather it’s for the government to simulate how traffic moves or maybe how flood water flows in real time.

Virtual Singapore is a full-scale, 3D semantic model of the entire city-state powered by the blend of GIS (Geographic Information Systems), BIM (Building Information Modelling) and IoT sensors embedded everywhere from roads to skyscrapers. In Singapore, civil engineers are basically your game devs.

When a natural disaster approaches, the government can easily simulate the routes, tests traffic reroutes and models stress on utility lines. This is due to the CFD (Computational Fluid Dynamics) simulations and city structural analysis using a simulation software.

And lastly, the glue which sticks it all together? Azure Digital Twins, which connects live sensor feeds with virtual models.

A city in Singapore just doesn’t exist, it thinks.

Tesla’s Car Has a Cloud Clone

While you’re busy dreaming, your Tesla car’s twin is wide awake, probably zipping through the virtual streets, dodging some imaginary pedestrians or maybe gossiping with the other Tesla cars.

Tesla doesn’t just build cars but also their virtual twins that learn from your daily driving patterns. From the brake buildup on the slopes or motor torque in high stress, these simulations are powered by Tesla’s custom AI built supercomputer, Dojo which by the way, feels suspiciously borrowed from Tony Stark’s garage. It thinks in parallel and trains self driving algorithms using real world sensors piped through Apache Kafka streaming system.

The simulation pipeline includes high-fidelity CAD (Computer Aided Design) models and physics-based dynamics engine which simulates both physical and software behavior. These insights feed into the car via OTA (over-the-air) updates turning it smarter each passing week.

It’s like your Tesla has its own personal JARVIS and you’re just lucky enough to be the driver.

From Twin Sparks to Global Waves

Digital twins are basically the ultimate “clone your homework” hack for the real world but way smarter and legal. What started as a lifesaver back in the days of Apollo has now expanded everything from hospitals to smart cities, making sure things run smoother than your favorite video game.

What’s really exciting is how digital twins will evolve next. Beyond just mirroring reality and spotting problems as they happen, in the future, they will predict complex scenarios like simulating entire city ecosystems to test how climate change might affect them. They will become more autonomous and will be able to make real-time decisions without human input, much like a digital co-pilot.

So here’s the deal: when reality gets a digital twin, your imagination becomes the new blueprint. The only limit? How far you're willing to think or how much caffeine you’ve consumed. Either way, it’s your move now. Think smarter, dream bigger, and maybe even have a little fun while you’re at it.

Quantum computers aren’t just lab experiments anymore. They’re real, evolving faster than a ‘Brainrot trend’ , and primed to detonate the encryption protecting almost everything you do online. Imagine hackers in 2035 cracking today’s encrypted data, which can include your bank details, medical records, and that cringe google search for “why do cats ignore me when I pspsps at them?”, as easily as popping a balloon.

This is not a Mission: Impossible movie plot, it’s the quantum time bomb lurking beneath our digital lives. So here is how it works, why your data is at risk, and how we’re defusing it - with lasers, math mazes, and a sprinkle of chaos.

Quantum Computing: The Atomic Clock on Espresso (and Why you should care)

Classical computers are like sundials, steady but slow. Quantum computers? They’re atomic clocks on espresso, with a side of Red Bull.

• Classical bits are light switches: 0 (off) or 1 (on).

• Qubits are like dimmer switches - thanks to superposition, they can be both 0 and 1, or anything in between. It’s like getting out of the labyrinth by exploring every path at once instead of step by step.

Entanglement: Einstein’s “Spooky” BFFs

• If two qubits are entangled, changing one instantly affects the other - even if they’re on opposite sides of the galaxy. Think of it as twin librarians who finish each other’s sentences… and shelves.

Why Quantum Isn’t Mainstream (Yet):
Qubits are like divas: they demand near‑absolute‑zero temperatures and complete isolation from vibrations, stray light, and even Wi‑Fi signals to prevent decoherence. Error correction, meanwhile, is the real nightmare—like herding a bunch of overly caffeinated cats.
Real-World Quantum Players:

• IBM’s Quantum Eagle: Handles 127 qubits but still makes mistakes.

• D-Wave’s Annealers: Solve optimization problems but can’t crack RSA… yet.

Why This Matters

Shor’s Algorithm - a quantum cheat code - could crack RSA encryption (the lock guarding 90% of the internet) in hours. For classical computers, it’s like running a marathon barefoot but for quantum computers it’s a hoverboard sprint.

Analogy: RSA is a timed safe whereas quantum computers are lockpicks with a stopwatch and a PhD in chaos theory.

Real-World Stakes:

• The NSA recommends preparing for PQC by 2030.

• China claims it’s built a quantum computer that cracks RSA-2048 in minutes. Yikes.

• The EU’s Quantum Flagship Program is investing €1 billion to deploy quantum-safe infrastructure by 2030, prioritizing defense and healthcare.

• JPMorgan Chase is stress-testing PQC to secure $10+ trillion in daily transactions, fearing quantum-driven financial chaos.

• Ransomware groups like LockBit are stockpiling encrypted data, betting on future quantum paydays.

RSA’s Rise and Fall: From Hero to Zero

A Brief History of RSA:
In 1977, three MIT nerds (Rivest, Shamir, Adleman) invented RSA, turning encryption into a math puzzle:

1. Pick two massive primes (like 1,000-digit monsters).

2. Multiply them.

3. Security relies on one fact: Factoring that product back into primes is brutally hard for classical computers.

Prime Factorization 101:
Factoring primes is like reverse engineering a cake. If you bake 17 x 23 = 391, it’s easy. But if I give you 391 and ask for the original primes … well it’s game over for you. Now imagine the numbers are 600 digits long.

Why RSA Ruled the World:
It’s like hiding a needle in a haystack which is almost the size of Jupiter. Even the world’s fastest supercomputer would take 300 trillion years to crack RSA-2048.

But Quantum computers couldn’t care less:
Shor’s Algorithm tests all possible factors at once using superposition. Imagine brute forcing a password by guessing every combination simultaneously.

Reality Check: RSA is like milk in the sun - already going bad. Quantum computing is the heatwave speeding it up.

Case in Point:
In 2022, Chinese researchers simulated breaking 2048-bit RSA. The fuse is now lit.

Harvest Now, Decrypt Later: The Heist of the Century

Hackers aren’t waiting for quantum tech - they’re hoarding encrypted data today. Your tax returns, corporate secrets, and that Spotify blend with your crush? All are sitting ducks in a digital storage locker.

Did You Know?

• 70% of organizations admit they can’t detect encrypted data theft (Ponemon Institute).

• 95% of web traffic is encrypted (Google Transparency Report).

How It Works:

1. Steal encrypted data (easy, since most traffic is encrypted).

2. Wait 5-10 years for quantum computers to mature.

3. Decrypt everything, from military intel to your middle-school blog-diary.

The Fallout? Digital Mayhem:

• 🔓 HTTPS/TLS: Secure websites become glass houses and even worse - your passwords added to rockyou2.txt .

• 💸 Blockchain: Crypto wallets? Emptied. NFTs? Repossessed by quantum-powered bots.

• 📜 Digital Signatures: Forged contracts, fake software updates, and literally counterfeit money.

Industry-Specific Chaos:

• Finance: Banks could lose billions overnight if transaction histories are altered.

• Healthcare: Your DNA data? Auctioned to the highest bidder.

• Government: Diplomatic cables leaked, sparking geopolitical crises.

Worst thing that can happen: A hacker group leaks 2030’s decrypted data, revealing your “visionary leadership” speech was ChatGPTed.

Post-Quantum Cryptography: Cybersecurity’s Avengers (Explained for Beginners)

Meet PQC—the superhero squad of encryption. These algorithms make quantum computers rage-quit.

🌀 Lattice-Based Cryptography: Math Mazes in 500+ Dimensions

Imagine navigating a maze, but instead of 2D walls, you’re dodging obstacles in nearly 500 dimensions. Lattice-based cryptography uses grids (lattices) in mind-bending dimensions to hide data. Quantum computers struggle here because solving these mazes requires guessing all paths at once - something even their multitasking qubits have a skill issue with. Why is it cool: Because it’s the backbone of algorithms like Kyber and powers privacy tools like secure messaging apps.

🔐 Hash-Based Signatures: Tamper-Proof Fingerprints
Hash-based signatures work like a wax seal for data. When you “sign” a document, it’s stamped with a unique hash - a fixed-length code (e.g., a random string like a3F9A2xZ). Tamper with the document? The hash changes completely and screams “FAKE!”
Drawback: They’re one-time use, like disposable glovesgr - eat for critical systems, clunky for Netflix binges.

📜 Code-Based Cryptography: Errors as Locks
This method borrows from error-correcting codes which means math is used to fix twisted texts. Imagine sending a message with intentional typos. Only someone with the “typo rulebook” (your private key) can decode it. Quantum computers hate this because finding errors in massive codes is like finding a single misspelled word in the Harry Potter novels written in Morse code.

📊 Multivariate Polynomials: Equations from Hell
These algorithms use systems of equations with hundreds of variables (e.g., x³y² + 4xy – 7z⁴ = 42). Solving them requires brute-forcing endless combinations - a nightmare even for quantum machines.
Real-world use: They’re niche but secure, like a vault guarded by a troop of kangaroos with boxing gloves.

⚡ Kyber & Dilithium: The Dynamic Duo, Explained

Kyber (Key Exchange):

• Uses lattice math to securely share encryption keys.

• Imagine whispering a password in a crowded room, but the password is hidden inside a 100D maze. Only your intended recipient has the map.

• Used in: Google’s PQ-TLS experiments, VPNs.

Dilithium (Signatures):

• Creates unforgeable signatures using lattices.

• Think of it as a wax seal that explodes if tampered. Even quantum bots can’t fake it.

• Used in: Software updates, legal e-signatures.

Why They’re Cool: They’re fast, efficient, and already being tested by tech giants.

🛡️ Hybrid Encryption: Double the Locks, Zero the Regrets

Hybrid encryption pairs RSA with PQC algorithms. Why?

1. Backward compatibility: Old systems still understand RSA.

2. Quantum-proofing: PQC adds a futuristic lock.
   How it works: Your data is wrapped in both RSA and PQC encryption. Hackers need to crack both - like breaking into a bank vault while dodging laser sharks.
   Real-World Use: AWS, Microsoft Azure, and Signal already use hybrid approaches

NIST’s PQC Timeline:

• 2016: Launched global competition for quantum-safe algorithms.

• 2022: Announced Kyber, Dilithium, and others as finalists.

• 2024: Final standards released

Who is NIST?

The National Institute of Standards and Technology (NIST) is the U.S. federal agency setting the gold standard for cybersecurity. Since 2016, they’ve spearheaded the global effort to vet and standardize quantum-resistant algorithms—because even hackers need rules to break.

Why This Squad Matters:

• Protects WhatsApp chats, online banking, and even your smart fridge.

• Guards critical infrastructure (power grids, self-driving cars) from quantum chaos.

• Future-proofs IoT: Medical implants, connected cars, and yes, even your smart fridge.

• The EU wants PQC in banks and hospitals by 2025 and even NASA’s testing it for space comms.

PQC is not just a shield, it’s more of a time machine which plans on securing today’s tech for tomorrow’s quantum world.

Your Countdown Checklist: How to Dodge the Quantum Apocalypse

1. Stay Calm (But Move Fast):

• Treat this like climate change: act now or drown later. Why? Transitioning to PQC takes years—start before the clock hits zero.

2. Nudge Your IT Team:

• “Hey, maybe peek at NIST’s PQC drafts?”

• Tools to Try: Open Quantum Safe (free PQC libraries).

3. Password Hygiene:

• Use a password manager (cough Bitwarden cough).

• “yourdogsname123” won’t save you and neither will “YourName@dob.”

4. Learn the Basics:

• YouTube “quantum for dummies.”

• Free Course: Coursera’s Cryptography I.

5. Advocate Loudly:

• CEOs should brag about PQC in earnings calls.

• Normies just tweet #QuantumProofMe on X.

Myth Busting: Quantum Nonsense vs. Reality

🔥 Myth: “Quantum computers exist already! My data’s gone!”
Reality: Today’s quantum machines are toddlers—cute but useless.

🔥 Myth: “PQC will slow the internet to dial-up.”
Reality: Modern PQC is faster than RSA.

🔥 Myth: “Only governments need to worry.”
Reality: If you use Wi-Fi or oxygen, you’re on the team.

Bonus Myth: “Quantum can break all encryption.”
Reality: Symmetric encryption (like AES-256) is quantum-resistant. PQC handles the rest.

What If We Do Nothing?

Imagine waking up in 2035 to:

• Bankrupt banks: Quantum hackers drain accounts globally.

• Fake news 2.0: Forged government documents spark wars.

• Identity apocalypse: Your medical history is just waiting to be meme fodder.

This is not me trying to be fearmonger, it’s just simple math.

FAQ: Your Quantum Questions, Answered

Q: When will quantum computers crack RSA?
A: Although projections place the timeline between 2030 and 2050, hackers have already begun hoarding data.

Q: How soon will PQC be everywhere?
A: Though NIST's standards were introduced in 2024, full adoption may take 5–10 years.

Q: Is my iPhone safe?
A: For the time being, Apple is looking into implementing PQC in upcoming iOS updates.

Q: Can I buy a quantum computer?
A: At $15 million per D-Wave annealer, your dog’s influencer career might need to be on hold for a bit.

Final Countdown: Encryption Isn’t Dead—It’s Evolving

The quantum time bomb isn’t doom — it’s a wake-up call. We survived Y2K, spam, and Flash; now, PQC is the next chapter.
TL;DR:

• Quantum will break RSA.

• PQC is the fuse we’re cutting.

• Your job? Stay alert, install upgrades, and for the love of the quantum gods stop reusing the same password for every social media account.

What’s Next?

For Tech Teams:
Start stress-testing Post-Quantum Cryptography (PQC) libraries like Open Quantum Safe—a free, open-source toolkit that lets you experiment with quantum-resistant algorithms today. Think of it as a “quantum-proof helmet” for your data. Dive into hybrid encryption prototypes, collaborate with frameworks like PQ-TLS, and join industry trials (Google and Cloudflare are already inviting beta testers). The goal? Ensure your systems aren’t caught with their encryption pants down when quantum arrives.

For Everyone Else:
Bookmark NIST’s PQC updates and follow tech giants like IBM and Microsoft, who blog about quantum readiness. Not a developer? No problem. Advocate for PQC adoption in your workplace (“Hey, shouldn’t our app be quantum-safe?”), and keep an eye on apps/software announcing PQC upgrades. Knowledge is power—and in this case, it’s also your underground bunker in case the quantum time bomb goes off.

Got questions? Drop them below. Conspiracy theories? We’ll bring popcorn. 🍿

Stay secure, stay snarky, and remember: Time’s ticking, but we’ve still got the codes. 🔒⏳

I open Netflix, ready to watch Interstellar for the hundredth time. I hit play.

Baam—the dreaded loading wheel appears.

Frustrating, right? Seconds feel like hours. But why does this even happen?

Every time you stream a movie, your device has to fetch a massive amount of data - loads of it. Think of it like ordering food at a restaurant. If the chef has everything prepped, your meal arrives in minutes. But if they’re starting from scratch, you’re in for a long wait.

That’s exactly how streaming works. If the data isn’t readily available, buffering kicks in, and suddenly, your movie night turns into a waiting game.

But what if your favorite movies started instantly— every single time? No buffering. No delays. Just play and enjoy. Sounds like magic?

Well, it’s not magic—it’s caching. And one of the most powerful tools for this? Redis.

In today’s world of streaming, where milliseconds can make or break an experience, even the slightest delay is a deal breaker. Users don’t just want their content fast—they want it now. No buffering. No waiting. Just hit play and go.

So how do platforms like Netflix and Hulu pull this off? What keeps them running smoothly even when millions of people are streaming at the same time? The answer lies in effective caching - and the secret behind it?

Redis. (There are others too, but for now, let’s roll with Redis.)

The Role of Caching in Streaming Services

Imagine this: our friend Jeremy wants some ice cream. He has two options—

1. Grab one from the nearby ice cream parlour.

2. Order it directly from the company’s storage.

If Jeremy picks the ice cream parlour, he gets his treat in just two minutes—quick and convenient! But if the parlour doesn’t have what he wants, he has no choice but to wait 10 minutes for the company’s storage to deliver it.

Naturally, the best way to get ice cream quickly is to check the parlour first. If it’s available, great! If not, he places an order, waits, and once it arrives, the parlour stocks that flavour for next time—ensuring others can grab it instantly later.

Now, swap out ice cream for video data, and you’ve got caching in streaming services.

If every request had to go all the way to a primary datastore, playback would be painfully slow. Instead, they use Redis, a high-speed, in-memory data store, to act as the “ice cream parlour” for frequently accessed data.

It helps in streaming services by:
1. Storing frequently accessed metadata (all your recommendations and angry ratings).
2. Caching video chunks and API responses (like storing frequently accessed calls, images, or data) helps reduce database load and improve performance.
3. Providing near-instant access to frequently used data.

This significantly reduces the need for repeated database queries, enhancing performance and efficiency. Making streaming feels instantaneous—(no more endless loading wheels).

But where did this idea come from? It all started with one frustrated developer…

Built out of pure frustration

Back in 2009, an Italian programmer named Salvatore Sanfilippo was working on a real-time web analytics system to track user activity on websites instantly but he ran into a wall.

The database he was using simply couldn't handle the load of tracking thousands of web pages in real-time! Every page view meant multiple database writes, and complex queries were bringing his servers to their knees.

The alternatives weren't great either.

Memcached could cache data but couldn't save it permanently.

MySQL was too slow for real-time operations he needed.

MongoDB was great for storing large amounts of data, but it was overkill for what he was trying to do.

Instead of giving up, Sanfilippo did what any passionate developer would—he built his own solution—something fast, lightweight, and capable of handling real-time data with ease. He wanted a system that could store and retrieve data quickly without the overhead of traditional databases.

With Redis, Sanfilippo didn’t just solve his problem—he changed the way the world handles data.

Redis’s Evolution

Originally developed by Salvatore Sanfilippo as a simple key-value store, Redis was designed to make data storage and retrieval more efficient. Over time, it evolved into a powerful in-memory data store, supporting advanced data structures like lists, sets, sorted sets, and hashes. Today, Redis is more than just a key-value store—it’s a high-performance, multi-purpose tool that powers everything from real-time analytics to AI-driven applications.

Unlike databases that store data on a hard drive, Redis keeps everything in the computer’s memory. Because of this, it can perform over 100,000 tasks every second, making it perfect for apps that need to respond immediately. Think of it as reaching into your pocket for a key, instead of running to the basement.

As Redis grew beyond a simple key-value store, its architecture evolved to handle even more demanding tasks.

The Architecture That Drives Speed

Let’s put ourselves in the shoes of Sanfilippo. Imagine you’re designing an alternative to the medley of problems you have. How would you go about it? You would probably consider the following factors:

Efficiency: You’d opt for efficiency first. This means processing tasks one by one rather than handling everything at once—you want a streamlined model, right? Redis achieves this with its Single Threaded Efficiency model. Instead of juggling multiple tasks like traditional databases, Redis processes requests one at a time, but at an incredibly fast pace.

Fast Data Retrieval: Next, you’d want to search for data quickly because you wouldn’t want your resources wasted just looking for the data you need. To do this, Redis uses Optimized Data Storage with smart, memory-efficient structures. These structures store data compactly, reducing memory usage and speeding up lookups.

Stability Through Separation: Finally, you’d want to keep different operations separate. For instance, writing new data shouldn’t slow down reading existing data. Redis handles this with Copy-on-Write for Stability. When saving data, it uses a method that ensures new writes don’t interfere with ongoing operations, keeping the system smooth and efficient.

How Redis Powers Real-Time Applications

Well, now that you've seen just how engrossed Salvatore was in cracking these challenges, here is a little list I have made to present Redis’s use case:

Caching:

In everyday terms, when you open an app—especially a chat app—you expect everything to load instantly. Redis helps achieve this by caching data that is accessed over and over, like user profiles. Instead of reaching out to a slower database every time you need to see someone’s profile, Redis keeps a copy in fast-access memory. This means that as soon as you open the app or click on a conversation, the profile information is already there, making the experience smooth and responsive.

Session Management:

What do we mean by it? Session management is like keeping a note of what a user is doing while they're using an app. It remembers details such as when you're logged in or what's in your shopping cart so you don't have to log in or fill your cart again on every page.

Instead of saving these notes in a slower database, Redis keeps them in memory. This means when you use a social media app, for example, your login stays active and your information is quickly remembered—even if you close your browser. Redis makes the whole experience faster and more secure by handling these "notes" in real-time.

Geolocation:

Heck, Redis can even be used to store and track real-time location data, making it possible to build applications that require location-based features, such as ride-hailing services or social media check-ins. For example, a ride-hailing service could use Redis to track the location of drivers and riders in real-time.

These diverse applications highlight Redis’s versatility, but what really makes it indispensable are its performance advantages:

Why is it the Go-To Choice for Real-Time Systems

A little list here too about the same

1. Ultra-Low Latency – Responds almost instantly—usually in less than a millisecond—so users don’t experience any delay.

2. High Throughput – Can handle millions of requests per second, keeping things running smoothly even during heavy traffic.

3. Pub/Sub Messaging – Lets different parts of your system talk to each other in real time, much like a live chat or news feed.

4. Scalability – Easily grows by adding more servers, ensuring that performance stays high as your system expands.

5. Time-to-Live (TTL) – Automatically removes old or unused data to keep the memory clean and efficient.

6. Multi-Model Storage – Supports various data types—from simple key-value pairs to more complex structures like lists and maps—making it versatile for different needs.

To see these benefits in action, let’s explore how industry giants like Netflix and Hulu harness Redis for scalable streaming

How Netflix and Hulu Use Redis to Power Scalable Streaming

In large-scale streaming services, delivering content swiftly and reliably is paramount (You don’t want trp ratings dropping because you delivered the content late). Both Netflix and Hulu have harnessed the power of Redis to meet these demands, a lil sneak peek

Netflix: Scaling with Dynomite and Redis

Netflix developed Dynomite, a distributed datastore that builds on Redis's features to support data availability across multiple regions.

This integration offers several advantages:​

1. Elastic Scalability: Netflix spreads its work across many servers. This means no single server is overwhelmed, and if one server has a problem, the others can pick up the slack. How does it do it? By deploying Redis clusters across multiple nodes, it effectively distributes workloads, minimizing single points of failure

2. Caching API Responses: Utilizing Redis to cache frequently accessed metadata reduces latency and alleviates the load on primary databases, this helps the system deliver content quickly without constantly going back to the main, slower database.

3. High Availability: Dynomite lets Netflix operate in multiple regions (or parts of the world) at the same time. This means that even if one region experiences issues, users in other regions can still enjoy uninterrupted streaming.

For an in-depth exploration of Dynomite's performance benchmarks on AWS, refer to Netflix's technical blog post. ​[3]

Hulu: Managing Billions of Video Requests with Redis [2]

Facing the challenge of serving over 4 billion videos, Hulu integrated Redis to bolster its infrastructure:​

1. Session Storage: Redis keeps track of user sessions across many servers. This means if one server fails, another can quickly take over, ensuring a smooth and continuous experience for the user.

2. Content Delivery Optimization: Hulu caches video details and thumbnails in Redis. This allows videos and images to load faster and reduces the load on the main servers, making the service more responsive.

3. Rate Limiting and Traffic Management: Redis efficiently manages a high number of requests at once. This helps prevent system overload during busy times, ensuring the service remains stable even under heavy traffic.

Challenges and Problems Faced

Despite Redis’s advantages, even the best have their weak spots—a chink in the armor, if you will. Large-scale implementations come with their own set of challenges:

• Memory Constraints: Being an in-memory store, Redis requires careful memory management to prevent excessive costs.

• Data Persistence Issues: Ensuring data consistency in case of crashes requires additional configurations.

• Replication Overhead: Scaling Redis clusters demands efficient replication strategies to balance performance and reliability.

• Sharding Complexity: Splitting data into manageable pieces (or shards) requires careful planning to distribute workloads effectively.

Conclusion

Redis has become an integral part of large-scale streaming services, eliminating lag and keeping content delivery smooth. But the road isn’t without challenges. Scaling efficiently, managing traffic spikes, and ensuring high availability across multiple regions require constant innovation. Fortunately, Redis continues to evolve, adapting to the ever-growing demands of real-time content delivery. As streaming services push the boundaries of speed and quality, Redis remains the silent hero, ensuring that every movie night is seamless.

Take a simple movie night, I had recently, for instance.

I settled into my couch, ready to watch Interstellar for the hundredth time. I hit play.

Baam—no buffering. No waiting. Just pure speed.

References:

1. https://architecturenotes.co/p/redis

2. https://venturenox.com/blog/the-power-of-redis-in-transforming-real-time-applications/

3. https://blogs.vmware.com/tanzu/case-study-how-hulu-scaled-serving-4-billion-videos-using-redis/

4. https://netflixtechblog.com/dynomite-with-redis-on-aws-benchmarks-5c942fc7ca38

If you’re passionate about competitive programming, you’ve probably heard of ICPC, the Olympics of Programming—the most prestigious algorithmic programming contest in the world. Every year, thousands of the brightest minds battle it out for a coveted spot in the regionals and beyond.

I’ve had the privilege of competing in ICPC regionals twice, securing AIR 69 in Chennai and AIR 51 in Amritapuri. These experiences have shaped me into a sharper problem-solver, teaching me how to break down complex problems and optimize solutions under extreme time constraints.

Beyond ICPC, I’m also a Specialist on Codeforces, where I’ve been actively competing for the past year, continuously refining my skills against some of the best programmers in the world. Competitive programming isn’t just a hobby—it’s an obsession. I’ve spent countless hours grinding problems on Codeforces, LeetCode, and AtCoder, tackling a diverse range of problem-solving paradigms, from graph theory and dynamic programming to bit manipulation and number theory.

But here’s the thing—no matter how many problems you solve, there’s always a new way to challenge yourself. Reverse Coding is one such challenge that breaks the traditional approach to problem-solving.

Most contests give you a problem statement, and your job is to write a program that satisfies the given constraints. But in Reverse Coding, the game is flipped entirely—you’re only given input-output pairs, and it’s up to you to decode the hidden logic behind them.

This concept is incredibly powerful because it doesn’t just test your coding ability—it tests how well you can reverse-engineer solutions, think critically, and deduce patterns from seemingly random outputs. These skills are crucial in real-world problem-solving, whether in competitive programming, software development, or AI research.

That’s why I’m writing this blog. If you’re a competitive programmer, a logic enthusiast, or someone who simply loves puzzles, I highly recommend participating in Reverse Coding at Yantra Week, organized by VIT’s ACM Chapter.

Let’s dive into the who, what, where, and how of Reverse Coding and why this event is a must-attend for anyone serious about problem-solving.

What is Reverse Coding?

Imagine this—you’re in a competitive programming contest, but instead of receiving a well-defined problem statement, you’re given only input-output pairs. The logic behind them? A complete mystery.

Your task: figure out the pattern and write a program that replicates it.

This is Reverse Coding, a twist on traditional problem-solving where you must reverse-engineer the underlying logic using only the given examples. Think of it as debugging in reverse—instead of fixing a broken program, you’re uncovering the hidden logic that connects inputs to outputs.

Here’s a simple example:

🔹 Input: 1 → Output: 1
🔹 Input: 2 → Output: 4
🔹 Input: 3 → Output: 9

Clearly, the logic is squaring the input (n²). So, your job is to write a function that squares a number.

But don’t let this simple example fool you. The actual competition will feature far more complex patterns, edge cases, and deceptive sequences that will test your ability to think outside the box.

Why Reverse Coding is a Game-Changer

Competitive programming is all about recognizing patterns—whether in dynamic programming states, number sequences, tree structures, or graph connectivity. The best programmers aren’t just those who can implement standard algorithms but those who can identify hidden patterns quickly and apply the right logic to solve problems efficiently.

But Reverse Coding takes this to another level by removing the problem statement entirely. Unlike traditional contests, where you start with a well-defined question and work toward a solution, Reverse Coding gives you only the output—leaving you to reconstruct the missing logic from scratch.

This fundamentally changes the way you approach problem-solving, forcing you to think in ways that most programming contests never train you for.

1. Think Like a Problem Setter

Competitive programming typically involves solving problems, but have you ever thought about how problems are created?

In Reverse Coding, you are essentially doing the job of a problem setter in reverse. Instead of solving a problem with a given approach, you must figure out what the problem even is.

• What mathematical or logical transformation is happening between input and output?

• Is the output following a known formula, sequence, or transformation?

• Are there multiple layers of logic, such as nested conditions, recursion, or modulo operations?

2. Sharpen Your Analytical Skills

Programming isn’t just about writing code—it’s about understanding data. Reverse Coding forces you to spot hidden patterns in numbers, strings, and sequences, making it an excellent exercise in data analysis and logical reasoning.

3. Enhance Debugging Abilities

Reverse Coding simulates this debugging process but in a more structured way. It teaches you to:

🔹 Break problems into smaller test cases to isolate patterns.
🔹 Analyze unexpected behavior systematically instead of randomly tweaking code.
🔹 Develop logical intuition for how transformations affect output, making bug-fixing much faster.

If you’ve ever struggled with debugging, Reverse Coding is the perfect training ground to sharpen your debugging mindset.

4. Develop Intuition for Edge Cases

One of the hardest skills in competitive programming is anticipating edge cases before they appear. Many people lose contests not because they can’t solve a problem, but because their solution fails on hidden edge cases.

Since Reverse Coding involves actively testing the system with different inputs, it naturally trains you to:

✔ Think about extreme values (What happens at n = 1 vs n = 10⁶?)
✔ Try negative and zero inputs (Does the pattern change for -5 or 0?)
✔ Identify hidden dependencies (Does the output depend only on n or also on some hidden variable?)

This ability is critical in contests where problems often have tricky constraints that aren’t explicitly stated. By developing this intuition, you become a stronger problem-solver overall.

How the Competition Works

At Reverse Coding, you’ll be working in a web-runner interface (similar to LeetCode’s environment). Here’s how the process unfolds:

1️ You input test cases based on the given constraints.
2️ The system generates an output for each test case.
3️ You analyze the input-output pairs to decode the hidden logic.
4️ Once you crack the pattern, you write a program that replicates it for all valid inputs.
5️ Your code is then evaluated for correctness and efficiency.

Unlike traditional contests where you start by reading a problem, here you start by experimenting with inputs, making every participant an investigator, a pattern-spotter, and a reverse-engineer.

What Kind of Problems Can You Expect?

While I can’t reveal the exact problems (where’s the fun in that?), I can certainly give you an idea of the types of challenges you’ll face in Reverse Coding. Unlike traditional contests where you have a well-defined problem statement, here you’ll need to uncover the logic behind the given input-output pairs. Some problems will be obvious at first glance, while others will require deep pattern analysis, algorithmic intuition, and creative thinking to decipher. The best strategy is to experiment with diverse test cases, analyze trends, and adapt dynamically.

One of the most common categories you might encounter involves mathematical sequences. Problems in this category often deal with Fibonacci numbers, factorials, prime sequences, bitwise transformations, and modular arithmetic. For example, you might be given a sequence of outputs that correspond to prime numbers, factorials, or values obtained using bitwise XOR operations. Recognizing these mathematical properties quickly is key to solving such problems. Another possible challenge could involve modular arithmetic, where outputs are generated based on numbers wrapped around a certain modulo constraint, a concept frequently used in cryptography and number theory problems.

Another exciting category of problems involves graph-based outputs, where the given input-output pairs may represent connectivity, adjacency lists, or shortest paths between nodes. You might need to recognize a hidden graph traversal pattern, such as outputs representing Breadth-First Search (BFS) levels or Depth-First Search (DFS) orderings.

Finally, some problems will involve data-driven outputs, where statistical computations or probabilistic transformations determine the output.

How to Approach Reverse Coding Like a Pro

As someone who has competed in ICPC regionals and Codeforces contests, I can tell you that success in Reverse Coding isn’t about brute force—it’s about smart thinking and strategic problem-solving. Unlike traditional programming contests where you can directly apply known algorithms, here you need to uncover the algorithm itself. The best way to do this is by experimenting with diverse inputs. Start small with numbers like 1, 2, 3 and check for common mathematical patterns such as squares, factorials, primes, or arithmetic sequences. Then, push the limits by testing boundary values—for instance, 0, 1, 10⁹—and see how the system reacts to negative or extreme cases. The more inputs you try, the more clues you gather about the hidden transformation.

Once you have some initial observations, look for mathematical relationships between the inputs and outputs. Is the output always a multiple of the input (n × k)? Does it involve bitwise operations like n XOR k? Could it be following a known mathematical sequence, such as Fibonacci, Catalan, or Tribonacci numbers? Recognizing these patterns early can give you a significant edge over competitors who are still guessing. If the problem isn’t purely mathematical, try to break down the output structure—for example, if the output is a string, check whether it’s being reversed, cyclically shifted, or encoded in ASCII values. If the output is an array, analyze how elements are being rearranged, sorted, or grouped.

A key strategy in Reverse Coding is to think in terms of state transitions. Ask yourself: What happens when you increase the input? Is the pattern strictly dependent on the current input, or do previous inputs influence the result? Many problems may involve hidden state machines where the output is affected by a previous sequence of inputs. Recognizing dependencies between inputs can help you reconstruct recursive functions or automata-based transitions. Finally, while some problems may seem cryptic at first, it’s important to not overcomplicate your approach. The logic is hidden, not impossible—once you uncover the underlying pattern, implementing the solution will often be straightforward. Staying calm, testing methodically, and thinking outside the box are the keys to dominating Reverse Coding challenges.

Final Thoughts

Reverse Coding is more than just a competition—it’s a completely different way to think about programming. Instead of solving problems, you’re creating them from scratch, decoding logic, and thinking like a problem setter.

As an ICPC regionalist and Codeforces Specialist, I’ve seen how crucial pattern recognition, logical deduction, and reverse engineering are in high-level contests. Reverse Coding is the perfect training ground for anyone who wants to level up their thinking and problem-solving abilities.

So, if you love puzzles, programming, and breaking the conventional way of thinking, this is the one competition you don’t want to miss.

That’s pretty mind blowing, isn’t it?

But what if I told you that in real life, we’re getting closer to having that machine plugged inside us?

Before you ask, “Drashti, what are you on?”—let me stop you right there. I’m serious!

I know, I know it might sound like something straight out of a cyberpunk novel, but humans merging with technology isn’t just some wild sci-fi fantasy. It’s actually been in the works for decades. Flashback to the '90s, when the first human microchip implants were used primarily for medical purposes, like tracking health data or helping people with disabilities.

But beyond that, the real vision was always bigger.

…what if we could actually use technology to enhance our minds?

Stumbled across Neuralink yet?

Founded in 2016, turning this once wild, almost outlandish dream into reality, Elon Musk’s Neuralink was created with the ambitious vision of merging the human brain with advanced technology to help solve neurological disorders and eventually enhance human capabilities. And it has been making some serious waves.

A recent survey states that Neuralink has a total of 61 patents globally, with 18 granted so far. Over 80% of these patents are still active, providing the company with ongoing protection for its innovations. (Source)

Now that we've had a glimpse of this groundbreaking innovation, it's time to dive into the who, what, where, and how of it all.

The Mind-Machine Integration

Think of it this way: Over 86 billion neurons make up your brain's complex and intricate network and above all, your brain never catches a break. For seamless operations, these neurons process information continuously and send and receive signals to and from many bodily sections.

However, it's a monumental effort to maintain such a multifaceted and nuanced data system around-the-clock without interruption. Imagine you are driving a car in an unfamiliar location with no map or GPS and numerous paths. One can easily veer off course or lose their way entirely. In a similar way, confusion can occasionally overwhelm your brain. If only there was a smart GPS in your brain– a system that could efficiently and precisely direct those brain signals.

This is where the concept of microchipping comes into play.

By creating a direct interface between your brain and external devices, the Neuralink microchip helps re-establish lost connections or enhance existing ones. For instance, in the case of someone with a spinal injury, the chip could act as a conduit, helping signals from the brain bypass the damaged area and restore mobility.

Decoding the ‘How’

The N1 chipset, a coin-sized device with a diameter of only 8 mm that is implanted straight into the skull, is the brains behind Neuralink's breakthrough. It blends perfectly with the neurons in the brain by using incredibly fine wires—thinner than a human hair strand. The operation is done by a robotic surgeon who steers clear of the arteries and veins in that particular area of the brain. Multiple chips can be inserted for complex circumstances, offering even more coverage and functionality.

Now I'm not saying that you could just plug in the microchip and learn kung fu in a few seconds like in the mind-blowing movie but think of it as the Matrix-lite version–no cables hanging out of your head, just a sleek, implantable device.

Following successful implantation, Neuralink records brain impulses, transforms them into digital data, and sends that data to external devices such as computers and prosthetic limbs,

Neuralink eliminates the need for large, obtrusive equipment by using wireless technology to transmit data between the brain and computers. Each of the 1,024 electrodes on the chip can record or stimulate impulses. These electrodes are set up in 64-thread layouts, with 200 microns separating each electrode and 16 electrodes per thread. The robotic surgeon makes an incision in the skull that is just a little bit bigger than the chip itself, then meticulously sews the electrodes into the brain. (Umm honestly, now that I think I’ll take back the idea of implanting a chip and listening to spotify)

The Neural Timeline

But how exactly did this Einsteinian moment come about? How did this bizarre idea take shape?

Let's rewind back to the pre-covid era of 2017 when Neuralink submitted its first patent application for "Neural Lace," a state-of-the-art technology. Neural Lace consisted of incredibly small electrodes that possessed the ability to monitor brain activity. It might sound a little surreal, but the chef-d'œuvre extraordinaire advancements that have followed were made possible by this foundational research.

Two years later in 2019, Musk and his team unveiled the groundbreaking Brain-Computer Interface technology, which involved inserting flexible, ultra-thin electrodes into the human brain. Theoretically, these electrodes could allow individuals to control external devices such as computers or prosthetic limbs—using nothing but their thoughts.

As a proof of concept, a demonstration was organized, streamed live from Neuralink’s headquarters, where the team introduced Gertrude, a pig implanted with Neuralink. This tracking of her neural signals during movement showcased the device’s impressive functionality and immense potential.

“The public's reaction was mixed. Some were captivated by the potential of this technology, envisioning future applications in medicine and human augmentation. However, others expressed skepticism and concern, questioning the ethical implications and the feasibility of such advancements. Critics also noted that while the demonstration was impressive, it primarily showcased existing neuroscience capabilities rather than groundbreaking innovations. -BBC”

Fast forward to 2021, the burning question was finally going through its trials;

Could Neuralink chips be implanted into humans?

Among the first experiments, the objective was clear: to utilize the brain-computer interface to restore mobility in patients with severe spinal cord injuries and neurological disorders. As part of Neuralink's PRIME (Precise Robotically Implanted Brain-Computer Interface) project, a wireless brain implant was tested on quadriplegic individuals, offering a glimmer of hope and giving a glimpse of the endless possibilities that could be unlocked.

Monkey MindPong

Neuralink gained international acclaim in April 2021 for an intriguing experiment involving the macaque monkey, Pager. The N1 chip was positioned in the parts of Pager's brain in charge of hand and arm movements. The electrical signals his brain sent to control those motions were picked up by these small electrodes. Here’s where it gets interesting! Pager was initially taught to use a joystick to play a basic video game. The Neuralink device continued to record his brain activity while he played, learning to decipher the signals associated with his hand movements.

Once the team had enough data, they took away the joystick. But Pager kept playing the game, this time controlling the action on the screen purely with his mind. The chip translated patterns of his brain activity into real-time commands for the game. By the end, Pager was successfully playing a pong-like video game, proving that Neuralink’s technology could interpret neural signals and control external devices seamlessly.

Allow me to amuse you even more—checkout this YouTube video posted by Neuralink.

Monkey MindPong

This remarkable demonstration displayed the potential of Neuralink to help people with paralysis or other motor impairments regain control over devices and, perhaps one day, parts of their own bodies. Technology sure has a way of blurring the lines between reality and the kind of futuristic worlds we usually only see in sci-fi movies.

The Now and The Next

Presently, Neuralink is diving into the medical world with groundbreaking applications—like treating neurological disorders such as Parkinson’s, epilepsy, and Alzheimer’s. It’s also exploring ways to restore movement for those affected by paralysis and working on enhancing both hearing and vision.

So, what is Neuralink's final aim?

It’s to make it possible for humans to connect directly with AI, effortlessly exchanging thoughts and commands in real-time through a brain-computer interface. This vision stems from Mr. Musk’s belief in the urgent need for humans to stay ahead of the curve, especially given the rapid, exponential evolution of artificial intelligence. Its goal is to close the gap between the human mind and machines, ensuring that people don’t just keep up with AI but play an active role in shaping the future it creates.

Imagine a world where telepathy is a reality, thoughts are transferred directly between minds, and communication transcends linguistic barriers.

Although, let’s not give in to the temptations of dystopia just yet.

Behind the Breakthrough

Have I been praising Neuralink a little too much? Well then, time to flip the coin.

Neuralink has a great deal of technical and medical hazards that should be carefully assessed. This technology requires an unpleasant operation that involves making an incision in the skull. While the process is designed to be precise with robotic assistance, complications such as infection, inflammation, or thread retraction remain possibilities.

The electrodes may eventually disrupt the brain's normal functioning. Moreover, problems like device malfunction, battery failure, or disturbances in communication signals might reduce the implant's performance and prompt additional surgeries for repair.

Privacy and security are also pressing concerns. The way Neuralink devices capture and transmit cerebral activity naturally begs the question-- who controls this extremely private information and how secure it is.

On a broader scale, the long-term effects of such a device remain unknown.

Could prolonged use of N1 chips lead to neurodegeneration or cognitive decline?

What about the psychological impact of using technology to even think, move, and communicate?

What would happen if someone who has become accustomed to using Neuralink for routine tasks faced device malfunction?

As Neuralink moves forward, these questions demand further research and open discourse. These risks serve as a reminder that striking a balance between the potential to transform medicine and advance human potential while ensuring safety and ethical conduct is no easy task.

Epilogue

This fine line between risk and reward raises profound questions about the kind of future we want to create. Mr. Musk, the visionary behind this creation, has often shared his thoughts on achieving AI alignment—a concept where humans can merge with AI in a controlled, harmonious way, ensuring that this integration remains beneficial and safe for society. In the long run, he sees Neuralink as a way to foster a symbiotic relationship where both humans and machines can enhance each other’s abilities.

Microchipping pushes the boundaries of what it means to be human, merging technology with the brain in ways that were once pure science fiction. It’s a step into a future where the line between man and machine begins to blur—a world where our thoughts could control devices, and technology in turn, could enhance our abilities. But with such innovation come big questions.

Will this lead to incredible advancements that improve our lives and understanding of each other? Or could it take us further from what makes us human in the first place?

The boundary between man and machine continues to fade, and maybe, in the end, the Matrix doesn’t close with a plug-in, but with the seamless integration of the mind and machine.

You get the impression that you've entered a tiny, private coffee shop where every little detail has been carefully considered thanks to the comfortable seats, pleasant lighting, and background sound of cups clinking. Every component has been thoughtfully crafted to ensure your comfort and a seamless experience that extends beyond just a cup of coffee.

"Feels familiar, doesn't it?"

Imagine a world in which each tap, click, and scroll is meant to be as purposeful as possible in order to guide, instruct, and entertain. This is the area of micro-interactions, the small elements that turn a simple interface into one that is genuinely engaging.

The Magic Unveiled

Even though micro-interactions may appear minor or incidental, they give digital experiences flavor, much like spices do in cooking. These are what give an interface life—subtle vibrations, animations, or visual clues. Micro-interactions, which provide immediate feedback, walk users through activities, and add charming details that leave an interface memorable, are essential to the user experience even though they are frequently disregarded. With these careful touches, a decent UI can become a genuinely delightful and fulfilling experience.

Dan Saffer, a leading voice in the world of UX, encapsulates this approach with insightful simplicity:

“Micro-interactions are an exercise in restraint, in doing as much as possible with as little as possible. Embrace the constraints and focus your attention on doing one thing well. Mies van der Rohe’s mantra of ‘less is more’ should be the micro interaction designer’s mantra as well.”

Micro-interactions are the little but powerful components that allow users to interact with an interface in a natural, human way in the digital age. They are much more than just bells and whistles. These little details give the digital world a somewhat livelier feel, whether it's the joyful bouncing of an app icon when it's opened or a faint glow that indicates a button is ready to be touched. They provide us with constructive criticism, lead us through tasks, and even inject some humor into the process

The Four Elements of Micro-Interactions

Micro-interactions are made up of four main components: Triggers, Rules, Feedback, and Loops & Modes. Each part plays a role in creating a seamless, enjoyable user experience. Let's explore each one with visual analogies to bring them to life.

1. Trigger

The trigger is the starting point of a micro-interaction, activated either by the user or the system. User-triggered actions may involve clicking, swiping, tapping, or scrolling, while system-triggered actions happen automatically under certain conditions.

Picture a chef preparing a meal. The moment they chop the first ingredient is a trigger, setting off a series of actions: sautéing, seasoning, and plating. Similarly, in digital design, a user’s initial click, refreshing a page or swipe initiates a series of responses that guide them through their task.

2. Rule

The rule defines the next steps once a micro-interaction is triggered; it’s the interaction’s "instruction manual." The rule responds to the user or system’s trigger,

Imagine tapping the theme icon on your device. The rule here is simple: tapping the icon should switch between light and dark modes. This principle applies to both user- and system-triggered interactions, ensuring that each response aligns with the user’s journey and expectations.

3. Feedback

Feedback lets users know what’s happening during a micro-interaction. It’s the visual or audio response to an action, providing users with real-time updates.

During a payment process, a red border might appear around a card number field if it’s incorrect, while a green border signals correctness. Users are energised and gain confidence as they navigate the experience thanks to this instant feedback, which is similar to an audience applauding an actor's performance. These answers improve user satisfaction and add interest to the interface.

4. Loops and Modes

The behaviors and length of a micro-interaction are determined by loops and modes. The current configuration that remains in effect until it is altered is called a mode. For instance, unless the user selects a different project, the default project in a time-tracking app stays selected. Conversely, loops control the duration of an interaction.

The timer loop in a time-tracking app keeps counting until the user stops it. Users are kept interested by the dynamic flow and constantly shifting performance created by this continuous motion.

Why Micro-Interactions Steal the Show

Technology's capacity to empower and engage individuals is one of its biggest benefits. Beyond simple usefulness, a truly great experience should be captivating and unforgettable. In order to do this, micro-interactions are essential since they improve a product or service's appearance, feel, and general usability.

Micro-interactions improve the user experience in the following significant ways:

• Promoting Engagement: Users are encouraged to interact with the UI through interactive touchpoints.

• Status Indication: Users' expectations are managed with subtle animations that notify them of loading times.

• Error Prevention: Quick feedback and direction cut down on errors and frustration, which lowers churn rates.

• Brand Identity: Distinct animations add character to a brand and leave a lasting impact.

• Human Touch: Fun and relatability are enhanced by whimsical nuances.

• Real-Time Feedback: Users feel reassured that their actions are acknowledged by prompt responses.

• Improved user interface: Delicate animations produce a natural, enjoyable experience.

• Faster Adoption: New users might adjust more rapidly when they have friendly interactions.

• Task Simplification: Micro-interactions simplify difficult tasks by dividing them into smaller, more manageable steps.

Iconic Micro-Interactions on the UX Stage

Micro-interactions have been effectively included into a number of well-known platforms to improve user experience, increase engagement, and communicate important information. Here are a few noteworthy instances:

Instagram’s Pull-to-Refresh

By letting users pull down a list, Loren Brichter's Pull-to-Refresh feature, which was first used in the Tweetie app (2010), makes it easier to refresh material. Since then, mobile platforms have widely embraced this user-friendly design.

Instagram's approach is commended for its fluid and captivating experience, which includes smooth transitions that produce fluid motion from pull to refresh, progressive feedback as the animation changes during the pull, and subtle physics for realistic bounce effects.

1. Basic Animation Flow

• Pull Gesture Detection: The pull-to-refresh animation is triggered by a downward swipe.

• Progressive Animation: As the user pulls, icons—like a spinning arrow—animate proportionately.

• Release and Load State: The animation changes to a loading spinner and the content refreshes when the threshold is reached.

• Finalization: The animation seamlessly resets after loading.

2. Smooth Transition Elements

• Easing Routines: Personalized easing functions offer a spring effect upon release and produce responsive, flowing animations.

• Physics-Based Animations: The pull to refresh experience is improved with realistic effects like bounce-back upon release.

3*. Implementation Techniques*

• UIRefreshControl is used for pull-to-refresh in iOS (UI Kit/Swift UI). For distinctive animations, subclass it and add new views.

• Android: Customize SwipeRefreshLayout in Recycler View with animations.

• Web: To animate SVGs and icons, use JavaScript touch events, CSS animations, or frameworks like GSAP.

The responsiveness and tasteful micro-interactions of Instagram's pull-to-refresh animation make it stand out. It provides a user experience that is both intuitive and visually captivating by combining subtle physics, progressive feedback, and smooth transitions.

Duolingo’s reward animations

A key component of Duolingo's entertaining and captivating app are its reward animations, which employ interactive graphics, audio, and behavioral psychology to strengthen users' feelings of achievement. Here's a thorough explanation of how they operate:

1. Instantaneous gratification and progressive rewards

• Gems, Badges, and XP: Duolingo provides users with immediate satisfaction by rewarding them with animated gems, badges, or XP for finishing lessons or streaks.

• Instant Feedback: Users are inspired to continue learning when they receive incentives right away, which gives them a sense of accomplishment.

2*. Visual and Audio Feedback*

• Confetti Effects & Animations: Pop-ups and colorful bursts generate excitement and visually celebrate victory.

• Music & Sound Effects: Chimes, fanfare, and lively music add to the festive atmosphere.

• Responsive animations: To maintain the interface's vibrancy, components such as jewels and streak counters respond with nuanced movements or lighting effects.

3. Gamified Elements

• Leaderboard Animations: These dynamic animations encourage users to keep up their activity and create a sense of rivalry by showing them moving up or down in the rankings.

• Achievements & Badges: Duolingo gives out badges for reaching milestones like streaks or finishing lessons. When a badge is acquired, it flashes or glows, which motivates users to get more.

4. Psychological Impact of Reward Animations

• Positive Reinforcement: To encourage learning and user retention, Duolingo employs immediate, visible rewards.

• Sense of Progress: Users are kept interested by animations that clearly convey their progress through achievements, levels, and streaks.

• Dopamine-Driven Engagement: Through anticipation and reward, reward animations cause dopamine to be released, which in turn creates a potent engagement cycle.

The reward animations on Duolingo combine gamification, instant gratification, and multisensory input to make learning a language fun and interesting. Duolingo's success can be attributed to the way it uses sound, animations, and character-driven interactions to turn a potentially boring activity into an enjoyable and rewarding experience.

Impact of Poorly Designed Micro-Interactions

The "undo send" feature in Gmail's original version is a well-known example of a failure caused by missing or poorly designed micro-interactions. When users pushed "send," there was initially no apparent indication that the email was being sent, and there was no real-time indication of how long they had to "undo" the action.

Consequences of Poor Micro-Interactions in This Case:

1. Uncertainty and Panic: When someone realizes they made a mistake in their email too soon, they may become anxious because they don't know how long they have to reverse the send. They were unable to determine how long it took for the email to be permanently delivered because there was no visible countdown or progress animation.

2. Missed Correctional Opportunity: Users may become agitated and regret their acts if they are unable to undo them in the brief amount of time provided by a clear feedback loop or animation showing the "undo" button.

3. Decreased Trust: Users were less confident in the feature's functionality in the lack of clear feedback, such as a countdown or a notification. The purpose of the feature is to serve as a safety net.

This example demonstrates how crucial micro-interactions are for users to understand and safely interact with features, especially when those features call for quick actions. Real-time feedback and visual cues are two examples of this. Without these small but important interactions, users may feel lost, unsatisfied, or uncertain, which can lead to negative experiences.

Encore: A Designer’s Epilogue

Micro-interactions are the small but important details that make the difference between usefulness and enjoyment. As I researched this subject, I discovered how frequently little things like animations or feedback cues might have a significant impact on how people view a website or app. Apps that provide instant, clear feedback or smoothly walk you through a job using animations, for example, don't just work—they feel correct. These encounters build relationships and leave a lasting impression.

When I discovered that sometimes the most memorable app experiences are the ones that appear invisible, like an animation that rewards you when you complete a task or a subtle visual cue that leads you through the interface, the idea of designing for these tiny but significant moments first caught my attention. A user-centered approach is promoted by this design philosophy, which holds that paying attention to small things has a big influence.

As you refine your own designs, always remember to ask yourself:

• Is the action intuitive and does it make the user feel confident about what to do next?

• Does the user know right away whether their action was successful, or should there be a clearer visual or auditory cue?

• Whether playful, elegant, or professional, do the animations and transitions match the overall tone of the brand?

• Does it feel fast and fluid, or does it cause delay and frustration?

• Does it enrich the user experience, or is it distracting or too complex for its purpose?

By consistently incorporating deliberate elements into each click, hover, and scroll, designers may leave a little magic in their wake, transforming the commonplace into the remarkable and adding a little more fun and significance to each user experience. These minor adjustments have the power to transform a simple interface into something genuinely unique, making consumers want to use it repeatedly.

Through deliberate attention to these micro-details, designers transform functional apps into delightful experiences, ultimately bridging the gap between technology and human emotion.

Hmm…feels familiar?

You run your hand along the wall, and with a single thought, you shift the entire scene. The walls dissolve into sand, revealing an endless desert under a blazing sky. Your mind races with the thrill of possibility—what else can I create? What if I just decided to—

But wait. This isn’t Inception.

There’s no dream architect guiding you through, no layers of subconscious built by a team of extractors. Instead, it’s you, in a controlled lucid dream, guided by AI. A headband rests on your temple, tracking your brainwaves, subtly shaping your dreams as you take charge. You’re not just visiting a world of imagination—you’re building it, with the help of technology that brings this surreal experience under your control.

The Dawn of Controlled Dreaming

Lucid dreaming—the art of staying conscious within a dream—has been known and practiced for centuries, a rare skill that allows us to take the reins of our subconscious. Ancient Tibetan Buddhist texts describe dream yoga practices, while modern oneirologists have documented its potential for psychological healing and creative breakthrough.

But what if I told you that this experience of lucid dreaming could be achieved on command, night after night, with the convergence of artificial intelligence and neuroscience?

The Basic Idea, Lucidly Put

AI-driven dream manipulation is about using technology, particularly AI, to monitor, influence, and guide your dreams in real-time. The core idea is to create dream states where you can become aware that you’re dreaming (lucid dreaming) and potentially control or shape your dreams. The process goes somewhat like this:

1. Monitoring an individual’s brain activity to detect specific stages of sleep (like REM- Rapid Eye Movement) using wearables equipped with EEG sensors that monitor your brainwaves.

2. Classifying these different sleep stages so that dream manipulation can be initiated only during appropriate stages.

3. Inducing lucidity with targeted stimulation. Using techniques like focused ultrasound or light and sound cues to stimulate the prefrontal cortex and trigger lucidity (awareness and consciousness while dreaming).

4. Influencing the dream’s content or mood, enhancing control and direction within the dream. This can be done by feedback mechanisms that adjust the stimuli to help steer the dream’s narrative.

5. Personalising the experience over time based on user-specific dream patterns and responses.

The Technical Foundation: Deep Learning in Dream Analysis

Modern dream manipulation rests on sophisticated deep learning architectures that process a complex symphony of sleep-related biosignals.

Deep learning algorithms are particularly adept at handling intricate information, like those generated during your sleep. This data includes brainwave activity, REM cycles, heart rate, muscle activity, breathing patterns, and body temperature.

Convolutional Neural Networks (CNNs) process your EEG spatial patterns to predict and categorise different stages of sleep and dreaming, Long Short-Term Memory (LSTM) networks analyse temporal sequences in your sleep stages, and Transformer models identify recurring dream themes and patterns, providing insights into your mental state and areas of interest.

Developing Neural Interfaces for Inducing and Controlling Dreams

Neural interfaces, or brain-computer interfaces (BCIs), interact with the brain's electrical activity to induce or control dreams. Techniques such as transcranial direct current stimulation (tDCS) and transcranial alternating current stimulation (tACS) apply electrical currents to the scalp (no, we’re not trying to electrocute you :) ), potentially inducing lucid dreams. Neurofeedback devices provide real-time input on brain activity, helping individuals control their brain waves to achieve lucid dreaming.

The Halo: Pioneering Dream Control Technology

The most promising development in this field is the Halo device, developed by Prophetic in collaboration with the Donders Institute for Brain, Cognition, and Behaviour.

This pioneering development by the tech startup, has the ambitious goal of inducing and stabilising lucid dreaming using a combination of AI algorithms and focused ultrasound (FUS) technology. The study aims to determine the device's effectiveness in enhancing participants’ control over dream content, marking a significant step in human-computer interaction and neural modulation.

Methodology

Participants in the study wear the Halo device while sleeping, allowing it to continuously monitor their brain activity through high-resolution EEG sensors. When REM sleep is detected, the device initiates ultrasound stimulation targeting the prefrontal cortex - a critical region for self-awareness and cognitive control. This stimulation is designed to enhance the likelihood of lucid dreaming—a state in which one is aware of and can manipulate the dream.

Key data collected includes:

• EEG Recordings: These capture brain activity patterns that are characteristic of lucid dreaming, providing insights into neurological indicators of dream lucidity.

• Dream Reports: Participants record their dream experiences immediately upon waking, providing qualitative data on lucidity, control over dream narratives, and emotional response to dream scenarios.

Preliminary Results: Early Indications of Effectiveness

Initial results from the study reveal a promising potential for the Halo device to enhance lucid dreaming experiences:

1. Increased Lucid Dreaming Incidence: Participants report a significantly higher frequency of lucid dreams after using the device compared to baseline readings before the study.

2. Enhanced Control in Dreams: Reports also indicate a greater ability to control dream environments and storylines, suggesting the device may facilitate improved mastery over one’s dream state.

Quantitative measures reflect these findings:

• 73% increase in lucid dream frequency among participants.

• 85% of subjects experienced improved control over their dream content and narratives.

• 15-minute extension in lucid dream duration on average.

• 90% dream recall rate post-study, a substantial increase from a baseline of 45%.

Qualitative improvements are also evident:

• Greater clarity and vividness in dream visuals.

• Enhanced control over the dream narrative, leading to a more immersive and directive dreaming experience.

• Notable improvements in emotional regulation, especially during nightmares, allowing participants to navigate distressing dream content with greater ease.

• Sustained awareness across different dream scenarios, contributing to an overarching sense of stability within the dream state.

Expert Insights

Professor Guy Leschziner, a neurologist and sleep medicine specialist, views the findings with cautious optimism. He acknowledges the intriguing potential of using ultrasound to stimulate the prefrontal cortex during REM sleep, yet emphasises the necessity of rigorous studies to evaluate long-term impacts and the ethical implications of brain modulation technology. Leschziner advocates for ongoing, detailed research to better understand any extended effects of frequent lucid dreaming.

Future Directions

Building on these promising findings, Prophetic aims to launch an expanded study involving a year-long brain imaging project slated for late 2024. The broader trial will focus on refining the device’s AI algorithms for more precise stimulation, making it suitable for regular consumer use. The ultimate goal is a universally accessible device capable of reliably inducing lucid dreams, appealing to both researchers and the broader public intrigued by the potential of controlled dreaming experiences.

The Halo device’s unique combination of hardware and AI-driven software represents a milestone in dream control technology.

Technical Components

• Advanced Sensors: High-resolution EEG electrodes, precision ultrasound emitters, and temperature and motion sensors.

• AI Processing Pipeline: Includes real-time signal processing, sleep stage classification, dream state prediction, and ultrasound targeting optimization.

• Neural Interface Mechanisms: Focused ultrasound stimulation specifically targets the prefrontal cortex with a closed-loop feedback system to maintain lucidity. Adaptive stimulation patterns are personalised to each participant’s neural responses.

Unlocking Potential: The Magic of AI-Enhanced Lucid Dreaming

Now imagine diving into your subconscious, where aspirations unfold without limits. AI-driven lucid dreaming technology is opening doors to personal growth, creativity, and professional development.

Picture a public speaker transforming anxiety into confidence by rehearsing presentations in vivid dream simulations. Surgeons, too, can practice complex procedures in hyper-realistic dreamscapes, enhancing their skills and reducing stress.

But it’s not just about skills; lucid dreaming is a powerful emotional exploration tool. Navigate dreams where you confront unresolved feelings and rehearse tough conversations, fostering self-awareness and stronger relationships.

Creatives can find inspiration in dreams, painting entire dreamscapes or interviewing characters to enrich their narratives. In health, chronic pain sufferers can practice relaxation techniques in dreams, while individuals visualise movements to reinforce muscle memory and speed up recovery.

The scientific community is also tapping into this potential, simulating climate change effects and prototyping virtual reality experiences in low-risk environments.

As AI reshapes our relationship with sleep, lucid dreaming evolves into a powerful tool for creativity, healing, and excellence. Welcome to an era where our dreams become a canvas for growth and exploration!

Ethical Considerations and Safety Protocols

Using AI and neural interfaces in dream manipulation raises significant privacy and ethical concerns. Collecting and storing sleep and dream data requires strong security measures to prevent misuse. Participants must give informed consent, understanding the risks and benefits. Ethical guidelines are needed to ensure dream manipulation doesn't harm users and to protect their well-being.

In a Nutshell

As we stand at the threshold of mastering dream consciousness, AI-driven dream manipulation technology promises to unlock the full potential of our sleeping minds. While challenges remain in ethics, safety, and technical implementation, the future of controlled dreaming is rapidly becoming reality.

The question is no longer whether we can control our dreams, but how we'll use this extraordinary capability to enhance human experience, creativity, and healing. As research continues and technology evolves, we're witnessing the dawn of a new era in human consciousness—one where the boundary between waking and dreaming becomes a bridge rather than a barrier.

And hence, the totem tumbles to the ground..

How do LLMs work?

LLMs(Large Language Models) have done the impossible task of speaking and listening like humans do, but underneath the facade of this ingenious marvel is an uncountable amount of mathematics.

The process by which LLMs run is a very complex procedure involving multiple small steps, but broadly speaking they can be divided into the following four:

1. Input: taking in the prompt by the user as input.

2. Tokenization: converting it into smaller parts or individual words.

3. Prediction: predicting output, or the next token based on the prompt through statistics and pattern recognition.

4. Output: the prediction process is repeated until it reaches a specific length or the end of the text.

Just like it took humans years to evolve abnormally large brains relative to the animal kingdom to synthesise speech, LLMs are also extremely resource intensive and need a lot of computational power to run and therein lies the problem.

Weights

The predicted token is decided by a measure called weights.

These weights are floating point real numbers that signify the importance the model gives to certain parts of the text that it has been trained on. This gives the model the ability to find patterns using these numbers, which it then uses to predict a token.

These floating point values coupled with the sheer number of arithmetic calculations that the machine needs to do per token is extremely resource heavy and needs highly capable hardware that isn't available to the average consumer.

This results in the large-scale implementations of LLMs over the web using APIs, and doing the actual computation using cloud infrastructure that uses the hardware that can support generative AI.

However, the rate at which the consumer has adopted LLMs into their daily life has been extremely rapid and the demand has only been increasing. Running costs and API costs are only getting more expensive and the average consumer has now been subject to paywalls for unlimited access to Generative AI.

Quantization

There is a solution on the horizon though- quantization. Put very simply, you shrink the model. How do you do it?

Convert the billions of large floating point weights of the model that make calculations more complex and convert them into integer values using specific algorithms.

For example, converting a 16-bit floating point into a 4-bit integer, or a 32-bit floating point into an 8-bit integer. This makes the millions of calculations that the model has to do much less intensive on the hardware. Simply put 1.5287678 and 1.098764 is much more complex to add then 2 and 1.

The amount of processing power and memory used by quantized models is markedly lesser than their raw counterparts, making it easier to run on consumer hardware.

Quantization Techniques

There are numerous techniques you can use to quantize an LLM, but the easiest to use is weight or static quantization as described earlier. Other techniques include:

• Dynamic quantization: dynamically quantize the weights as needed during inference.

• Quantization aware training: Simulates the effects of quantization while training the model itself.

• Clustered Quantization: clusters similar weights together and replaces them with the centroid of this cluster.

The easiest way to quantize a model is to use llama.cpp, a program that was initially intended to get inference of llama models in pure cpp. However, the library also includes methods to quantize numerous models by using GGUF, a framework and file format that can store and run quantized LLM’s.

The library helps you choose the method of weight quantization which is denoted by an indicator. For example “Q2_k.gguf'' indicates that 2-bit quantization has been used, meaning that each weight can have a possible of 4 (n²) values.

The K here denotes that the K-means clustering algorithm was used, the weights were clustered into 2k clusters and the centroid of these clusters is calculated and taken as the quantized value of all the weights in that cluster. Similarly, there are multiple similar formats, the golden rule being that the higher n bit quantization, the more possible values that a weight can have.

Risks & Tradeoffs

And in this, there is a delicate game to play.

The higher the bit quantization, the more unique values a weight can have, increasing the load on the hardware and memory. On the other hand, too small of a bit quantization and the number of unique weights drastically decreases, decreasing the quality of inference because the precision and uniqueness of the weights is what lends to the quality of inference.

The key is to strike a balance and create harmony between performance and load.

Exercise

The following are some inferences of a model I quantized using the llama.cpp framework, a raw model based on Codellama 7B was used - EvolCodeLLama 7B fine tuned by Mlabonne on Huggingface. The model has been fine tuned to answer coding based questions across all domains using a varied dataset.

The “Q4_K_M” variation was used to quantize this model, i.e., each weight could have a possible 16 values, similar weights were clustered together while quantizing as denoted by the ‘k’ and the residual weights or error weights are then quantized again by clustering these residual weights as denoted by the ‘M’. This approach was in hopes of striking a balance between quality and resource intensiveness.

The model is running completely natively and offline and all the processing is done by the local machine itself, however the processing was offloaded onto a consumer level GPU in the machine as it is more optimised to carry out the processes than a CPU and to take the load of the CPU that is already running the heavy LM studio application, the interface used to interact with the model.

Image 1: A simple question in python that prints the factorial of a non-negative integer, we can see that the code is extremely simple but it does satisfy all the requirements that were asked along with base cases

Image 2: A much more complex question that involves image classification using specific python libraries. Here the LLM does give an accurate answer in terms of what needs to be done, but there is no elaboration, there is no explanation and in these cases, the overall quality of the output is satisfactory at best.

Here we can clearly see the compromise we have to make with quantization, but the flexibility it gives in relation to the many variations of the techniques that exist means that the ratio of performance and load can be altered to fit all kinds of machines with varying hardware capabilities. Ranging from high end consumer PCs that can run high bit quantization models which give higher quality inference to mobile phones that can be optimised to give above average inference even with low bit quantization.

To see the difference in inference that comes with different levels of bit quantization, I used the small 3B parameter phi-2 model created by Microsoft and fine-tuned by TheBloke on Huggingface and ran it on my local machine with the same settings and configurations. I used a Q2_K version with 2 bit quantization and a Q8_0 version with 8 bit quantization, the two extremes of quantization’s for the model. To see the difference in inference the same questions were asked to both these variations to conclude a subjective comparison based on the inference.

Image 3: Q2_K phi-2 model

Image 4 and 5: Q8_0 phi 2 model

Results

If we compare the results:

1. Coherence and Clarity:

   • The response from the 8-bit quantized model provides a more detailed and structured explanation of ocean acidification, its causes, and its impacts on coral reef ecosystems. It follows a logical progression, starting with the mechanism of ocean acidification and then discussing its effects on corals and symbiotic relationships.

   • On the other hand, the response from the 2-bit quantized model appears to be less coherent and concise. It jumps directly into discussing the decline in ocean pH levels without providing as much context or explanation of the underlying processes involved.

2. Accuracy and Detail:

   • The 8-bit quantized model response includes specific scientific terminology and references to research studies, such as mentioning the "carbonate system" and citing studies by Hansen et al. (2007) and Diaz et al. (2006). This indicates a higher level of detail and accuracy in the explanation.

   • In comparison, the 2-bit quantized model response lacks specific scientific terminology and references. It provides a more general overview of ocean acidification without delving into as much detail about the processes involved or supporting evidence.

3. Amount of Data:

   • The response from the 8-bit quantized model appears to contain more information and data, covering various aspects of ocean acidification and its impacts on coral reef ecosystems in depth.

   • In contrast, the response from the 2-bit quantized model seems to be more concise and less detailed, potentially due to limitations in the amount of data that can be processed by the model.

4. Factuality:

   • Both responses convey accurate information about ocean acidification and its effects on coral reef ecosystems. However, the response from the 8-bit quantized model provides more specific details and references to scientific studies, which may enhance its credibility.

The response from the 2-bit quantized model, while accurate in its general statements, lacks the specific scientific references and details that would strengthen its factual accuracy. While the 8 bit quantized model has superior quality, it takes more time to create the inference, requires more memory and processing power to predict tokens as well.

Thus the balance argument is compounded in this demonstration.

What Next?

Just as humanity's linguistic abilities set us apart from the animal kingdom, the development of Large Language Models (LLMs) has redefined our relationship with technology. These marvels of artificial intelligence emulate our capacity for speech and understanding, transforming the way we interact with machines.

However, their immense computational demands have created barriers to widespread adoption, with high costs and resource requirements limiting access.

Quantization emerges as a game-changer, shrinking models to make them more accessible without sacrificing their core capabilities. By converting complex floating-point weights into simpler integer values, quantization strikes a balance between performance and resource efficiency. This innovation democratises LLMs, allowing even consumer-grade hardware to harness their power.

The true potential of LLMs lies in finding this balance, much like our own evolution in language and communication.

By making these models more efficient, we open the door to a future where advanced AI is a tool for everyone, seamlessly integrated into our daily lives. As we refine and optimise these techniques, we pave the way for a new era of human-AI collaboration, unlocking possibilities we have yet to imagine!