HTTP request and response and how web applications work

With so many different programming languages, frameworks, databases, web servers, and other buzz words out there, knowing how everything fits together can be overwhelming.

When I first learned web development, I was also confused about which programming language to start, which framework to study, which database engines I should use, as well as how to scale my server if my application grew and the site started getting lots of traffic.

In this article, I will share an analogy I’ve used in my teaching career that seemed to really help people understand what happens behind the scene for a web server and how all technologies fit together. This concept is called an HTTP request and response cycle and will be explained using an analogy.

By the end of this analogy (in 20–30 minutes), you’ll know how all the different pieces for web come together, including what a back-end is, what a front-end is, how different programming languages fit in, what frameworks are, what databases are and why there are so many of them, as well as what an API is and how it all fits together. This will also give you a good foundation to understand how to troubleshoot any scalability issues.

Before we jump into this analogy, let me first give a very simplified overview of how a web server works.

A high-level overview of how a website is rendered

Imagine that each computer in the world has a unique address, just like how every single house on earth has a unique “residential address”. This unique address for each computer is called an ‘IP address’ and is composed of numbers separated by a period (e.g. 192,135.23.185).

When you open up a browser such as Chrome, Firefox, or Safari, and type in the website URL (e.g. hackerhero.com), your browser doesn’t know which computer out there has information for that site, so it essentially looks a phone book that has all the list of websites in the world and the “IP address” that corresponds to each website. This phone book is basically called a DNS server.

After looking at the DNS server and finding which computer to talk to, the HTTP request is sent to the appropriate server (basically just means a computer). The server processes that request and sends information back to the browser. The browser then renders that information on its screen.

Looking inside what is inside a server

To explain what happens inside a server, let me introduce an “Ikea store analogy”.

Imagine that in this particular Ikea store, things are set up as follows:

The way that this store works is according to these steps:

1. People wait in line to get into the store. Assume that each person in line has already completed an “order form” indicating which furnitures they want to purchase.
2. Once they get inside the store, a receptionist takes the order hands it off to the “workers”.
3. “Workers” speak one common language. They do what is needed to put together the final order. They also visit the warehouse that has all the furniture parts. For this example, let’s say that these workers speak ‘Italian’ to assemble furnitures and communicate with their team members.
4. Once the workers get all the parts they need, they assemble the furnitures together and hand off their work to the receptionist.
5. The receptionist gives the final assembled furnitures to the customer.
6. This step repeats for each customer in the line.

How this applies to a web server

Receptionists in the store represent a web server (this name is confusing as the server is often called also a ‘web server’). The language that the workers spoke represents a programming language. Just like the Ikea store example, the workers are expected to speak one common language between each other. The warehouse represents different databases where the server can store various information

Some key points:

1. A web server such as Apache HTTP or Nginx takes in the customer request and hands it off to a particular programming language.
2. A worker, “speaking” either PHP, Python, C#, Java, or Node.js, assembles furnitures together. They also visit the database to obtain the necessary parts. What language they “speak” to assemble the furniture has little, if any, the implication to the customer. In fact, customers don’t really care whether the furniture was assembled using PHP, Python, C#, Jave, or Node.js.
3. The warehouse represents a database. Just like how there are various ways of organizing furniture inventories, there are various databases optimized for different objectives.
4. The “furnitures” in the analogy represents HTML, CSS, and Javascript. These are the final output of the web server and were assembled by workers who spoke a particular language.
5. The programming instructions inside the server is called a ‘back-end’.
6. The output: HTML, CSS, and Javascript are called “front-end” as these would be displayed on the browser for the user to see. HTML basically contains the texts you see on the website, CSS contains information on where these texts should go and how they should look, and Javascript has information on what to do when the user clicks, hovers, drags an item, etc. In short HTML has structure, CSS its “appearance” or “style”, and Javascript instruction on interaction from the user.
7. It is possible to use any combination of programming languages and databases. They are not dependent on each other. Web servers on the other hand are often built to work with a specific programming language. For example, you can choose to build a server with “Python” as the worker language and choose any database in that list to make your server work.

With this knowledge now, think about the following scenarios:

1. Imagine that you were the warehouse manager and you were given authority to organize the cabinets however you thought would be best for the store. Imagine there were 50 other warehouse managers who were given each a separate warehouse to manage, who were also given the same instruction as you. Would all warehouse managers organize the inventories exactly the same way? Why or why not? This would explain why there are so many different databases in the world.
2. Imagine that you walked into a leadership meeting for the store and they were discussing which worker they should hire. One argues that Korean workers are the best for assembling furnitures. One argues that no, it’s the Spanish workers. Another group argues that it’s ‘English’ and how that’s the best language in the world…. Who is right? Who is wrong? Is one language really better for assembling furnitures than another language? What would you do if you were in that scenario?

Introducing Frameworks

So why is it that some developers prefer one language vs another? Often times, except with Node.js and some other cases, it almost always has to do with “frameworks” that each of the languages comes with.

A framework is essentially a collection of code with certain philosophies about how the workers should operate. A popular framework called an MVC framework, for example, is basically where the workers agreed to break themselves into three sub-groups and do tasks as follows:

1. A group called ‘Models’ (M in the MVC) handles all the work related to the databases.
2. A group called ‘Views’ (V in the MVC) handles putting together the HTML, CSS, and Javascript that is ultimately viewed by the user.
3. A group called ‘Controller’ (C in the MVC) takes the information that the customers are requesting and controls and coordinates who do the work. Think of a controller as the boss for Models and Views where they are managing the flow of information.

For example, to see some of the most popular MVC frameworks (or MVC-like frameworks) for each programing language, see below:

Note that this is not the only way that workers could do the work. MVC framework is just one way of organizing the workers. You could also imagine how other people would disagree that breaking the workers into these three buckets is the right way; maybe they would argue that each worker should do all three tasks, or the work should be split among 4 groups instead, etc.

These difference in opinions about how to best organize the workers is essentially why there are so many frameworks. In fact, someone who disagrees with the approach followed by a framework can spend a few days to create an entirely new framework for people to use and give this framework a new name.

No wonder why there are so many frameworks for each programming language as 1) it’s very easy to create and 2) it’s not possible for every developer to have one identical preference.

For example, if anyone in the world could create a car in a few days and give this car a new name, would you expect to see only one car in the world that everyone loves and drives? In other words, would they all drive one particular car and agree that this particular car is the best car every driver who speaks that particular language should like and use?

On top of this, if a developer could create a “car” in a few days, how long would it take for them to create an “accessory” for that car, and give it a new name? Significantly less! When each of these “accessories” is also given a name, and are also called a ‘framework’ or ‘libraries’ or ‘tools’, no wonder why there are so many frameworks and libraries in the world!

Now, to put this into perspective, imagine the following:

1. say that in order for anyone to create a car, they must pick one of the languages to create a car. Again, once they master one language, they can create a brand new car in a few days.
2. Once a car is built, with certain opinions about how the best car should be like by the person who created the car, other people can adopt that car and use it instead of them creating their own car. Also remember that only those who speak the same language as the creator of that car can drive that car.
3. For each person that accepts a specific car as their “choice” car, they can also create accessories for that car. They can create an accessory even faster than creating a brand new car (say in a few hours). The accessories are also created using the same language as the creator of that car.

Cars in this example represent major frameworks for creating web applications. Accessories represent other frameworks, libraries and tools available for each specific framework.

Even if there were say 7 major programming languages in the world, you can easily imagine how there could be 10 frameworks that people have adopted for each major programming language. Even if each framework only had 50 accessories that went with that framework, you can imagine there would be 7 * 10 * 50 = 3500 frameworks and libraries that developers could use to create a web application!

No wonder recruiters and also new developers can get confused navigating through all these keywords!

Before we put all of this into perspectives, we still need to cover frameworks for front-end. Yes. What I am saying is that 3500+ frameworks/tools/libraries are not the end. There are more…

Frameworks for Front-End

Over the years, web applications have become more complex. In the early days, the HTML, CSS, and Javascript output were relatively small in size. Now, with a lot of interactions and applications that have the feel of a desktop application where everything seems to be fluid and happen in a single page (instead of jumping from one page to another), HTML, CSS, and Javascript files have gone very large.

In order to help developers better organize these codes, they have also created their own way of organizing these codes and created additional tools. These are also called frameworks. For each of these frameworks, they have also created “accessories” to go with each of the frameworks.

jQuery, React, Angular, Ember, Vue.js, BackBone are examples of Javascript Frameworks. With each of these, there are dozens, if not hundreds, of ‘accessories’ that can be used with each of these frameworks. The accessories are also given names and again are often called frameworks, making this ever confusing.

SASS, LESS, Bootstrap, Foundation are more ‘accessories’ rather than a new framework. SASS/LESS allows developers to organize their CSS code with more structure (by creating variables, nesting one CSS inside another, etc). Bootstrap, Foundation, Materialize are CSS files created by other designers/developers that allows developers to make awesome looking websites without having to create their own CSS from scratch.

It’s important to remember that a good developer can create their own framework within days and “accessories” for significantly less time than that.

Back to the diagram

We went detour a bit and talked about how frameworks and accessories for each framework are available for the developers to use. Putting some of the major frameworks for both back-end and front-end, our web server diagram now looks like this:

Again, many developers will have a strong opinion on which framework is better than another. Just like how people have a strong opinion on which car is better than another, this is all just based on preference and taste. Usually as developers get exposed to multiple programming languages and frameworks, their preference for a particular programming language or framework dissipates and they become language agnostic and framework agnostic. Reaching that mastery, often takes a lot of time however.

Other minor detail — thread-based vs event-based web server

In the example shown above, we showed a thread-based web server. Note how there are three lines of customers waiting. That represents three “threads”. A thread-based system is where a customer waits in line until the customer in front of them is served and leaves the thread. It’s a first in, first out system (FIFO in short). Another popular way of handling HTTP requests in a web server is an event-based web server. Nginx is a good example of this.

To visualize how an event-based web server works, imagine that when you go to the store, you don’t wait in line. In fact, there is no line! You simply fill out the order form, hands it off to the receptionist and then wait until the receptionist calls out your number. That’s it. That’s how Nginx and other event-based systems basically operate.

There are pros and cons for each method but if you remember about this analogy and particular web applications you’re building, you’ll know whether an event-based system is what you need or whether you absolutely need a thread-based system. For most of the webs, an event-based system works just fine.

Summary

I hope this analogy helped you understand how all the different technologies come together to create a web application. Please feel free to leave your feedback and share your thoughts.