Exploring Networking Basics: Creating Protocols with Node.js
As backend or full-stack developers, much of our work revolves around the internet, utilizing protocols like HTTP, FTP, SMTP, and WebSocket, each with its own role in internet communication. Understanding how these protocols function under the surface is crucial for building efficient applications that operate seamlessly on the web. While delving into the intricacies of networking may not be necessary for all developers, having a basic grasp of networking concepts can greatly benefit our work.
In this series, we’ll explore the fundamentals of networking. Although we’ll primarily use Node.js for coding, the concepts we cover are applicable across various programming languages. Our journey begins with an exploration of application layer protocols, such as HTTP. We’ll delve into how these protocols function and their significance in facilitating communication between networked devices. Furthermore, we’ll construct a simple application layer protocol atop TCP in Node.js. Subsequently, we’ll leverage our custom protocol to craft a chat application, showcasing practical implementation of networking concepts in real-world scenarios. Stay tuned for an enriching exploration of networking fundamentals and their application in building robust web applications.
What is TCP?
TCP (Transmission Control Protocol) is like a trustworthy mail delivery service for computers. When you send a message or data over the internet, TCP makes sure it arrives safely and in order, just like how the postal service ensures your letters reach their destination without getting lost or jumbled up. It breaks your data into small pieces called packets, sends them across the internet, and then puts them back together in the correct order when they reach their destination. So, TCP is like the reliable postman of the internet, ensuring your information gets where it needs to go without any hiccups. In the OSImodel, TCP is usually referred to as a transport layer, and many application protocols, like HTTP (Except HTTP/3) and FTP, are built on top of TCP, leveraging its reliable delivery mechanisms for smooth communication between devices.
So, to make our life easier we are going to use TCP to build our simple protocol.
Build Protocol88:
We will call our protocol Protocol88. We want to create a protocol that we will use later to create a chat application. First, let’s think about how we can achieve that! We need a server for sure, which a client can connect to. We have to maintain this connection so the client can send data to the server and the server can send data to the client without the need to fetch for data. And this is another reason why we will use TCP. In TCP, we can create a two-way communication between two endpoints; we call that endpoint a socket.
Under the hood, TCP ensures that our data won’t be lost and that it’s ordered once it arrives. To establish a connection between two different sockets, TCP uses a Three-way Handshake. This handshake involves three steps:
- SYN (Synchronize): The client sends a SYN packet to the server, indicating its intention to initiate a connection.
- SYN-ACK (Synchronize-Acknowledge): The server responds with a SYN-ACK packet, acknowledging the client’s request and indicating its readiness to establish a connection.
- ACK (Acknowledge): Finally, the client sends an ACK packet back to the server, confirming the receipt of the server’s acknowledgment.
Once this three-step process is completed successfully, a connection is established, and data can be exchanged freely between the client and server.
Create a TCP Connection:
In Node.js, we utilize the netmodule to create TCP connections. This module serves as the foundation of networking, offering the lowest level of network operations within Node.js.
First, let’s create a server that listens for incoming connections:
const net = require("net");
// Create a TCP server instance
const server = net.createServer();
// Event listener for incoming connections
server.on("connection", (socket) => {
// Event listener for data received from the client
socket.on("data", (data) => {
console.log(data);
});
});
// Start the server, listening on port 3000 and IP address 127.0.0.1
server.listen(3000, "127.0.0.1", () => {
console.log("Server is listening on ", server.address());
});In this code snippet, we’re using Node.js net module to create a TCP server. This server waits for incoming connections from clients. When a client connects, the server listens for any data sent by the client.
The net.createServer() method returns an instance of the net.Server class, which extends the EventEmitterclass. This class allows us to handle incoming connections. When a client connects to our server, the connection event is emitted. This event provides us with an instance of the net.Socket class, which extends the stream.Duplex class. Since it's a duplex stream, we can both read from it and write to it, enabling communication with the client. Additionally, the net.Socket class is also an EventEmitter. The data event is emitted when the client sends a request, providing us with the data sent by the client as a Buffer or String argument.
The listen() function starts the server to listen for incoming connections on port 3000 and IP address 127.0.0.1.
With our server file executed using Node.js, we initiate the server by running:
$ node server.jsUpon successful execution, the server logs the message:
Server is listening on { address: '127.0.0.1', family: 'IPv4', port: 3000 }Our server is now actively listening for incoming connections on the specified address and port.
To connect to our server, we need a client. The client can be written in any language, as long as it establishes a connection to the specified address and port. This client could be a person, another service, or anything else capable of making network connections.
To keep things simple, let’s create a new JavaScript file and call it “client.js”. We’ll use the net module again to connect to our server.
const net = require("net");
// Establish a connection to the server
const client = net.createConnection({ host: "127.0.0.1", port: 3000 });
// Event listener for data received from the server
client.on("data", (data) => {
console.log(data);
});
// Send data to the server
client.write("Hello! From client");We specify the address and port of our server in the net.createConnection() function. This returns a net.Socket instance, just like the one we use on the server. With this socket, we can interact with the server. Since net.Socket is a duplex stream, we can send data to the server using the write() method and listen for incoming data from the server by listening to the data event.
When you execute the client.js file using Node.js, you should see a log message in the server shell that looks like this:
<Buffer 48 65 6c 6c 6f 21 20 46 72 6f 6d 20 63 6c 69 65 6e 74>This is a buffer representation of the data received from the client. Since we didn’t specify any encoding when logging the data, it is displayed as a buffer.
Create our protocol:
We’ve seen how TCP allows us to establish a connection between two endpoints, enabling free flow of data. However, a simple issue arises: how should we send the data? Since TCP sends raw data, processing incoming data becomes challenging without knowing its format. One client might send a simple string, while another might send data in JSON format or even a file. To simplify data handling for both the server and the client, we need to establish specifications on how data should be sent. In basic terms, this is what a protocol is; a set of specifications and rules that dictate how data should be formatted and transmitted.
We’ll divide the data into two parts: headers and body. Each header should end with a ;, and we must specify the type of data we're sending using the content-type header. For simplicity, we'll use two types: text and json. Each part of the data should start and end with four *, and to combine the two parts, we'll use \n in the middle.
A simple request would look like this:
****content-type:text;****\n****Hello!****According to our format:
****${headers}****\n****${body}****Let’s streamline our project organization by creating a folder named “protocol88” where we’ll put all the files related to our protocol. To simplify data parsing, we’ll create a parser file within this folder. This parser file will contain functions to assist us in parsing incoming data into text or JSON objects, and vice versa. This approach will make our development process more manageable and organized.
function parseHeaderName(header) {
if (header.includes("-")) {
const nameParts = header.split("-");
return (
nameParts[0] + nameParts[1][0].toUpperCase() + nameParts[1].substring(1)
);
}
return header;
}
function extractHeaders(rawRequest) {
return rawRequest
.substring(4, rawRequest.substring(4).indexOf("*") + 3)
.split(";");
}
function extractBody(rawRequest) {
const requestWithoutHeaders = rawRequest.split("\n")[1];
return requestWithoutHeaders.substring(
4,
requestWithoutHeaders.substring(4).indexOf("*") + 4
);
}
function parseBody(contentType, body) {
if (contentType === "json") return JSON.stringify(body);
else return body;
}
module.exports = {
parseToRequest: function (data) {
try {
const headers = extractHeaders(data);
let request = {
headers: {},
};
headers.forEach((h) => {
const [name, value] = h.split(":");
request.headers[parseHeaderName(name)] = value;
});
if (!request.headers.contentType)
throw new Error("ContentType is undefined");
const body = extractBody(data);
if (body) {
if (request.headers.contentType === "json")
request.body = JSON.parse(body);
else request.body = body;
}
return request;
} catch (err) {
throw new Error("Bad Request!");
}
},
parse: function (request) {
return `****content-type:${
request.headers.contentType
};****\n****${parseBody(request.headers.contentType, request.body)}****`;
},
setContentTypeHeader: function (body) {
if (
typeof body === "string" ||
typeof body === "boolean" ||
typeof body === "number"
) {
return "text";
} else if (typeof body === "object") return "json";
else return "unknown";
},
};Here’s the parser.js file. The functions are self-explanatory. We export three functions:
- parseToRequest: This function parses incoming raw data into a request object. Instead of working with the raw data as a string or buffer, we parse it into an object. A simple request object would look like this:
{
headers: {
contentType: 'text'
},
body: 'Hello!'
}- parse: This function converts a request object back into a formatted string that adheres to our protocol.
- setContentTypeHeader: This function determines the content type of the body based on its data type.
Now let’s create our server class that will handle creating a TCP server and uses the parsing functions we defined.
const EventEmitter = require("events");
const net = require("net");
const { parse, parseToRequest, setContentTypeHeader } = require("./parser");
class Server extends EventEmitter {
constructor() {
super();
this.sockets = [];
this.tcpServer = net.createServer();
this.tcpServer.on("connection", this.handleConnection.bind(this));
}
handleConnection(socket) {
this.sockets.push(socket);
socket.on("data", (data) => {
this.emit("request", parseToRequest(data.toString()));
});
socket.on("close", (hadError) => {
this.emit("close", hadError);
});
socket.on("error", (error) => {
this.emit("error", error);
});
}
send(body) {
this.sockets.forEach((socket) => {
socket.write(
parse({
headers: { contentType: setContentTypeHeader(body) },
body: body,
})
);
});
}
startServer(host, port, callback) {
this.tcpServer.listen(port, host, () => {
callback(this.tcpServer.address());
});
}
}
module.exports.Server = Server;The Server class extends EventEmitter. Upon a new connection, it stores the socket in an array to facilitate communication with clients using the send() method. When data is received, the server emits a request event, passing the parsed data as a request object.
With these clarifications, the Server class effectively manages connections, emits events, and facilitates communication with clients in a structured manner.
Similarly let’s create our client class that will take care of establishing connection with a TCP server and sending data to it.
const EventEmitter = require("events");
const net = require("net");
const { parse, parseToRequest, setContentTypeHeader } = require("./parser");
class Client extends EventEmitter {
constructor(host, port) {
super();
this.client = net.createConnection({ host: host, port: port });
this.init();
}
init() {
this.client.on("data", (data) => {
this.emit("response", parseToRequest(data.toString()));
});
this.client.on("error", (error) => {
this.emit("error", error);
});
this.client.on("end", () => {
this.emit("end");
});
}
send(body) {
this.client.write(
parse({
headers: { contentType: setContentTypeHeader(body) },
body: body,
})
);
}
}
module.exports.Client = Client;As we know, the createConnection method in Node.js returns a socket. With this socket, we can communicate with the server by writing data to its socket or reading data from it. When the socket emits a data event, it means that data has been received from the server. In response to this event, we emit a response event from our Client class and pass the received data to any listeners.
This mechanism allows our client application to asynchronously handle responses from the server and take appropriate actions based on the received data. By emitting a response event, we provide a structured way for other parts of the application to respond to incoming data and implement custom logic as needed.
The server and client classes are ready now. To keep things organized and encapsulated, we’re creating a protocol88.js file. From this file, we'll export both the server and the client classes. This approach allows us to neatly package our Protocol88 module, making it easy for developers to integrate and use within their applications.
const { Client } = require("./client");
const { Server } = require("./server");
module.exports = {
server: new Server(),
connectToServer: function (host, port) {
return new Client(host, port);
},
};Test our protocol:
Our protocol implementation is now complete and ready for use. To demonstrate its functionality, we’ll create an index.js file to run the server and a client.js file to act as a client. These files will showcase how to utilize our Protocol88 module in practice.
const { server } = require("./protocol88/protocol88");
server.startServer("127.0.0.1", 3900, (address) => {
console.log("server is listening on ", address);
});
server.on("request", (request) => {
console.log(request);
server.send("Welcome!");
});
server.on("error", (error) => {
console.log(error);
});const { connectToServer } = require("./protocol88/protocol88");
const client = connectToServer("127.0.0.1", 3900);
client.send("Hello! from client");
client.on("response", (response) => {
console.log(response);
});Upon running the index.js file, you'll see the following message indicating that the server is listening on a specific IP address and port:
node index.js
Server is listening on { address: '127.0.0.1', family: 'IPv4', port: 3900 }When you then run the client.js file in another terminal, you'll get a response from the server:
{ headers: { contentType: 'text' }, body: 'Welcome!' }This means the server has successfully received and responded to the client. If you check the server terminal, you’ll find the request sent from the client:
{ headers: { contentType: 'text' }, body: 'Hello! from client' }This shows how the client and server communicate with each other using our Protocol88.
You can find the full source code here.
In our next article, we’ll dive into building a chat application using the Protocol88 we’ve just developed. This served as a simple introduction to networking, and we’ll continue to expand our knowledge as we progress. Feel free to follow me here for updates, or connect with me on LinkedIn or X. See you next time!