Senior Software Engineer (2021 – 2025)

• Worked in the Deliveries organization on creating and enhancing gRPC and REST APIs and microservices that manage food and general delivery (express) orders from creation to completion, at a scale of more than 2 million orders per day.
• Built a Python automation that analyzed AWS infrastructure spending patterns and used ML algorithms to highlight services with anomalous cost increases, then tuned the infrastructure of those services — reducing AWS infra cost across 7 microservices in my team by 30%.

Software Engineer (2019 – 2021)

• Designed and implemented a partner-wise food-order-data streaming and collection pipeline used to share food-order data with big partners across all countries on a daily basis (Kafka, SQS, REST APIs).
• Recognized twice as an employee who went “Above and Beyond” to deliver value to the firm (Feb 2020, Nov 2020).

Tech stack: GoLang, gRPC, Protobuf, Redis, Apache Kafka, Apache Flink, Terraform, GoMock, Testify, AWS (CloudFront, S3, Route 53, Lambda, Lambda@Edge, SQS)

Technology Associate

• Implemented data engineering pipelines fetching high-volume (~500 million rows/day) financial-trade data from files, streams, and databases onto Apache Spark, and ran data quality rules on it to generate post-trade reports.
• Felicitated with the Rookie Award for being the best newcomer in the Institutional Securities Tech division (Apr 2019).

Technology Analyst

• Completed the Technology Analyst Program (TAP) by Mallon Associates — covering operating systems, DBMS, data science, NLP, C++, Java, Scala, and financial-domain training (stocks, bonds, derivatives, markets).
• Built a web-based dashboard collecting Securities Reference Data from multiple sources with visualizations on top (Angular 6, Java 8, Mocha, JUnit).

Tech stack: Apache Spark, Scala, Java 8, Angular 6, JUnit

• Worked in data engineering, creating real-time message pipelines to perform real-time data quality checks on financial orders using business rules.

Tech stack: Apache Kafka, Java 8, Spring, JBoss Drools (rule engine), JUnit

• Worked on communication protocols like MQTT and XMPP to enable home automation by creating smart switches.
• Worked on AWS cloud and developed an Android application through which users control IEDs (Intelligent Electronic Devices).
• Wrote Lua programs for Node MCU and Raspberry Pi devices that receive instructions from the Android application and map them to IED actions.

Areas I worked on:

1. Image downloading and caching
2. Profile activities for coach and athlete
3. Call and message functionality

I worked on Tally Reports, which automatically fetches a company’s accounts data from Tally ERP on any change and displays it to the company owner in a mobile app.

Areas I worked on:

1. Report retrieval from Tally ERP
2. Data retrieval from the server
3. UI/UX — populating charts, ListViews, and CardViews of the “Tally Report Genie” app

AWS offers various certification programs aimed towards Software Development, Solution Architecture, Machine Learning Engineering, Cyber Security, etc.

Taking these certifications is a good idea if:

1. You are someone who has been using AWS services as a part of the solution in the tech stack you are working on for more than a year
2. Or, you are someone who wants to learn more about cloud tech and the services offered and how to use them for your working domain (ML, Cyber Security, Solution Architecture, etc.)

Since I checked both of the above points, I decided to sign up for the certification and cleared it in Dec 2020.

How I prepared for my AWS certifications

There are many different paths available.

1. You can head to the AWS certification page and choose the exam that you want to prepare for. Preparation included:

Source: AWS Certification Prep Homepage

a. Reading standard white-papers illustrating solutions for a given type of business problem.

This gives an idea about general patterns of solutions you are expected to come up with during the examination.

b. Reading up about common services using AWS Documentation (more about it in point 2).

c. Solving sample questions, previous exam questions, and taking a practice exam.

d. You can take AWS Certification online training given by their experts also.

2. For the various services offered by AWS for your domain, browsing into AWS documentation for those services, to understand things like:

Source: AWS CloudFront Documentation

• How is this service priced? What are the different types of variants of the service available?

• Is this service auto-scalable? Is this service serverless?

• Hands-on labs -> Setting up the service in your developer account (in free tier) and trying out requests and checking the responses from the service.

• Comparing the service to its alternative offerings. I.e. RDS for MySQL vs. Aurora for MySQL -> Which service should be used when? What are the pros and cons of the service when compared to its alternatives?

• Understanding where does this service fits in the whole flow. I.e.

  • AWS Cloudfront fits in the flow whenever some content needs to be available to the public at low latencies.
  • AWS DynamoDB and DAX (DynamoDB Accelerator) fits in the flow whenever highly scalable E-commerce, web applications need a highly scalable DB for micro-second latencies.

3. You can also choose to go through a comprehensive certificate preparation course with all of the above-mentioned things (documentation, white-papers, general patterns, sample questions, hands-on labs, etc.) collected in one place.

Some good courses are (in my case, for AWS Solutions Architect Associate Certification):

1. A Cloud Guru Solutions Architect Associate
2. Pluralsight: AWS Certified Solutions Architect — Associate
3. Ultimate AWS Certified Solutions Architect Associate 2021 by Stéphane Maarek

• (In addition to the official AWS websites, I purchased and used this one to prepare)
• Instead of watching the videos. I went through the slides provided for all the videos in this course.
• After completing the slides for one section (i.e. EC2), I would solve the quiz provided for the section.
• After completing going through all slides and quizzes, I would go through sample solution architectures in the course and on the AWS website.
• At last, I solved a full practice paper with 65 questions and a 130-minute time limit, like in the real exam for practice.

Things I learned from the certification

Apart from learning in detail about the AWS services when doing the certification prep, one also learns a lot about system design and how to come up with solution architectures for various business needs.

Most of these concepts are cloud-platform agnostic and help build a foundational understanding of system design, things to consider while designing a new solution, or suite of microservices, or any other web/cloud components talking to one another.

These concepts are wrapped around general cloud platform agnostic (is for any cloud provider Like AWS, Azure, Google Cloud , etc.) questions. For example:

1. What is meant by scalability? (i.e. Horizontal scalability vs vertical scalability, What is auto-scaling, on which metrics can we scale in or out (CPU %, Network in, Network out, etc.)
2. What are the different ways of storing data? Structured (SQL), semi-structured (NoSQL, JSON), unstructured (Files in NFS, tapes, etc.)
3. What is load balancing? Different types like Load Balancing on Application layer vs. Network Layer?
4. What is rate-limiting? And why it is important? What are its different levels (Rate limit on the gateway vs. Rate limit per microservice vs. Rate limit per instance vs. Rate limit per client)
5. What is a firewall? Why is it necessary? How does it prevent internal services from DDoS attacks, etc.?
6. What are the common ports used on different protocols and applications? I.e. Port 22 is for SSH, Port 80 is for HTTP, Port 443 is for HTTPS, Port 3306 is for MySQL, etc.
7. What is caching? Why it is important for performance and saving compute? How to optimally do cache invalidation?
8. Write-through vs. Write-back for writing cache values to DB.
9. Why are CDNs important? How does caching work in CDNs? How does having servers at the edge helpful in reducing latencies to serve consumers?
10. What is serverless architecture? Why is it so popular nowadays? What is a typical serverless alternative to a 3 tier normal cloud architecture having a Gateway, Load Balancer, Scaling Group, Computer Instances, and DB Instances?
11. When to use pub-sub mechanism vs. when to use queues? How these two messaging channels can be used to decouple systems from one another? What is the fan-out architecture?
12. What is the difference between OLTP DB (like Aurora) vs. OLAP DB (like Redshift)? Why are OLAP DBs having columnar storage, unlike OLTP DBs which are row based?
13. How are graph DBs (i.e. Neptune) different from OLTP, OLAP, or NoSQL DBs and when should they be used?
14. How being multi-region or multi-availability helps in being highly available and DR (i.e Disaster recovery) compatible? How does failover happen when one region goes down? (Hint: Based on health checks of the instances)
15. For DBs, how does having read-replicas help in increasing DB performance? Can we couple read replicas and caching for a DB?
16. What is encryption at rest vs. encryption in transit? What is symmetric vs. asymmetric encryption? What is server-side vs. client-side encryption?
17. What is authentication and how it is different from authorization? Why is the least needed privilege ideal for authorization? How to scale authorization using entities like users, groups, roles, and policies? What are the common standards for auth like the OAuth API flow? What is MFA and why is it recommended nowadays days?
18. What is the difference between SDK, framework, and library? And when to provide which of these to the clients?
19. What is whitelisting? What is inbound IP vs. outbound IP whitelisting?

Thanks for reading till the last bit! Wishing you the best with your cloud certification journey!

As the world moves from compute-intensive system architectures to data-intensive system architectures, new technology roles like Product Analysts, Data Analysts, Data Engineers, ML Engineers, and Data Scientist have come up.

So, statistics & probability become important concepts and hence, candidates interviewing for these new roles are generally asked questions to test these concepts.

Even if you are not in these roles, and in the core Software Engineering domain instead, still statistics, probability, and SQL remain the core concepts for beginning to understand any dataset and derive any meaning or predictions out of it, or write any computer logic around it.

Let us look at some of interesting questions that were asked in FAANG interviews for these roles:

1. Amy and Brad take turns in rolling a fair six-sided die. Whoever rolls a “6” first wins the game. Amy has the first turn. What is the probability that Amy wins?

   Solution: The flow of the game is that the die is rolled multiple times till a “6” is rolled.

   So, Amy can win on the first roll, third roll, fifth roll, and so on.

   Probability of Amy winning in the first roll = P(six rolled by her) = 1/6

   Probability of Amy winning in the third roll = P(six NOT rolled by her in first try) * P(six NOT rolled by Brad in first try) * P(six rolled by her in 2nd try) = (5/6) * (5/6) * (1/6) = 1/6 * (5/6)^2

   Similarly, the probability of Amy winning in the fifth roll = (1/6) * (5/6)^4

   Similarly, the probability of Amy winning in the seventh roll = (1/6) * (5/6)^6

   Hence, total probability of Amy winning = Sum of all such events = (1/6) + (1/6 * (5/6)^2) + (1/6 * (5/6)^4) + (1/6 * (5/6)^6) + and so on..

   This series of events is an infinite geometric progression i.e. GP with a = 1/6, and r = (5/6)^2 = 25/36.

   The sum of such infinite GP series is = a/(1-r) = (1/6) / (1 - 25/36) = (1/6) / (11/36) = 6/11

   Hence, probability of Amy winning in any of her turns = 6/11.

   Note: This also means that probability of Brad winning in any of his turns = 1 - P(Amy winning) = 5/11

   Intuition: This shows you that in each board game, the probability of winning for the player that takes the first turn is always slightly greater than his opponent. Intuition is that, yes that should happen right? It makes sense because player one always gets the first chance of making a potential right move in 1st turn, 3rd turn, 5th turn, and so on, while player two gets this chance afterward in this series with 2nd turn, 4th turn, 6th turn, and so on.

2. In a village in India, the authorities were alarmed about a weird family planning custom & started investigating the issue. They found out that in that village, couples were told that if they have a girl child, they should try again for a boy child, and stop only when they have a boy child born. And if they have a boy child born as a firstborn, they should stop planning for more children. Authorities were very worried that this ritual may create gender ratio imbalance. So they hired you, to verify using statistics and probability, if this problem will occur. Calculate the impact of this ritual on gender ratio.

   Solution:

   Long way to solution: Let us take an example case study, suppose 50 couples in the village are fit to have children.

   So, given that every couple will give birth to one child approx. (not considering twins as a case because they are rare), and the probability of a girl or boy being born for an individual birth is 50% each approximately, and that they follow the custom, we can chart down the sequence of births, which is:

   

   As the gender ratio remains approx. 50% as the no. of couples becomes larger and larger than 50. We can conclude that this practice doesn’t have an impact on it.

   Intuition/Short way to solution: The ratio would still be 50% because the probability of a boy or a girl being born is just related to the couple themselves and it is not related to (in other words, is disjoint to) any separate external event of weird rules being followed in society. In other words, the birth of a boy or a girl to any couple is an independent event each time it occurs. Explanation: the probability of a girl or a boy being born to a couple doesn’t get influenced by whether the couple had ‘x’ number of boy or girl child before that. Hence, it always stays approximately equal, at 50% for both the genders. The only way in which gender ratio can be externally influenced, is when someone starts taking away babies of a given gender away from the village, which is not reported in the problem.

3. Four people A, B, C & D get in a lift on the ground floor. Each of them has a choice of getting down on floors 1,2,3 & 4. What is the probability that all of them come out on different floors?

   Solution: Total no. of choices of getting out for each person = 4

   Hence, total number of ways people can get out = 4 * 4 * 4 * 4 = 256

   Now, total number of ways in which people can get out on different floors = ABCD, BACD, …, etc = 4P1 = 24.

   Therefore, probability P(E) = 24/256 = 3/32.

4. Imagine a deck of 500 cards numbered from 1 to 5000. If all the cards are shuffled randomly and you are asked to pick 3 cards, one at a time, what is the probability of each subsequent card being larger than the previously drawn card?

   Solution: Let us assume that the 3 drawn cards have numbers A, B, C, where A < B < C.

   They can be pulled in the sequence of ABC, BAC, … etc.

   Hence, the total numbers of sequences they can be pulled out = 3P1 = 6

   In those 6 sequences, only in ABC sequence, each subsequent card being larger than the previously drawn card.

   Hence, no. of favorable sequences: 1

   Therefore, probability P(E) = 1/6.

Thanks for reading till the last bit! Cheers to learning! :)

The other day, a friend asked me on how I go about choosing a design pattern in a given scenario, because it was not always very clear.

Some design patterns are similar in terms of functionality (i.e. factory vs. builder)

I presented my intuitions via chat, and at the end of the discussion we both realized that with these intuitions for some of the design patterns in mind, choosing next time among them will be easier.

Hence, I thought of documenting and sharing it with everyone.

Factory

Say that you are shopping from a factory outlet of a clothing company. You will only get t-shirts based on the size grades i.e. S, M, L, XL etc.

Hence, single input from our side i.e. S, M, L, XL etc. will define the whole t-shirt.

Similarly, when you want to specify a single input argument like a keyword or an ENum, and get a variation of the main abstract class/interface TShirt created for you like SmallSizeTShirt, then you should go with Factory pattern.

Similar example is covered here, with abstract class/interface Person and its custom variations Villager and CityPerson created for use that are created on the basic of a single input, the enum PersonType.

Factory Design Pattern in C#. Source: Wikipedia

Builder

Now say that instead of going to a factory outlet, you want to have something that fits you better, so you went to a tailor for custom tailored shirt.

Now, the tailor will take note of various other inputs like your chest size, your arm length, your shoulder length etc.

Hence, you can see that the tailor doesn’t have grades based on only one input like size, instead he has a number of input parameters and is makes unique shirts for each customer based on the input values for each of them.

Here, tailor is synonymous with a builder.

Hence, when there are multiple inputs from your side to construct a variation of the main abstract class/interface Shirt, use ShirtBuilder to return a CustomShirt for you based on various inputs.

Similar example is covered here, with ICarBuilder using various inputs like NumDoors, Colour, BrandName and ModelName and some code logic to construct a Ferrari 488 Spider Car.

Builder Design Pattern in C#. Source: Wikipedia

Singleton

Suppose, you have a DB connection to initialize, and you want this DB connection to be established with your DB, and then every time you want to query for some data in a function, you get the same Connection object in establishedConnection state returned, to quickly query data.

In such scenarios where, not only do you want the same type of object to be returned, but you also want it to have a given state always, and for it to be a static method call to get that object, then you use the Singleton pattern.

Additional feature: A thread safe singleton can be created so that singleton property is maintained even in multi-threaded environment. To make a singleton class thread-safe, getInstance() method is made synchronized so that one one thread can access it at a time.

Singleton Design Pattern supporting multi-threading in Java. Source: Wikipedia

Mixins

Suppose you are being told to make a system to accept pizza orders for a restaurant.

Here, class Pizza itself has many variations like ThinCrust, ThickCrust, ExtraCheese etc.

And, on-top of pizza you have a extra class like class Topping which has variations like Mushroom, BlackOlives, Pineapple etc.

Here, any Topping can go with any Pizza and vice versa.

Hence, for this kind of multiple inheritance requirement where:

1. You want to provide a lot of optional features for a class.
2. You want to use one particular feature in a lot of different classes.

For cases like these, you can use the Mixin pattern.

Note: Mixins only exist in languages that support multiple-inheritance. You can’t do a mixin in Java or C#.

Mixin Design Pattern (Pizza-Topping Example) in Python

Output for above example

Adapter

Suppose that you have to build an app interface for users to orders food from different restaurants. Here, menu shown to the user on your app is same for every restaurant, but the order information that each restaurant gets in slightly different.

In cases like these, where one common input has to be mapped to different kinds to input or actions taken we should use the Adapter Pattern.

Intuition is that adapters like charging adapter in real world do exactly the same thing. They send normal AC current coming at a fixed voltage-current range from the main socket into different charger pins for different devices like camera, phone, usb charger pin etc., all with their own voltage-current range.

Similar example is covered here, where different charging styles for LightningPhone and MicroUsbPhone are defined via interfaces.

Class IPhone implements only LightningPhone interface (it can only be charged via Lightning) and class Android implements only MicroUsbPhone interface (it can only be charged via Micro USB).

Now, the class LightningToMicroUsbAdapter helps to implement MicroUsbPhone interface and helps charge class Android with that, which otherwise was not possible.

Adapter Design Pattern in Java. Source: Wikipedia

Reference Links

• Factory pattern
• Builder pattern
• Singleton pattern
• Mixin Pattern: Forum Discussion, Blog
• Adapter Pattern
• Other design patterns and their categories:
  • GangOfFour (Classical book on Design Patterns)
  • Blog

Thanks for reading till the last bit! Cheers to learning! :)

How is ML different from Traditional Programming?

Traditional Programming involves the manual or scaffolded (i.e. already written previously and hence cloned/repeated) logical process of writing a way to convert a certain input into desired output according to certain predefined constraints. So in short:

Traditional Programming:
Input + Program (by coder) = Output

Machine Learning (ML) and its related fields, in contrast, automate this manual process of coming up with the logical program by seeing a big set of correct inputs and their corresponding correct outputs, and “learning” this mapping of input to output called as model.

And then use this generated model with a new set of inputs to predict a new set of outputs. So in short:

Machine Learning:
a.) Input + Output = Model
b.) Model + new Input = new predicted Output

How does this “learning” happen?

“Learning” happens in the training phase when a huge set of correct inputs and their corresponding correct outputs are assembled and used in training to get a model which can then be used to make predictions on new input values.

This huge set of correct input and output together is called the training set.

Finally, when the model is done training, we can then predict the outputs for a new set of input values.

This set is known as the test set on which we test the accuracy of the network.

History

If we look at the world of tech in the last 10 years (2010–2019), one field of study which has made humongous progress is ML.

Before this decade, due to the absence of compute resources supporting heavy parallelization of tasks required for ML algorithms, these algorithms were only used in small pockets of E-Commerce and Digital Marketplace websites to serve recommendations.

At the beginning of the decade, when GPUs started trending and heavy parallelization of calculations was a reality, this field and sub-fields like Deep Learning, other related fields like Computer Vision saw rapid advancements with new State-of-the-Art models released every 3 to 6 months, performing better from the predecessor models in terms of one or more measures like accuracy, generalization, robustness, ease-in-training, and explainability.

What does each of these terms mean?

Accuracy refers to the degree to which the ML model was able to predict the correct output. This output can be a categorical variable (i.e. a yes or a no answer) or a number (i.e. price of certain company’s stock).

Generalization means how an ML model is able to predict for a domain of tasks after being “trained” to make predictions for a similar domain of tasks.

i.e. Can a model trained to distinguish between cats and dogs also be extended to other animals, like a human and a chimp?

Robustness means how an ML model can deal with “weird edge-cases” of input. i.e. Can a model being trained to distinguish between human and dog also distinguish them when the human is wearing a dog-suit and the dog is wearing a t-shirt?

Ease-of-training measures to how much compute resources and training time (i.e. time taken for a model to “learn” to predict correct output given was spent to train a model to reach a given accuracy measure on a varied set of test inputs. i.e. If a model A requires 4 hours of training time to distinguish between a human and a dog with 80% accuracy and a model B does the same in 12 hours of training with same accuracy when tested on 1000 new images of humans and dogs, then model A is better because it takes less time to train and re-train.

Note: Apart from training time, inference time (i.e, time taken for a model to give out prediction for a given input) of different models can also be different, but generally this time is much smaller than the training time, and hence is not that much of an issue unless input supplied is enormously huge.

Notice how similar it is to traditional programming where to solve a task A and give correct output if one piece of code A does it in O(n) linear time complexity, and another piece of code B does it in O(n^2) polynomial time complexity, then the first piece of code A will be considered better because it will perform immensely faster for a huge input.

Explainability refers to the ease with which what the model is learning at each step of training, and what small results it is predicting at each phase of inference can clearly be demonstrated to any person having no prior introduction to ML, who just understands the domain of the problem for which the model was built.

i.e. If a model is used to decide when to sell or buy stocks of a given company. Explaining what factors does the trained model take into consideration after being trained to get to the results to an experienced Technical Stock Analyst by a ML engineer counts as its explainability.

There has been huge push after 2017–2018 for models to be more explainable to domain stakeholders so that questioned like whether it is biased towards a certain type of input, or it treating all input fairly, or is not breaking any rules of the ecosystem can be verified by the domain experts who might not be ML experts.

These advancements garnered much-needed support due to a recent common interest in data protection of consumers and what data points do sophisticated ML algorithms consider for making any decision and do they follow all moral, legal rules while doing so.

What are the different types of ML algorithms?

On the basis of the training set being given to them, ML algorithms are broadly divided into:

Supervised Learning

In supervised learning, both the input (also known as non-target attributes) and its correct output (also known as the target attribute or ground truth) that should be predicted by the model in the best case are provided.

And the task is to learn a function F, that takes the non-target attributes X and output a value that approximates the target attribute, i.e. F(X)≈y. The target attribute y serves as a teacher to guide the learning task since it provides a benchmark on the results of learning. Hence, the task is called supervised learning.

i.e. In the Iris data set, the category of iris flower can serve as a target attribute. The data with a target attribute is often called “labeled” data. Based on the above definition, for the task of predicting the category of iris flower with the labeled data, one can tell that it is supervised learning.

Unsupervised Learning

Different from supervised learning, we do not have the ground truth in an unsupervised learning task. One is expected to learn the underlying patterns or rules from the data, without having the predefined ground truth as the benchmark.

Clustering algorithms are one of the examples of unsupervised learning.

Semi-supervised Learning

In a scenario where the data set is massive but the labeled samples are few, one might find the application of both supervised and unsupervised learning. We can call this task as semi-supervised learning.

i.e. If one would like to predict the label of images, but only 10% of the images are labeled. By applying supervised learning, we train a model with the labeled data, then we apply the model to predict the unlabeled data. It would be hard to convince ourselves that the model would be general enough, after all, we learned from only the minority of data set. A better strategy could be to first cluster the images into groups (unsupervised learning), and then apply the supervised learning algorithm on each of the groups individually.

The unsupervised learning in the first stage could help us to narrow down the scope of learning so that the supervised learning in the second stage could obtain better accuracy.

On the basis of the type of predictions the ML algorithms are making, ML algorithms are broadly divided into:

Regression algorithms

The algorithms that predict a continuous number value, like temperature prediction in a day, or stock price for a given stock at a particular time are called regression algorithms. Examples of regression algorithms are Linear Regression, Multi-variate Regression, etc.

Classification algorithms

The algorithms that predict a discrete categorical value, like the prediction of whether someone has COVID-19 or not based on the chest X-Ray (true or false), or whether the given image has a cat, dog, or human in it (cat or dog or human) are called Classification algorithms.

One example of classification algorithms is Logistic Regression.

What are the common issues with predictions made with ML models?

Although there are many issues with the predictions made with ML models, we are going to focus on two broad types:

Example of underfitting, just-right fit, and overfitting for a Linear Regression model predicting housing prices.

Source: Publicly Open MOOC Machine Learning by Stanford from Coursera

Underfitting

An underfitting model is the one that does not fit well with the training data, i.e. significantly deviated from the training set target variables.

One of the causes of underfitting could be that the model is over-simplified for the data, therefore it is not capable to capture the hidden relationship within the data.

As one can see in the above picture, in part 1, in order to predict the house prices, the almost linear line is not able to predict housing prices correctly and the difference between the predicted price and actual price is too much. Here, a simple linear model (a line) is not capable to “fit” the price curve, which results in significant error is price prediction.

As a countermeasure to avoid the above cause of underfitting, one can choose an alternative algorithm that is capable to generate a more complex model from the training data set.

One can also have a big and diverse training set to avoid the underfitting of the model.

Underfitting is also can be intuitively linked to high bias, as here the model is under the high bias of the principle that the prices of the house increase linearly with the area (size) of the house.

Overfitting

An overfitting model is the one that fits well with the training data, i.e. little or no error, however, it does not generalize well to the unseen data.

Contrary to the case of underfitting, an over-complicated model that is able to fit every bit of the data, would fall into the traps of noises and errors.

As one can see from the above picture, in part 3, the model managed to have mostly zero error for price prediction in the training data, yet it is more likely that it would stumble on the unseen data.

Similar to the underfitting case, to avoid the overfitting, one can try out another algorithm that could generate a simpler model from the training data set.

Alternatively, one stays with the original algorithm that generated the overfitting model, but adds a regularization term to the algorithm, i.e. penalizing the model that is over-complicated so that the algorithm is steered to generate a less complicated model while fitting the data.

Overfitting is also can be intuitively linked to high variance, as here the model is predicting very specific prices decreasing or increasing suddenly the compared to the area of the house just smaller or higher than the current input area of the house because it has overfitted on the training data.

And hence, proving our intuition, the overfitted model has a high statistical variance for the prices of the house when plotted w.r.t. the independent variable, which is the area (size) of the house.

Resources

From here on, you can learn further about the above-presented topics from:

• LeetCode Machine Leaning 101
• Machine Learning Crash Course — Google Developers
• Machine Learning by Stanford from Coursera
• Deep Learning Specialization from Coursera
• fast.ai’s Machine Learning and Deep Learning Course

Thanks for reading till the last bit! Cheers to learning! :)

I’m Ravi Vats, a Lead Data Scientist at Thoughtworks, based in Singapore.

I have spent 7+ years building scalable backend services and AI/ML systems — from transformers and reinforcement learning to production-grade LLM applications. Earlier in my career I built high-traffic Go APIs with rate limiting, caching, and performance optimizations. These days I’m focused on RAG systems and LLM fine-tuning pipelines.

I write about the problems I’m solving and the patterns I discover along the way.

Previously

Before Thoughtworks, I was a Senior Software Engineer at Grab in Singapore, in the Deliveries organization — building Go microservices that handled 2M+ food and express-delivery orders per day. Before that, I worked at Morgan Stanley in the Institutional Securities Tech division, doing backend development and big-data engineering on post-trade reporting pipelines. I’ve also interned at Bosch (research, IoT/home automation) and worked on mobile apps early in my career.

The full history with details is on the Work page.

Education

• Georgia Institute of Technology — M.S. in Computer Science (2021 – 2024)
• Ramaiah Institute of Technology, Bangalore — B.E. in Computer Science (2014 – 2018)

Certifications

• AWS Certified Solutions Architect – Associate
• Microsoft Certified: Azure Fundamentals, Azure Data Scientist Associate

Elsewhere

• GitHub: @ravivats
• LinkedIn: ravi-vats
• X/Twitter: @ravivats_
• Email: mailravivats@gmail.com