How an engineer builds software? Library App

13 min readAug 7, 2023

Interestingly, I noticed that no one shares this type of content. How do you think an engineer builds software? I am a computer engineer developing software and hardware professionally since 2018. My area of expertise is designing software, drawing its architecture, and playing an active role in developing and deploying. Over the years, I have developed dozens of mobile applications, websites, and web services. Now I will tell you how I followed the steps from scratch in order.

Don’t forget to watch the video. I explained in a little bit more detail the steps below!

Let me share all the links in advance so you can browse.

Links:

Detailed diagrams: https://cacoo.com/diagrams/Fn5YRoycQFh6ukzk/2D204

GitHub Web Tier Repository and Detailed Document: https://github.com/paradyo/library-ui

GitHub App Tier Repository and Detailed Document: https://github.com/paradyo/library-app

REST API document: https://documenter.getpostman.com/view/12550271/2s9XxyRDBY

My Website: https://emrecan.co/

First, let’s review our steps.

Listen to the client and take notes.
Set user requirements.
Set technical requirements.
Draw UML diagrams.
Draw system designs.
Draw the AWS architecture.
Draw the ER diagram.
Prepare the REST API design.
Draw the user interface.
Prepare CI/CD flow.
Make the version control system connections.
Write your code.
Do the tests.
Bring the AWS architecture to life.
Bring the CI/CD flow to life.
Deploy.
Track errors.

Of course, these steps can be a little more detailed. But I will try to explain as best I can. I would like to state from the beginning that I did not do everything in the field of coding. I just told you what we should do and why we should do it. You can check the Github repos for coding examples.

Let’s get started.

Listen to the client and take notes.

Imagine. You are speaking with a client. He is constantly trying to tell you the software he wants to make. But I do not think that he will be able to explain everything in detail and clearly. Never! Clients explain everything simply as they can imagine it. You, on the other hand, should take note of his wishes and rewrite them by thinking in more detail in terms of software. It can be difficult to take notes while your client is describing the software. It may be a very logical move to record your speech in digital media. Sometimes you may need to have multiple meetings with the client, not just one. So be prepared carefully. Below, I am sharing what our client wants to do as an example. Read and notice the shortcomings. Even try yourself and try to write user requirements.

What do they want and what is the problem?

Set user requirements.

Yes, we listened to the client and recorded what he said. We have released several user requirements. It’s not over. We will send these requirements to the customer and ask for revision or adjustment again. In this way, we will basically be ready to issue software technical requirements. Examine the user requirements that I have extracted below and try to interpret them.

Set technical requirements.

We talked and discussed with the client and as a result, all the requirements are ready. Now we begin to determine the requirements on the technical side. Some experience will be required at this point. If you’ve tested many things in the industry before, you should be realizing what works best and where. For example, if you need to process 100,000s of data per second, MySQL, unfortunately, will not be a good solution. You should consider NoSQL solutions like DynamoDB or MongoDB. Or, to think more simply, if your client wants to use the same software in a mobile application in the future, designing a REST API should come to your mind directly. Otherwise, you will have to spend more time in the future and make all your services stateless. Of course, this will incur costs. Now let’s examine the technical requirements that I have prepared.

Draw UML diagrams.

Now that we have prepared our technical requirements, we will draw out how the users will do the actions in a loop next. This will be very useful for us to embody everything. It will be very useful for functions we need to code. At the same time, the more detailed we can make these diagrams, the better. I am trying to show an example in general. For example, let’s examine the diagrams for the Admin user.

As you can see, I tried to draw how the Admin user will do a CRUD operation in general. I neglected to take a few notes. If you take notes in this way, you can easily see the points that you can pay attention to while designing your system. At the same time, I would like to point out that this diagram is not at its best at the moment. We also need to add some error statements. For example, the Admin may receive some errors when trying to update an employee. Try to see the potential errors from the front and arrange the flow accordingly.

Do not forget to review the file I shared online for other diagrams.

Draw system designs.

Being able to draw a system design is a very important step in software development. Many people skip this. However, it is not difficult at all. You just need to be able to embody how the functions in your system work. As an example, I want to show how a system works when a site visitor wants to see the books. But there is more than one function on the system like this. You will have to draw them all separately.

Examine and try to interpret.

Now let me tell you a little bit about what’s going on in this design. First of all, sorry for my drawing ability, I can draw better on paper :) Anyway, if a visitor wants to see the books in the library, he sends a request to the server for it. The server, on the other hand, should prefer to look at the caches first, rather than sending a request to the database each time. Because keeping the database busy every time would be extremely costly at peak times. For this, it should examine the caches and then show the books directly to the user more quickly. But the books in caches may not always be up to date. That’s why we have to synchronize data from the database at a certain frequency. Fortunately, Redis and In-Memory cache systems do this for us.

Draw the AWS architecture.

One of my favorite spots. At the same time, I can confidently make such drawings with the confidence of being an AWS Certified Developer. Now that we have prepared the system designs, we can more or less imagine what kind of architecture we need. But preparing an AWS architecture is not that simple. There are some rules and steps we have to follow. Let’s take a look at the design below.

There are still gaps and room for improvement in the design. But remember that every design is open to development and has shortcomings. There are dozens of services on AWS that can improve our design. But my aim is generally to show that we have to apply certain rules correctly. We always need to prepare a highly available and scalable design. At the same time, we need to know the rules of the 6 Pillars of AWS Well-Architected Framework. After making a design, we will need to examine these rules in detail.

Now let’s examine these 6 pillars for our own AWS architecture.

Operational Excellence

The goal of operational excellence is to get new features and bug fixes into clients' hands quickly and reliably. Handling failures via CloudWatch Alerts and continuously integrating the new features via Elastic Beanstalk — CI/CD connections, the architecture has an operational excellence pillar.

Security

The goal of the security pillar is to provide maximum secure architecture. Using https connection helps us to secure the data in transit. And using encrypted RDS services helps us to secure the data at rest. Also, app and data services are in the private subnet within the security group. They don’t retrieve data from the public internet. They only send data to the NAT gateway in a public subnet.

Reliability

The goal of the reliability pillar is to provide maximum availability in case of failure of some services. Using ELB and ASG, we don’t guess the capacity. If something is wrong with the instances they will restart or terminate and create a new one.

Performance Efficiency

The goal of the performance efficiency pillar is to use computing resources efficiently to meet system requirements and to maintain that efficiency as demand changes and technologies evolve. The best way to have this architecture this pillar is by using serverless services like Lambda, Redis etc. In this case, we use a load balancer and auto-scaling groups to meet this pillar. Also for the serverless part, we use Redis Caching.

Cost Optimization

The goal of the cost optimization pillar is to run systems to deliver business value at the lowest price point. We use an auto-scaling group to use only the needed piece of instances. It scale out and scale in automatically. Also, we use ElastiCache for Redis to not send a request to the database every time.

Sustainability

The goal of the sustainability pillar is to address the long-term environmental, economic, and societal impact of your business activities. Using Elastic Beanstalk is giving us the opportunity to manage the framework easily. Also, our choice of region is really important.

I’ll be exploring these 6 pillars in more detail in another Medium post soon. You can follow me on Medium right now :)

Draw the ER diagram.

In fact, if the architecture is ready now, our database is next. Because we cannot move to the REST API part without preparing the database. For this, we need to consider the possible tables in the system. Let’s take a look at the ER diagram I prepared.

As we can see now our users table is linked with roles. ie multiple users can be linked with a single role. We can see many-to-one relationships. At the same time, we see the guest_books table to keep track of which books library visitors borrowed. It’s linked to both the books and the users table. This is one of the simplest diagrams you’ll ever see. Once you understand this, you can develop it further. Because in order to draw diagrams with dozens of tables, you need to be able to draw such simple diagrams.

Prepare the REST API design.

Yes, we have come to one of my favorite areas again. Starting to design a REST API means we are very close to coding now. Let’s not forget that we haven’t written a single line of code yet. Because we should never, ever jump directly to coding. That would be the worst move we could have made.

When designing a REST API, we must comply with coding standards. Because this will be very useful for us when developing in the future. It also allows us to create a scalable structure. Now let’s examine the design I prepared.

I will examine this design in more detail in another article. But in general, I would like to tell you the things that we should definitely pay attention to. First of all, we have 5 different methods in total. We must establish these methods correctly and in a certain standard form. We should know the difference between PUT and PATCH well. We should NEVER update data with the POST method. We should know well which of our endpoints can be cached. In cases where multiple data can come, we should be able to integrate filters and ordering. We should evaluate our authorization options well. We should know how much security we need and choose the Authorization type accordingly. We have to choose which endpoints will need Authorization. We should choose the API versioning type well. Frankly, I prefer to do versioning via headers, not URLs. I think it is more scalable and does not extend the URL unnecessarily.

Draw the user interface.

You need to draw the pages that will be shown to the user. To make it embody is the secret. Simply imagine and draw all the pages individually, as if you were drawing them in a notebook. Create components to implement all your designed UML diagrams. Identify them. It will be very helpful when coding. I’ll just give an example on the Home Page. However, you can see and examine the drawings of other pages from the diagrams link I shared at the beginning.

Yes, if you are ready you will see one of my bad drawings again. But there is nothing to do. As far as I can, I’ll just show you an example of one of the pages in general terms.

As you can see, things get better when coded :)

Prepare CI/CD flow.

The last step before moving on to coding. Drawing the CI/CD stage. This step is actually very important. Because it is a very costly process to build the project every time, zip it manually, connect it to the server from the terminal, and replace it with the current project. Most importantly, it’s not scalable. But everything is automated thanks to the CI/CD process. You are simply pushing the development you have written into your version control system (Github in this example). CI/CD works for you and gets your code live. You just deal with development. Now let’s examine the flow.

As you can see, Jenkins is running to our aid at this stage. One of my favorite CI/CD tools. AWS CodeDeploy is in second place, of course. I’m thinking of preparing an article about CI/CD processes soon. Don’t forget to stay tuned.

Make the version control system connections.

Yes, we are coding. However, we need to keep a record of what we have coded and be able to return to it easily. It’s also a very valuable thing for us to see our progress. That’s why I prefer Git. Create your new repository in your GitHub account and link it with your editors. Create your projects and make your first commits.

At this point, the branch system comes into play. Frankly, if you are working alone, I am against opening different branches and developing there. But if you are working with more than one developer, you should definitely create branches for the developments to be made and send your codes there. Then you can merge the codes from sub-branches to your master/main branch.

Write your code.

Of course, the first site I visit is https://start.spring.io/ since I prefer Spring Boot with Java while coding. Below you can see which services I chose to start.

You can review the GitHub app tier repository for the codes that I actively use. At the same time, I am already going over the codes in the video. From there, you can see in more detail what I use and how.

Do the tests.

We wrote our codes. Now it’s time to test. Of course, we will use Mockito and JUnit for this. Mockito is here for direct testing of our API endpoints. At the same time, it is scalable because it tests on mock data. I want to show an example request.

@Test
    public void getAllRecords_success() throws Exception{
        List<Book> records = new ArrayList<>(Arrays.asList(
                RECORD_1, RECORD_2, RECORD_3,
                RECORD_4, RECORD_5, RECORD_6,
                RECORD_7, RECORD_8, RECORD_9,
                RECORD_10));
        Mockito.when(bookRepository.findAll()).thenReturn(records);

        mockMvc.perform(MockMvcRequestBuilders
                        .get("/books/test/all")
                        .header("Api-Version", "v1.0")
                        .contentType(MediaType.APPLICATION_JSON))
                .andExpect(MockMvcResultMatchers.status().isOk())
                .andExpect(MockMvcResultMatchers.jsonPath("$", Matchers.hasSize(10)))
                .andExpect(MockMvcResultMatchers.jsonPath("$[2].name", Matchers.is("Designing Data-Intensive Applications")));
    }

As you can see, there are no concepts like assertNotNull on Mockito as in JUnit. You are directly imitating requests. To be honest, this is a healthier test method for me.

But let’s take a look at the JUnit test. You see a simple create user unit test.

@Test
@Order(1)
public void testCreateSuccess() {
    User user = new User();
    user.setId(1);
    user.setUsername("john_doe");
    user.setFullName("John Doe");
    user.setEmail("john.doe@example.com");
    user.setPassword("mysecretpassword");
    userRepository.save(user);
    assertNotNull(userRepository.findById(1).get());
}

Bring the AWS architecture to life.

When it comes to bringing our AWS architecture to life, it can be one of the most painful for some. Because the more complex the design you prepare, the more adjustments you will have to make. One of the most complicated parts is dealing with Security Groups. I know because I worked so hard at the time. Because private subnets never receive outside traffic. In order to send traffic to the outside, they must be connected with NAT Gateways in public subnets. This one was just a simple challenge.

I create an environment in Elastic Beanstalk for the example and created an access key for Jenkins.

Bring the CI/CD flow to life.

Since we prefer Jenkins for CI/CD, we need to install it first. I explained this in detail in the Github repository. But one of the most important settings would, of course, be to set a check frequency for CI/CD. It would be best to prepare a CRON job in the Build Triggers section of the Manage Configurations section in Jenkins. You can set it to check the Git webhook every 1 minute.

Deploy.

Now is the time to deploy our project. We shared the documents we wrote. We deployed our project. If our DNS and Nameserver settings are ready, our project will now be accessible from all over the world. Let’s not forget to do our live tests. It’s always best to retest all functions.

Track errors.

Remember that it is not possible to create a system with zero errors or bugs. Every system has bugs. Every system can break and every system can crash. For this, we have to follow the errors both from the client and from AWS services such as CloudWatch, X-Ray, etc. We should take regular database backups and examine the Load Balancer Health Checks.

There are many more things we can improve, I know. But for now, these are the only ones I can think of and write about. I am very excited to create new articles. It makes me very happy to share my experiences with you in this way. At the same time, I contribute a little to the industry.

I wish it to be a day when we break our boundaries and learn new things,

Emrecan Ozkan.