Creating an Application Load Balancer and querying it’s logging data using Athena


When building a highly scalable website like amazon.com, there would be thousands of web servers and all of them would be fronted by multiple load balancers as shown below. The end user would be pointing to the load balancer which would forward the requests to the web servers. In the case of the AWS ELB (Elastic Load Balancer), the distribution of the traffic from load balancer to the servers is in a round-robin fashion and doesn’t consider the size of the server or how busy/idle the servers are. May be AWS will add this feature in the upcoming releases.

In this blog, we would be analyzing the number of users coming to a website
from different ip addresses. Here are the steps at a high level which we
would be exploring in a bit more detail. This is again a lengthy post where would be using a couple of AWS services (ELB, EC2, S3 and  Athena) and see how they work together.
   
    – Create two Linux EC2 instance with web servers with different content
    – Create an Application Load Balancer and forward the requests to the above web servers
    – Enable the logging on the Application Load Balancer to S3
    – Analyze the logging data using Athena

To continue further, the following can be done (not covered in this article)

    – Create a Lambda function to call the Athena query at regular intervals
    – Auto Scale the EC2 instances depending on the resource utilization
    – Remove the Load Balancer data from s3 after a certain duration

Step 1: Create two Linux instances and install web servers as mentioned in this blog. In the /var/www/html folder have the files as mentioned below. Ports 22 and 80 have to be opened for accessing the instance through ssh and for accessing the web pages in the browser.

     server1 – index.html
     server2 – index.hml and img/someimage.png

Make sure that ip-server1, ip-server2 and ip-server2/img/someimage.png are accessible from the web browser. Note that the image should be present in the img folder. The index.html is for serving the web pages and also for the health check, while the image is for serving the web pages.

1507838205_762_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838205_27_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 2: Create the Target Group.

1507838205_503_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838205_468_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 3: Attach the EC2 instances to the Target Group.

1507838205_186_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838205_141_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838205_187_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 4: Change the Target Group’s health checks. This will make the instances healthy faster.

1507838205_729_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 5: Create the second Target Group. Associate server2 with the target-group2 as mentioned in the flow diagram.

1507838205_355_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 6: Now is the time to create the Application Load Balancer. This balancer is relatively new when compared to the Classic Load Balancer. Here is there difference between the different Load Balancers. The Application Load Balancer operates at the layer 7 of the OSI and supports host-based and path-based routing. Any web requests with ‘/img/*’ pattern would be sent to the target-group2, rest by default would be sent to target-group1 after completing the below settings.

1507838205_964_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838205_565_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_596_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_45_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_969_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 7: Associate the target-group1 with the Load Balancer, the target-group2 will be associated later.

1507838206_926_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_208_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_6_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_934_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 8: Enable access logs on the Load Balancer by editing the attributes. The specified S3 bucket for storing the logs will be automatically created.

1507838206_160_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_383_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Few minutes after the Load Balancer has been created, the instances should turn into a healthy state as shown below. If not, then maybe one of the above steps has been missed.

Step 9: Get the DNS name of the Load Balancer and open it in the browser to make sure that the Load Balancer is working.

Step 10: Now is the time to associate the second Target Group (target-group2). Click on View/edit rules) and add a rule.

1507838206_498_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_10_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_484_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Any requests with the path /img/* would be sent to the target-group2, rest of them would be redirected to the target-group2.

1507838206_686_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

1507838206_446_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 11: Once the Load Balancer has been accessed from different browsers a couple of times, the log files should be generated in S3 as shown below.

1507838206_170_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

Step 12: Now it’s time to create tables in Athena and then map it to the data in S3 and query the tables. The DDL and the DML commands for Athena can be found here.

1507838207_562_creating-an-application-load-balancer-and-querying-its-logging-data-using-athena Creating an Application Load Balancer and querying it's logging data using Athena

We have seen how to create a Load Balancer, associate Linux web servers with them and finally check how to query the log data with Athena. Make sure that all the AWS resources which have been created are deleted to stop the billing for them.

That’s it for now.


Source link

Previous Learn About The Power Of Live Video With Experts From Experian, Procore, and Performics
This is the most recent story.