Reference Architecture

Reference architecture for a generic interface for Cloud Search on AWS with a broker in any lambda-supported runtime. For the particular implementation, I chose and used Node.js. Hence any client request is authorized from an API key and hits the aws api gateway which in turn invokes the lambda function. In this function internally the code will do necessary normalization and pass it on to aws Cloud Search and if any response the same is reformatted for adapting as aws api gateway response. Along with this functionality, the lambda broker will write a human-readable version of the request as analyzed from the request with request method as verb keywords and sort direction with a prefix of JSON property names, etc into AWS cloud watch with simple console.log methods. Tried to make it as generic as possible.

An event bridge scheduler will trigger another lambda which will analyze these human readable messages and try to detect any missing indexes which will be auto-created into the Cloud Search and updated into a config file on aws S3. Lots of production testing and fine tuning is pending along with necessary documentation as well as the AWS sam template to deploy the same. As of now, this is just a blueprint and the components are lying in different locations and need orchestration there are no plans to open this on any public repository. But anyone who wants to adopt the design is free to pick this and do it on his own without any commitment to me. By creating this with the self-learning capabilities this system can be used literally by many applications even those that already depend on some kind of custom clumsy backend.

A few real-time use cases could be community member databases, hospital patient records, pet shops and many more. Generally, the request methods should work like POST create a new record, PUT updates a record, DELETE deletes ( or trash ) a referenced record, and GET fetch according with proper documentation the feature can be defined as the client software is designed and developed.

The reference architecture drawing is attached here and that is just my thoughts. Please share if you think this is good enough.

Architecture in a Serverless Mindset

Consider designing a simple serverless system to process orders within an e-commerce workflow. This is an architecture for a REST micro-service that is simple to implement.

Simple Rest Architecture

How an order will be processed by this e-commerce workflow is as follows.

  1. Amazon API Gateway handles requests and responses to those API calls.
  2. Lambda contains the business logic to process the calls.
  3. Amazon DynamoDB provides persistent JSON document storage.

Though this is simple to implement, this can cause bottlenecks and failures resulting in frustrated clients at the web front end. Analyze the flow and see the possible failure points. Amazon API Gateway integrates with AWS Lambda with a synchronous invocation method and expects AWS Lambda to respond within 30 seconds. As long as this happens, all is well and good. But what if a promo gets shared over social media and very large users pile up with orders, Scaling is built into the AWS Services, but can reach the throttling limits.

The configuration of Amazon DynamoDB, where capacity specifications do play a lot. AWS Lambda throttling as well as concurrency can also create failures. Large dynamic library linking which requires initializing time also affects the cold start time and eventually the latency of AWS Lambda which could get lost with the http request timeout of Amazon API Gateway. Getting deep into the system, the business logic could have some complications and in case one request cannot be processed due to the custom code written in AWS Lambda could fail without any trace of the request saved into any persistent storage. Considering all these factors as well as suggestions by veterans in this walk of life this architecture could be further expanded to something like the below.

Revised Order Processing Architecture

What is the revision and what do the additional components provide as advantage? Let’s discuss it now.

  • Order information comes in through an API call over HTTP into Amazon API Gateway
  • AWS Lambda validates and populates the request into Amazon Simple Queue Service (SQS)
  • SQS integrates with AWS Lambda asynchronously and automatic retries for failed requests as well as Dead Letter Queues (left out in illustration) could help out
  • Business logic Processed requests could be stored to DynamoDB
  • DynamoDB Streams could trigger another AWS Lambda to intimate through SNS about the order to Customer Support

Digging more into the illustrations and explanations there are more to be done to make this a full production-ready blueprint let’s leave those thoughts to upcoming Serverless enthusiasts.

Conclusion

I strongly believe that I have been loyal to the core thoughts of being in a Serverless Mindset. Further thoughts of cost optimizing and scaling can be considered with savings plans, AWS Lambda Concurrent provisioning, Amazon DynamoDB on-demand capacity setting and making sure to optimize business logic and reduce latency.

Rearchitecting an Old Solution

It was in 2016 that a friend approached me with a requirement of a solution. They were receiving video files with high resolution into an ftp server which was maintained by the media supplier. They had created an in-house locally hosted solution to show these to the news operators to preview video files and decide where to attach them. They were starting to spread out their news operational desk to multiple cities and wanted to migrate the solution to the cloud, which they did promptly by lift and shift and the whole solution was reconfigured on a large Ec2 instance which had custom scripts to automatically check the FTP location and copy any new media to their local folders. When I was approached, they were experiencing some sluggish streaming from the hosted Ec2 instance as the media files were being accessed from different cities at the same time. Also, the full high-definition videos had to be downloaded for the preview. They wanted to optimize bandwidth utilization and improve the operator’s response times.

Continue reading “Rearchitecting an Old Solution”

CloudFront with multiple origins

A while ago I had bragged about how this site is published – heavily customized WordPress deployed on S3. Though I had left out some of the indigenous parts which I would like to explain in this article. The main aspect explained would be the convergence of multiple origins with CloudFront and configuring behaviours for different cache settings.

Continue reading “CloudFront with multiple origins”

Publish Lambda Layers across Regions

This could even be classified as re:Inventing the wheel but was a quick hack workaround that I found at the early stages where we required a set of node.js libraries and custom modules across multiple regions, in fact, I needed this in only 5 regions. Since these libraries and modules could have frequent updates, I wanted the latest package to be updated and published as a new layer version to each region and concerned developers be notified about the new layer version as well as the changelog. Well, to make the long story short, I was at that time familiar with subversion, and the project code was committed to an svn repository. This could have biased me to use the following solution at that time period, rather now my preference would be either “Multi-Region Deployment” using cloud formation or “CodeCommit and CodePipeline with SNS” instead of S3 triggers.

The architecture is simple and was deployed manually at that time. Sorry to say that I no longer have any access to the aws account, as it is a discarded community project and the account is closed at this time. Thought about posting the idea over here since there was a discussion or rather query about this in a community forum.

Continue reading “Publish Lambda Layers across Regions”

Architecting SaaS Applications on AWS

During my career in different organizations, though I had the opportunity to architect multiple SaaS applications and most of them on AWS, I was not aware that I was already following most of the best practices. Only recently when I had a chance to view the Architecting Next Generation SaaS Applications on AWS by Tod Golding in AWS re:Invent 2016, I came to know the reality.

From the SaaS models which Tod had referred to, I had gone through the Silo, Bridge and Pool over several projects and had been bitten by most of the cons when using Silo I had experienced the maximum complication on the Agile deployment. Since I learned everything from my own experiences at that time, automated deployments or what is known as Continuous Integration and Continuous Deployment using standard tools was not known to me. And mostly I used to re-invent the wheel, using shell, Perl or PHP scripts. Skewed releases across tenant installations were mostly the nightmares.

Continue reading “Architecting SaaS Applications on AWS”

Anything with Cloudformation

As part of the DevOps Masters Program on Simplilearn, had to configure a jenkins pipeline. For the same, even though they do provide a lab environment, I feel at home with AWS and cli. I myself being part of the AWS Community Builders, should normally prefer this approach.

For the particular project, the infrastructure was visualized by me as two AWS::EC2 pre deployed one for Jenkins master node, and the other for java+tomcat to deploy a sample app. The Jenkins would be configured with Cloud Plugin configured to manage EC2 nodes for build and test and finally deploy to the tomcat using remote deployment using war. Making the long story short lets jump straight into the steps. Agree that I completed the Project Run in about a couple of hours and creating such a template and running through aws-sam was purely on academic interest. Download the template file: cf-template-ec2-jenkins-tomcat-ubuntu-bionic.yaml

Continue reading “Anything with Cloudformation”

Wild walk with #sam and #aws

Mostly these days, I am working with IaC using aws sam cli which gives me a kind of satisfaction – its cumbersome for me to go into the myriad of web gui and continuously clicking. By creating templates and running them from cli has been my choice for too long.

Getting straight into the job, will summarize the initial requirements and the architecture, then move on to additions.

Required Output

Host a Static Site on S3, deliver it globally through Cloud Front CDN with SSL over HTTPS. Once deployed the Route53 tables should be updated. The deployment should use aws sam cli and IaC.

Though there is not much complication in the architecture, while deploying this during the first pandemic wave, after multiple attempts I found that the cloudfront should be created in specific region such that the SSL certificate can be attached.

Continue reading “Wild walk with #sam and #aws”

Case Study – WIP Reporting and Timeline video on Completion

Requirements (My hallucinations):

Design and architect a highly available, large user base system which is going to be used by the National Highways, the regular employees updating photos of WIP on different stages and when work completed, archive all images after creating a timeline video. WIP sequence should keep the most latest photo thumbnail linked to a project blog page, with a gallery linking to the last photo per day.  Post-processing of completed work can take even up to a week giving more importance to the lowest cost possible. The system should be capable of handling hundreds of thousands of high-quality mobile photographs per day. Runtime costs should be as low as possible. For each WIP a minimum of one photograph in six hours is desired. 

Solution on AWS (My views):

Application to be developed in some kind of single-page-app with progressive-web-app support, javascript and CSS libraries. This can be hosted on the AWS S3 bucket with Cloud Front default origin pointed here. The standard secure approach of HTTPS (Redirect HTTP to HTTPS), OAI and custom domain with a certificate from ACM is recommended. The dynamic part uses Cognito User Pool, Amazon API Gateway (regional), Lambda, STS etc. API Gateway stage should be the behaviour point for the route.

Continue reading “Case Study – WIP Reporting and Timeline video on Completion”

WordPress to Static – Pushing the limits

Well this is not a kind of DIY or hands-on, but just a note on how I am doing it. The architecture of this portal.

The process is as follows

  1. Write an email with the subject having title and a signature, checksum of title plus a predefined string, separated by a double semicolon
  2. Send the same to a virtual id on CloudMailin.
  3. That is routed to an API Gateway, which triggers a lambda.
  4. Lambda (Node.js) evaluates the signature.
  5. If signature is valid, write email json into s3 bucket and triggers an EC2 spot instance with an EBS Volume attached.
  6. The user-data is injected with startup and delayed startup to pick and post the article from S3 into the WordPress on EBS
  7. Inline images are stripped and uploaded into the media manager, and links are replaced appropriately.
  8. Custom scripts utilizing mirror-website and other cli tools will convert the www.jijutm.com website to a static site
  9. WordPress has S3 support for Media Manager through a plugin.
  10. Downloads are directly uploaded into a download bucket.
  11. Then site is synced to s3 with proper expiry headers
  12. CloudFront is invalidated using the aws cli command.
  13. The services are stopped internally and EBS is unmounted
  14. Finally, the EC2 instance is terminated.

The EBS volume was prepared with the data, html and server configuration files. The spot instance is created from a custom ami which is updated time to time and provided to the lambda through environment variables.

This will run for me since I am the sole author of this blog, and my frequency of posting is very low hardly once in two months. For a high frequently updated portal or blog, this process may even fail totally and if there are more than one author, don’t even think of this. I do agree that there are too many cons, like preview editing, making changes etc are not there. But the most important part for me is this WordPress blog is rock solid, Not Hackable, unless AWS S3 or CloudFront is Hacked. Also page load times are pretty good, though tools like google lighthouse or webpage test are still suggesting more improvements.