Architecture – Page 2

Publish Lambda Layers across Regions

This could even be classified as re:Inventing the wheel but was a quick hack workaround that I found at the early stages where we required a set of node.js libraries and custom modules across multiple regions, in fact, I needed this in only 5 regions. Since these libraries and modules could have frequent updates, I wanted the latest package to be updated and published as a new layer version to each region and concerned developers be notified about the new layer version as well as the changelog. Well, to make the long story short, I was at that time familiar with subversion, and the project code was committed to an svn repository. This could have biased me to use the following solution at that time period, rather now my preference would be either “Multi-Region Deployment” using cloud formation or “CodeCommit and CodePipeline with SNS” instead of S3 triggers.

The architecture is simple and was deployed manually at that time. Sorry to say that I no longer have any access to the aws account, as it is a discarded community project and the account is closed at this time. Thought about posting the idea over here since there was a discussion or rather query about this in a community forum.

Architecting SaaS Applications on AWS

During my career in different organizations, though I had the opportunity to architect multiple SaaS applications and most of them on AWS, I was not aware that I was already following most of the best practices. Only recently when I had a chance to view the Architecting Next Generation SaaS Applications on AWS by Tod Golding in AWS re:Invent 2016, I came to know the reality.

From the SaaS models which Tod had referred to, I had gone through the Silo, Bridge and Pool over several projects and had been bitten by most of the cons when using Silo I had experienced the maximum complication on the Agile deployment. Since I learned everything from my own experiences at that time, automated deployments or what is known as Continuous Integration and Continuous Deployment using standard tools was not known to me. And mostly I used to re-invent the wheel, using shell, Perl or PHP scripts. Skewed releases across tenant installations were mostly the nightmares.

Anything with Cloudformation

As part of the DevOps Masters Program on Simplilearn, had to configure a jenkins pipeline. For the same, even though they do provide a lab environment, I feel at home with AWS and cli. I myself being part of the AWS Community Builders, should normally prefer this approach.

For the particular project, the infrastructure was visualized by me as two AWS::EC2 pre deployed one for Jenkins master node, and the other for java+tomcat to deploy a sample app. The Jenkins would be configured with Cloud Plugin configured to manage EC2 nodes for build and test and finally deploy to the tomcat using remote deployment using war. Making the long story short lets jump straight into the steps. Agree that I completed the Project Run in about a couple of hours and creating such a template and running through aws-sam was purely on academic interest. Download the template file: cf-template-ec2-jenkins-tomcat-ubuntu-bionic.yaml

Wild walk with #sam and #aws

Mostly these days, I am working with IaC using aws sam cli which gives me a kind of satisfaction – its cumbersome for me to go into the myriad of web gui and continuously clicking. By creating templates and running them from cli has been my choice for too long.

Getting straight into the job, will summarize the initial requirements and the architecture, then move on to additions.

Required Output

Host a Static Site on S3, deliver it globally through Cloud Front CDN with SSL over HTTPS. Once deployed the Route53 tables should be updated. The deployment should use aws sam cli and IaC.

Though there is not much complication in the architecture, while deploying this during the first pandemic wave, after multiple attempts I found that the cloudfront should be created in specific region such that the SSL certificate can be attached.

Case Study – WIP Reporting and Timeline video on Completion

Requirements (My hallucinations):

Design and architect a highly available, large user base system which is going to be used by the National Highways, the regular employees updating photos of WIP on different stages and when work completed, archive all images after creating a timeline video. WIP sequence should keep the most latest photo thumbnail linked to a project blog page, with a gallery linking to the last photo per day. Post-processing of completed work can take even up to a week giving more importance to the lowest cost possible. The system should be capable of handling hundreds of thousands of high-quality mobile photographs per day. Runtime costs should be as low as possible. For each WIP a minimum of one photograph in six hours is desired.

Solution on AWS (My views):

Application to be developed in some kind of single-page-app with progressive-web-app support, javascript and CSS libraries. This can be hosted on the AWS S3 bucket with Cloud Front default origin pointed here. The standard secure approach of HTTPS (Redirect HTTP to HTTPS), OAI and custom domain with a certificate from ACM is recommended. The dynamic part uses Cognito User Pool, Amazon API Gateway (regional), Lambda, STS etc. API Gateway stage should be the behaviour point for the route.

WordPress to Static – Pushing the limits

Well this is not a kind of DIY or hands-on, but just a note on how I am doing it. The architecture of this portal.

The process is as follows

Write an email with the subject having title and a signature, checksum of title plus a predefined string, separated by a double semicolon
Send the same to a virtual id on CloudMailin.
That is routed to an API Gateway, which triggers a lambda.
Lambda (Node.js) evaluates the signature.
If signature is valid, write email json into s3 bucket and triggers an EC2 spot instance with an EBS Volume attached.
The user-data is injected with startup and delayed startup to pick and post the article from S3 into the WordPress on EBS
Inline images are stripped and uploaded into the media manager, and links are replaced appropriately.
Custom scripts utilizing mirror-website and other cli tools will convert the www.jijutm.com website to a static site
WordPress has S3 support for Media Manager through a plugin.
Downloads are directly uploaded into a download bucket.
Then site is synced to s3 with proper expiry headers
CloudFront is invalidated using the aws cli command.
The services are stopped internally and EBS is unmounted
Finally, the EC2 instance is terminated.

The EBS volume was prepared with the data, html and server configuration files. The spot instance is created from a custom ami which is updated time to time and provided to the lambda through environment variables.

This will run for me since I am the sole author of this blog, and my frequency of posting is very low hardly once in two months. For a high frequently updated portal or blog, this process may even fail totally and if there are more than one author, don’t even think of this. I do agree that there are too many cons, like preview editing, making changes etc are not there. But the most important part for me is this WordPress blog is rock solid, Not Hackable, unless AWS S3 or CloudFront is Hacked. Also page load times are pretty good, though tools like google lighthouse or webpage test are still suggesting more improvements.

Podcasting Solution on AWS

PodCasting – on AWS can be damn cheap while being ready for a bigbillion hit…

Ingredients..

Amazon S3
Amazon Lambda ( S3 Events [and @ Edge ( if not authenticating from Cognito) ] )
Amazon CloudFront
Amazon Route 53
Amazon Cognito ( optional if social login is required )
A pinch of html and some javascript ( will be provided by me )

S3 stores raw files in one bucket, and trigers lambda to do the transcoding, if mobile from any format to mp3. Meta information should be uploaded to same bucket as flat file. Also multiple quality files will be generated. Interface will upload meta.json and pod.raw files to S3 bucket.