Attempt to create animated representation of AWS DevOps pipeline

Though the title says something technical this is just a self-promotion and cheap boasting

Continuing with the boosting as I have been doing this for the past couple of days. No, I am not insane, but wanted to do this by hand and use some shell commands. Initially the scenes were identified as 10 and folders created with a base flowchart made using Libre Office Draw copied into each of the folders.

Finally the full image sequence was copied into “full” with renaming in sequence with the following command.

Before that, the same command was previewed using echo instead of cp as seen below.

And finally all images were in the “full” folder as below.

It was time to invoke ffmpeg as shown below.

ffmpeg -i dop%04d.png -c:v libx264 -an -r 30 ../dop-anim.mp4 -hide_banner

What could have been achieved with paid tools like Canva or many others, with some effort and free tools available with Ubuntu Linux achieved with minimal expense, without considering my work time earnings that should be a concern.

Complete Managed Development Environment on AWS

Amazon CodeCatalyst, a Unified Software Development Service it was only a few days back that I suggested about Run your Development Environment on Cloud, and as though our dear fellows at AWS had heard my thoughts the preview of Amazon CodeCatalyst was announced two days back as of this post.

As we go through the explanation and blog post we find that it is really intriguing and exciting to hear about the features. Well, I did give a run through the preview and I found that this could change the way we work. At least it did change the way I worked but not for the full-time job as that would violate the compliance complications. But mostly this would be used by me for my leisure time and commitments towards FOSS and my GitHub presence.

Project templates – or blueprints as they define the term do help in fast-tracking the initial development phase and creating a boilerplate to start working. On-demand development environment hosted on the AWS cloud, automated ci-cd pipelines with a multitude of options and drag and drop building, browser-based ide cloud9 with terminal access on the development instance running amazon linux2 which is based out of centos, invite collaborators across the globe to inspect your code with just a few clicks are just a few of the facilities of this unified development environment as service.

I am still very much excited to dig into this service and will go further into this and maybe come out with more like a session with the awsugtvm very soon as time and health permits. Last month I was bedridden after a bike accident involving a stray dog.

Refactored a Complicated Lambda to use Layers and split it up

Till recently, in fact, till last week, was not too worried about writing all code into single code folder, and mapping multiple AWS::Serverless::Function into individual named handlers. Till I stumbled on this article, where I started wondering how my folder structure and sam templates were going into the stack. A detailed inspection was not required, though this was the time when I used the GUI ( after a long time ). But the outcome showed how pathetic the condition was.

The lambda console with the filter “aws:cloudformation:stack-name: <stack>”

Well, it is clear that the whole mess is being uploaded into all the function code. What does this mean – holy grail, any one small change here or there, would update all the functions – last modified is the same, all functions will have the node_modules and other artifacts like templates and custom modules.

Continue reading “Refactored a Complicated Lambda to use Layers and split it up”

Anything with Cloudformation

As part of the DevOps Masters Program on Simplilearn, had to configure a jenkins pipeline. For the same, even though they do provide a lab environment, I feel at home with AWS and cli. I myself being part of the AWS Community Builders, should normally prefer this approach.

For the particular project, the infrastructure was visualized by me as two AWS::EC2 pre deployed one for Jenkins master node, and the other for java+tomcat to deploy a sample app. The Jenkins would be configured with Cloud Plugin configured to manage EC2 nodes for build and test and finally deploy to the tomcat using remote deployment using war. Making the long story short lets jump straight into the steps. Agree that I completed the Project Run in about a couple of hours and creating such a template and running through aws-sam was purely on academic interest. Download the template file: cf-template-ec2-jenkins-tomcat-ubuntu-bionic.yaml

Continue reading “Anything with Cloudformation”

Wild walk with #sam and #aws

Mostly these days, I am working with IaC using aws sam cli which gives me a kind of satisfaction – its cumbersome for me to go into the myriad of web gui and continuously clicking. By creating templates and running them from cli has been my choice for too long.

Getting straight into the job, will summarize the initial requirements and the architecture, then move on to additions.

Required Output

Host a Static Site on S3, deliver it globally through Cloud Front CDN with SSL over HTTPS. Once deployed the Route53 tables should be updated. The deployment should use aws sam cli and IaC.

Though there is not much complication in the architecture, while deploying this during the first pandemic wave, after multiple attempts I found that the cloudfront should be created in specific region such that the SSL certificate can be attached.

Continue reading “Wild walk with #sam and #aws”

WordPress to Static – Pushing the limits

Well this is not a kind of DIY or hands-on, but just a note on how I am doing it. The architecture of this portal.

The process is as follows

  1. Write an email with the subject having title and a signature, checksum of title plus a predefined string, separated by a double semicolon
  2. Send the same to a virtual id on CloudMailin.
  3. That is routed to an API Gateway, which triggers a lambda.
  4. Lambda (Node.js) evaluates the signature.
  5. If signature is valid, write email json into s3 bucket and triggers an EC2 spot instance with an EBS Volume attached.
  6. The user-data is injected with startup and delayed startup to pick and post the article from S3 into the WordPress on EBS
  7. Inline images are stripped and uploaded into the media manager, and links are replaced appropriately.
  8. Custom scripts utilizing mirror-website and other cli tools will convert the www.jijutm.com website to a static site
  9. WordPress has S3 support for Media Manager through a plugin.
  10. Downloads are directly uploaded into a download bucket.
  11. Then site is synced to s3 with proper expiry headers
  12. CloudFront is invalidated using the aws cli command.
  13. The services are stopped internally and EBS is unmounted
  14. Finally, the EC2 instance is terminated.

The EBS volume was prepared with the data, html and server configuration files. The spot instance is created from a custom ami which is updated time to time and provided to the lambda through environment variables.

This will run for me since I am the sole author of this blog, and my frequency of posting is very low hardly once in two months. For a high frequently updated portal or blog, this process may even fail totally and if there are more than one author, don’t even think of this. I do agree that there are too many cons, like preview editing, making changes etc are not there. But the most important part for me is this WordPress blog is rock solid, Not Hackable, unless AWS S3 or CloudFront is Hacked. Also page load times are pretty good, though tools like google lighthouse or webpage test are still suggesting more improvements.