JijuTM.COM

Case Study – WIP Reporting and Timeline video on Completion

Requirements (My hallucinations):

Design and architect a highly available, large user base system which is going to be used by the National Highways, the regular employees updating photos of WIP on different stages and when work completed, archive all images after creating a timeline video. WIP sequence should keep the most latest photo thumbnail linked to a project blog page, with a gallery linking to last photo per day. Post-processing of a completed work can take even up to a week giving more importance to the lowest cost possible. The system should be capable of handling hundreds of thousands of high-quality mobile photographs per day. Runtime costs should be as low as possible. For each WIP a minimum of one photograph in six hours is desired.

Solution on AWS (My views):

Application to be developed in some kind of single-page-app with progressive-web-app support, javascript and CSS libraries. This can be hosted on AWS S3 bucket with Cloud Front default origin pointed here. The standard secure approach of https (Redirect HTTP to HTTPS), OAI and custom domain with certificate from ACM is recommended. Dynamic part uses Cognito User Pool, Amazon API Gateway (regional), Lambda, STS etc. API Gateway stage should be the behaviour point for route.

WordPress to Static – Pushing the limits

Well this is not a kind of DIY or hands-on, but just a note on how I am doing it. The architecture of this portal.

The process is as follows

Write an email with the subject having title and a signature, checksum of title plus a predefined string, separated by a double semicolon
Send the same to a virtual id on CloudMailin.
That is routed to an API Gateway, which triggers a lambda.
Lambda (Node.js) evaluates the signature.
If signature is valid, write email json into s3 bucket and triggers an EC2 spot instance with an EBS Volume attached.
The user-data is injected with startup and delayed startup to pick and post the article from S3 into the WordPress on EBS
Inline images are stripped and uploaded into the media manager, and links are replaced appropriately.
Custom scripts utilizing mirror-website and other cli tools will convert the www.jijutm.com website to a static site
WordPress has S3 support for Media Manager through a plugin.
Downloads are directly uploaded into a download bucket.
Then site is synced to s3 with proper expiry headers
CloudFront is invalidated using the aws cli command.
The services are stopped internally and EBS is unmounted
Finally, the EC2 instance is terminated.

The EBS volume was prepared with the data, html and server configuration files. The spot instance is created from a custom ami which is updated time to time and provided to the lambda through environment variables.

This will run for me since I am the sole author of this blog, and my frequency of posting is very low hardly once in two months. For a high frequently updated portal or blog, this process may even fail totally and if there are more than one author, don’t even think of this. I do agree that there are too many cons, like preview editing, making changes etc are not there. But the most important part for me is this WordPress blog is rock solid, Not Hackable, unless AWS S3 or CloudFront is Hacked. Also page load times are pretty good, though tools like google lighthouse or webpage test are still suggesting more improvements.

Export Cloudwatch Logs to AWS S3 – Deploy using SAM

With due reference to the blog which helped me in the right direction, the Tensult blogs article Exporting of AWS CloudWatch logs to S3 using Automation, though at some points I have deviated from the original author’s suggestion.

Some points are blindly my preference and some other due to the suggested best practices. I do agree that starters, would be better off with setting IAM policies with ‘*’ in the resource field. But when you move things into production it is recommended to use least required permissions. Also, some critical policies were missing from the assume role policy. Another unnecessary activity was the checking of the existence of s3 bucket and attempt to create if not exists, at each repeated execution. Again for this purpose the lambda role needed create bucket permission. All these were over my head, and the outcome is this article.

Well if you need CloudWatch logs to be exported to S3 for whatever reason, this could save your time a lot, though this needs to be run in every different region where you need to deploy the stack. Please excuse me as the whole article expects to have aws-cli and sam-cli pre-installed.

Low-Code/No-Code path to Business Applications – AWS Scores again

Introducing HoneyCode a new, fully managed low-code/no-code development tool that aims to make it easy for anybody in a company to build their own applications. All of this, of course, is backed by a database in AWS and a web-based, drag-and-drop interface builder.

Developers can build applications for up to 20 users for free. After that, they pay per user and for the storage their applications take up. There is no wait for applications to be approved on play store / app store as the applications are not directly deployed, rather through a pre deployed player ( interpreter ).

Like similar tools, Honeycode provides users with a set of templates for common use cases like to-do list applications, customer trackers, surveys, schedules and inventory management. Traditionally, AWS argues, a lot of businesses have relied on shared spreadsheets to do these things.

Take advantage of AI/ML to do your Code Reviews and Profiling

Get application performance recommendations and automated code reviews through Amazon CodeGuru, which is a machine learning service. Find the most expensive lines of code that can affect application performance and frustrate you with troubleshooting. The service gives you best recommendations to fix or write better code.

Powered by machine learning, best practices, and hard-learned lessons across millions of code reviews and thousands of applications profiled on open source projects and internally at Amazon, CodeGuru is ready to face any challenge. Find and fix code issues such as resource leaks, potential concurrency race conditions, and wasted CPU cycles, using CodeGuru. Also with moderate, on-demand pricing, it is affordable enough to use for almost all code review and application one might need. Java applications are currently supported by CodeGuru, with support for more languages in the anvil. Catch and resolve problems earlier and with better efficency, with CodeGuru such that you can build and run better software.

EBS Provisioning VS Performance – Confusions cleared

For almost over the last decade ( since 2009 ), I was never worried about the EBS performance indexes. Used to create a single volume and attached to an instance as and when required. Today just for wandering, and to entertain myself, did a couple of tests. Thanks to aws-cli without which this could have taken more than what it would.

Straight into what I found in a short summary. Note that the values are Bps.

	T1	T2	T3	T4	T5	T6	T7
Single	272M	492M	268M	1.3G	393K	272M	8954.02M
Raid 0	631M	671M	740M	1.3G	366K	631M	8851.47
Raid 5	336M	250M	332M	1.2G	9.9k	315M	8306.52

Performance across different combination of EBS Volumes

Kicked up an EC2 instance and mounted a 200gb EBS volume to run a series of tests. Thanks to nixCraft article titled “Linux and Unix Test Disk I/O Performance With dd Command“.

Podcasting Solution on AWS

PodCasting – on AWS can be damn cheap while being ready for a bigbillion hit…

Ingredients..

Amazon S3
Amazon Lambda ( S3 Events [and @ Edge ( if not authenticating from Cognito) ] )
Amazon CloudFront
Amazon Route 53
Amazon Cognito ( optional if social login is required )
A pinch of html and some javascript ( will be provided by me )

S3 stores raw files in one bucket, and trigers lambda to do the transcoding, if mobile from any format to mp3. Meta information should be uploaded to same bucket as flat file. Also multiple quality files will be generated. Interface will upload meta.json and pod.raw files to S3 bucket.

Postman JSON formatting

When we started working on some API First projects, was confused at peer shared Postman exported JSON. And the out come was a jQuery based formatter.

http://bz2.in/e3cbl7

Piwik Analytics Custom Variables Bug

After a long gap, I had the opportunity to dig into Piwik Analytics, the latest version (2.13.1), which has many new features from the last one we were using. During the time when implementing the same, I wanted to apply some custom variables, which showd the logged in user, the internal reference numbers and some other parameters. Whatever I did according to the documentations, the custom variables were not showing up.
Continue reading “Piwik Analytics Custom Variables Bug”

SQL – Always use aggregate functions if you can

One would wonder what the title means. Well it was a thunder bolt for me when I was trying to optimize some headless scripts. Well we at Saturn do heavily use headless scripts written in php. Some use mysql some use xml and some other use memcache, in fact pretty much all would use memcache. But that is not the situation now.

In an attempt to multi thread a cron job part of optimizations done an year back, instead of sequential processing, we had switched it to single row processing. Which required to have a method getNextRow which was passed a parameter, the primary key of the table. The cron starts with a value 0 (zero) and after each is processed, the value is supplied as the last processed one. The getNextRow had

select ID from [table] WHERE ID > [LAST_ID] ORDER BY ID ASC LIMIT 1;

Continue reading “SQL – Always use aggregate functions if you can”