Category: Architecture

Building a Fully Mobile DevOps + Web Dev Stack Using Android + Termux

Overview

This is a journey through my personal developer stack that runs entirely on Android devices using Termux, a few custom scripts, and AWS infrastructure. From hosting WordPress on ECS to building serverless REST APIs in under 90 minutes, every part of this pipeline was built to work on mobile with precision and control.

📱 No laptop. No desktop. Just Android + Termux + Dev discipline.

🔧 Core Stack Components

  • Android + Termux: Primary development environment
  • Docker + Jenkins + MySQL/MariaDB: For CI/CD and content management
  • Static blog pipeline: Converts WordPress to static site with wget, sed, gzip, AWS CLI
  • AWS S3 + CloudFront: Hosting & CDN for ultra-low cost (\$8/year infra)
  • Custom shell scripts: Shared here: GitHub – jthoma/code-collection
  • GitHub integration: Direct push-pull and update from Android environment

🖥️ Development Environment Setup

  • Base OS: Android (Galaxy M14, A54, Tab S7 FE)
  • Tools via Termux: git, aws-cli, nodejs, ffmpeg, imagemagick, docker, nginx, jq, sam
  • Laptop alias (start blog) replaced with automated EC2 instance and mobile scripts
  • Jenkins auto-triggered publish pipeline via shell script and wget/sed

🔐 Smart IP Firewall Update from Mobile

A common challenge while working from mobile networks is frequently changing public IPs. I built a serverless solution that:

  1. Uses a Lambda + API Gateway to return my current public IP

echo-my-ip
https://github.com/jthoma/code-collection/tree/master/aws/echo-my-ip

  1. A script (aws-fw-update.sh) fetches this IP and:
  • Removes all existing rules
  • Adds a new rule to AWS Security Groups with current IP
    → aws-fw-update.sh

🧹 Keeps your firewall clean. No stale IPs. Secure EC2 access on the move.

🎥 FFmpeg & ImageMagick for Video Edits on Android

I manipulate dashcam videos, timestamp embeds, and crops using FFmpeg right inside Termux. The ability to loop through files with while, seq, and timestamp math is far more precise than GUI tools — and surprisingly efficient on Android.

🧠 CLI = control. Mobile ≠ limited.

🌐 Web Dev from Android: NGINX + Debugging

From hosting local web apps to debugging on browsers without dev tools:

  • 🔧 NGINX config optimized for Android Termux
  • 🐞 jdebug.js for browser-side debugging when no console exists
    Just use: jdbg.inspect(myVar) to dump var to dynamically added <textarea>

Tested across Samsung Galaxy and Tab series. Works offline, no extra apps needed.

Case Study: 7-Endpoint API in 80 Minutes

  • Defined via OpenAPI JSON (generated by ChatGPT)
  • Parsed using my tool cw.js (Code Writer) → scaffolds handlers + schema logic
  • Deployed via my aws-nodejs-lambda-framework
  • Backed by AWS Lambda + DynamoDB

✅ Client testing ready in 1 hour 20 minutes
🎯 Client expectation: “This will take at least 1 week”

Built on a Samsung Galaxy Tab S7 FE in Termux. One cliche is that I do have the samsung full keyboard book case cover for the tab.
No IDE. No laptop.

🔁 Flow Diagram:

🔚 Closing Thoughts

This entire DevOps + Dev stack proves one thing:

⚡ With a few smart scripts and a mobile-first mindset, you can build fast, secure, and scalable infrastructure from your pocket.

I hope this inspires other engineers, digital nomads, and curious tinkerers to reimagine what’s possible without a traditional machine.

👉 https://github.com/jthoma/code-collection/

Apart from what explained step by step there are a lot more and most of the scripts are tested on both Ubuntu linux and Android Termux. Go there and explore whatever is there.

💬 Always open to collaboration, feedback, and new automation ideas.

Follow me on linkedin

Build a Spark-Based BI Environment on AWS EC2 Using AWS CLI

Performing business intelligence (BI) analysis using Apache Spark doesn’t need an expensive cluster. In this tutorial, we’ll use AWS CLI to provision a simple but powerful Apache Spark environment on an EC2 instance, perfect for running ad-hoc BI analysis from spreadsheet data. We’ll also cover smart ways to shut down the instance when you’re done to avoid unnecessary costs.

What You’ll Learn

  • Launching an EC2 instance with Spark and Python via AWS CLI
  • Uploading and processing Excel files with Spark
  • Running PySpark analysis scripts
  • Exporting data for BI tools
  • Stopping or terminating the instance post-analysis

Prerequisites

  • AWS CLI installed and configured (aws configure)
  • An existing EC2 Key Pair (.pem file)
  • Basic knowledge of Python or Spark

Step 1: Launch an EC2 Instance with Spark Using AWS CLI

We’ll use an Ubuntu AMI and install Spark, Java, and required Python libraries via user data script.

🔸 Create a user-data script: spark-bootstrap.sh

#!/bin/bash
apt update -y
apt install -y openjdk-11-jdk python3-pip wget unzip
pip3 install pandas openpyxl pyspark findspark matplotlib notebook

wget https://downloads.apache.org/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xvzf spark-3.5.0-bin-hadoop3.tgz
mv spark-3.5.0-bin-hadoop3 /opt/spark

echo 'export SPARK_HOME=/opt/spark' >> /etc/profile
echo 'export PATH=$PATH:$SPARK_HOME/bin' >> /etc/profile
echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /etc/profile

Make it readable:

chmod +x spark-bootstrap.sh

🔸 Launch the EC2 Instance

aws ec2 run-instances \
  --image-id ami-0c94855ba95c71c99 \  # Ubuntu 20.04
  --count 1 \
  --instance-type t3.medium \
  --key-name YOUR_KEY_PAIR_NAME \
  --security-groups default \
  --user-data file://spark-bootstrap.sh \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=SparkBI}]'

Replace YOUR_KEY_PAIR_NAME with your EC2 key name.

🗂️ Step 2: Upload Your Excel File to the Instance

🔸 Find the Public IP of Your Instance

aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=SparkBI" \
  --query "Reservations[*].Instances[*].PublicIpAddress" \
  --output text

Upload your Excel file (sales_report.xls)

scp -i your-key.pem sales_report.xls ubuntu@<EC2_PUBLIC_IP>:/home/ubuntu/

🧠 Step 3: Create and Run Your PySpark Script

sales_analysis.py:

import os
import pandas as pd
from pyspark.sql import SparkSession

xls_file = "sales_report.xls"
csv_file = "sales_report.csv"

df = pd.read_excel(xls_file)
df.to_csv(csv_file, index=False)

spark = SparkSession.builder.appName("SalesBI").getOrCreate()
df_spark = spark.read.csv(csv_file, header=True, inferSchema=True)

# Sample Analysis
df_spark.groupBy("Region").sum("Sales").show()

Run it on EC2:

bash:
spark-submit sales_analysis.py

📊 Step 4: Export Data for BI Tools

You can save output as CSV for use in Power BI, Excel, or Apache Superset:

python:
df_spark.groupBy("Product").sum("Sales").write.csv("product_sales_output", header=True)

Use scp to download:

bash:
scp -i your-key.pem -r ubuntu@<EC2_PUBLIC_IP>:product_sales_output/ .

💰 Step 5: Stop or Terminate EC2 to Save Costs

Stop the Instance (preserves data, costs ~$0.01/hr for EBS)

bash:
aws ec2 stop-instances --instance-ids i-xxxxxxxxxxxxxxxxx

🧭 Pro Tips

  • Use Amazon S3 for persistent storage between sessions.
  • For automation, script the entire process into AWS CloudFormation or a Makefile.
  • If you’re doing frequent BI work, consider using Amazon EMR Serverless or SageMaker Studio.

Conclusion

With just a few CLI commands and a smart use of EC2, you can spin up a complete Apache Spark BI analysis environment. It’s flexible, cost-efficient, and cloud-native.

💡 Don’t forget to stop or terminate the EC2 instance when not in use to save on costs!

Optimizing WordPress Performance with AWS, Docker and Jenkins

At Jijutm.com, I wanted to deliver a fast and reliable experience for our readers. To achieve this, I have implemented a containerized approach using Docker and Jenkins for managing this WordPress site. This article delves into the details of our setup and how it contributes to exceptional website performance.

Why Containers?

Traditional server management often involves installing software directly on the operating system. This can lead to dependency conflicts, versioning issues, and a complex environment. Docker containers provide a solution by encapsulating applications with all their dependencies into isolated units. This offers several advantages:

Consistency: Docker ensures a consistent environment regardless of the underlying operating system. This simplifies development, testing, and deployment.
Isolation: Applications running in containers are isolated from each other, preventing conflicts and improving security.
Portability: Docker containers are portable across different environments, making it easy to migrate your application between development, staging, and production.

The Containerized Architecture

This WordPress site leverages three Docker containers:

  1. Nginx: A high-performance web server that serves the content of this website efficiently.
  2. PHP-FPM: A FastCGI process manager that executes PHP code for dynamic content generation in WordPress.
  3. MariaDB: A robust and popular open-source relational database management system that stores the WordPress data and is fully compatible with MySQL.

These containers work together seamlessly to deliver a smooth user experience. Nginx acts as the front door, handling user requests and routing them to the PHP-FPM container for processing. PHP-FPM interacts with the MariaDB container to retrieve and update website data.

Leveraging Jenkins for Automation

While Docker simplifies application management, automating deployments is crucial for efficient workflow. This is where Jenkins comes in. Jenkins is an open-source automation server that we use to manage the build and deployment process for our WordPress site.

Here’s how Jenkins integrates into this workflow:

  1. Code Changes: Whenever we make changes to the WordPress codebase, we push them to a version control system like Git.
  2. Jenkins Trigger: The push to the Git repository triggers a job in Jenkins.
  3. Build Stage: Jenkins pulls the latest code, builds a new Docker image containing the updated WordPress application, and pushes it to a Docker registry.
  4. Deployment Stage: The new Docker image is deployed to our hosting environment, updating the running containers with the latest code.

This automation ensures that our website stays up-to-date with the latest changes without any manual intervention.

Hooked into WordPress Post or Page Publish.

Over and above maintaining the code using Jenkins, each content Publish action triggers another Jenkins project, which runs a sequence of commands. wget in mirror mode to convert the whole site to static HTML files. sed to rewrite the URLs from local host to realtime external domain specific. gzip to create .html.gz for each HTML files. aws cli to sync the static mirror folder with that in AWS S3 and finally apply meta headers to the files to specify the content type and content-encoding. When all the files are synced, the AWS CLI issues an invalidate request to the CloudFront distribution.

Benefits of this Approach

Improved Performance: Docker containers provide a lightweight and efficient environment, leading to faster loading times for this website.
Enhanced Scalability: I don’t need to bother about scaling this application by adding more containers to handle increased traffic, as that is handled by aws S3 and CloudFront.
Simplified Management: Docker and Jenkins automate a significant portion of the infrastructure management, freeing up time for development and content creation. With the docker and all components running in my Asus TUF A17 Laptop powered by XUbuntu the hosting charges are limited to AWS Route53, AWS S3 and AWS CloudFront only.
Reliable Deployments: Jenkins ensures consistent and reliable deployments, minimizing the risk of errors or downtime.
Well for minimal dynamic content like the download counters, AWS Serverless lambda functions are written and deployed for updating download requests into aDynamoDB table and to display the count near any downloadable content with proper markup. Along with this the comments are moved into Disqus, as it is a comment system that can be used on WordPress sites. It can replace the native WordPress comments system.

Conclusion

By leveraging Docker containers and Jenkins, I have established a robust and performant foundation for this site. This approach allows me to focus on delivering high-quality content to the readers while ensuring a smooth and fast user experience.

Additional Considerations

Security: While Docker containers enhance security, it’s essential to maintain secure practices like keeping Docker containers updated and following security best practices for each service.
Monitoring: Monitoring the health and performance of your containers is crucial. Tools like Docker Stats and Prometheus can provide valuable insights.

Hope this article provides a valuable perspective on how Docker and Jenkins can be used to optimize a WordPress website. If you have any questions, feel free to leave a comment below!

Leveraging WordPress and AWS S3 for a Robust and Scalable Website

Introduction

In today’s digital age, having a strong online presence is crucial for businesses of all sizes. WordPress, a versatile content management system (CMS), and Amazon S3, a scalable object storage service, offer a powerful combination for building and hosting dynamic websites.

Understanding the Setup

To effectively utilize WordPress and S3, here’s a breakdown of the key components and their roles:

  1. WordPress:
  • Content Management: WordPress provides an intuitive interface for creating and managing website content.
  • Plugin Ecosystem: A vast array of plugins extends WordPress’s functionality, allowing you to add features like SEO, e-commerce, and security.
  • Theme Customization: You can customize the appearance of your website using themes, either by choosing from a wide range of pre-built themes or creating your own. Get it free from the maintainers directly and free: https://wordpress.org/download/
  1. AWS S3:
  • Scalable Storage: S3 offers virtually unlimited storage capacity to accommodate your website’s growing content.
  • High Availability: S3 ensures your website is always accessible by distributing data across multiple servers.
  • Fast Content Delivery: Leveraging AWS CloudFront, a content delivery network (CDN), can significantly improve website performance by caching static assets closer to your users.

The Deployment Process

Here’s a simplified overview of the deployment process:

  1. Local Development:
  • Set up a local WordPress development environment using tools like XAMPP, MAMP, or Docker.
  • Create and test your website locally.
  1. Static Site Generation:
  • Use a tool like WP-CLI or a plugin to generate static HTML files from your WordPress site.
  • This process converts dynamic content into static files, which can be optimized for faster loading times.
  1. S3 Deployment:
  • Upload the generated static files to an S3 bucket.
  • Configure S3 to serve the files directly or through a CloudFront distribution.
  1. CloudFront Distribution:
  • Set up a CloudFront distribution to cache your static assets and deliver them to users from edge locations.
  • Configure custom domain names and SSL certificates for your website.

Benefits of Using WordPress and S3

  • Scalability: Easily handle increased traffic and content without compromising performance.
  • Cost-Effective: S3 offers affordable storage and bandwidth options.
  • High Availability: Ensure your website is always accessible to users.
  • Security: Benefit from AWS’s robust security measures.
  • Flexibility: Customize your website to meet your specific needs.
  • Performance: Optimize your website’s performance with caching and CDN.

Conclusion

By combining the power of WordPress and AWS S3, you can create a robust, scalable, and high-performance website. This setup offers a solid foundation for your online presence, whether you are a small business owner or a large enterprise.

Start your cloud journey for free today with AWS! Sign up now: https://aws.amazon.com/free/

Choosing the Right Database for High-Performance Web Applications on AWS

In any web application project, selecting the optimal database is crucial. Each project comes with unique requirements, and the final decision often depends on the data characteristics, the application’s operational demands, and future scaling expectations. For my most recent project, choosing a database meant evaluating a range of engines, each with strengths and trade-offs. Here, I’ll walk through the decision-making process and the architecture chosen to meet the application’s unique needs using AWS services.

Initial Considerations

When evaluating databases, I focused on several key factors:

  • Data Ingestion and Retrieval Patterns: What type of data will be stored, and how will it be accessed or analyzed?
  • Search and Select Complexity: How complex are the queries, and do we require complex joins or aggregations?
  • Data Analysis Needs: Will the data require post-processing or machine learning integration for tasks like sentiment analysis?

The database engines I considered included MariaDB, PostgreSQL, and Amazon DynamoDB. MariaDB and PostgreSQL are widely adopted relational databases known for reliability and extensive features, but DynamoDB is particularly designed to support high-throughput applications on AWS, making it a strong candidate.

The Project’s Data Requirements

This project required the following data structure:

  • Data Structure: Each row was structured as JSON, with a maximum record size of approximately 1,541 bytes.
  • Attributes: Each record included an asset ID (20 chars), user ID (20 chars), a rating (1 digit), and a review of up to 1,500 characters.
  • Scale Expectations: Marketing projections suggested rapid growth, with up to 100,000 assets and 50,000 users within six months, resulting in a peak usage of about 5,000 transactions per second. Mock Benchmarks and Testing

To ensure scalability, I conducted a benchmarking exercise using Docker containers to simulate real-world performance for each database engine:

  • MariaDB and PostgreSQL: Both performed well with moderate loads, but resource consumption spiked sharply under simultaneous requests, capping at around 50 transactions per second before exhausting resources.
  • Amazon DynamoDB: Even on constrained resources, DynamoDB managed up to 24,000 requests per second. This performance, combined with its fully managed, serverless nature and built-in horizontal scaling capability, made DynamoDB the clear choice for this project’s high concurrency and low-latency requirements. Amazon DynamoDB – The Core Database

DynamoDB emerged as the best fit for several reasons:

  • High Availability and Scalability: With DynamoDB, we can automatically scale up or down based on traffic, and AWS manages the underlying infrastructure, ensuring availability across multiple regions.
  • Serverless Architecture Compatibility: Since our application was API-first and serverless, built with AWS Lambda in Node.js and Python, DynamoDB’s seamless integration with AWS services suited this architecture perfectly.
  • Flexible Data Model: DynamoDB’s schema-less, JSON-compatible structure aligned with our data requirements.

Read more about Amazon DynamoDB.

Extending with Sentiment Analysis: The DynamoDB and Elasticsearch Combo

The project’s requirements eventually included sentiment analysis and scoring based on user reviews. Full-text search and analysis aren’t DynamoDB’s strengths, especially considering the potential cost of complex text scanning. So, we created a pipeline to augment DynamoDB with Amazon OpenSearch Service (formerly Elasticsearch Service), which can handle complex text indexing and full-text queries more cost-effectively.

  • DynamoDB Streams: Enabled DynamoDB Streams to capture any changes to the data in real time. Whenever a new review was added, it triggered a Lambda function.
  • Lambda Processing: The Lambda function post-processed the data, calculating preliminary sentiment scores and preparing it for indexing in Amazon OpenSearch Service.
  • OpenSearch Indexing: The review data, now pre-processed, was indexed in OpenSearch for full-text search and analytics. This approach allowed efficient searching without burdening DynamoDB.

Read more about Amazon OpenSearch Service.

Leveraging Amazon S3 and AWS Athena for Historical Analysis

With time, the volume of review data would grow significantly. For long-term storage and further analysis, we used Amazon S3 as a durable and cost-effective storage solution. Periodically, the indexed data in OpenSearch was offloaded to S3 for deeper analysis using Amazon Athena.

  • Amazon S3: Enabled periodic data archiving from OpenSearch, reducing the load and cost on OpenSearch. S3 provided a low-cost, durable storage solution with flexible retrieval options.
  • Amazon Athena: Athena allowed SQL querying on structured data in S3, making it easy to run historical analyses and create reports directly from S3 data.

This setup supported large-scale analytics and reporting, allowing us to analyze review trends and user feedback without overburdening the application database.

Read more about Amazon S3 and Amazon Athena.

Final Architecture and Benefits

The final architecture leveraged AWS’s serverless services to create a cost-effective, high-performance database system for our application. Here’s a breakdown of the components and their roles:

  • DynamoDB: Primary database for high-throughput, low-latency data storage.
  • DynamoDB Streams & Lambda: Enabled real-time data processing and integration with OpenSearch.
  • Amazon OpenSearch Service: Provided efficient full-text search and sentiment analysis.
  • Amazon S3 & Athena: Archived data and performed large-scale, cost-effective analytics.

This combination of DynamoDB, OpenSearch, and S3, with Athena for analytics, proved to be an efficient architecture that met all project requirements. The AWS ecosystem’s flexibility allowed us to integrate services tailored to each specific need, maintaining cost-effectiveness and scalability.

  • #DynamoDB #OpenSearch #AmazonS3 #AWSAthena #AWSLambda #Serverless #DatabaseSelection #CloudArchitecture #DataPipeline

This architecture and service setup provides a powerful example of how AWS’s managed services can be leveraged to achieve cost-effective performance and functionality.

Automating Laptop Charging with AWS: A Smart Solution to Prevent Overheating

In today’s fast-paced digital world, laptops have become indispensable tools. However, excessive charging can lead to overheating, which can significantly impact performance and battery life. In this blog post, we’ll explore a smart solution that leverages AWS services to automate laptop charging, prevent overheating, and optimize battery health. I do agree that Asus does provide premium support for a subscription, but this research and excercise was to brush up my brains and learn to create on aws with some useful solution. The solution is still in concept and once I start using it in production to the full extend, the shell scripts and cloudformation template will be pushed into github handle jthoma repository code-collection/aws

Understanding the Problem:

Overcharging can cause the battery to degrade faster and generate excessive heat. Traditional manual charging methods often lead to inconsistent charging patterns, potentially harming the battery’s lifespan.

The Solution: Automating Laptop Charging with AWS

To address this issue, we’ll utilize a combination of AWS services to create a robust and efficient automated charging system:

  1. AWS IoT Core: Purpose: This service enables secure and reliable bi-directional communication between devices and the cloud.
    How it’s used: We’ll connect a smart power outlet to AWS IoT Core, allowing it to send real-time battery level data to the cloud.
    Link: https://aws.amazon.com/iot-core/
    Getting Started: Sign up for an AWS account and create an IoT Core project.
  2. AWS Lambda: Purpose: This serverless computing service allows you to run code without provisioning or managing servers.
    How it’s used: We’ll create a Lambda function triggered by IoT Core messages. This function will analyze the battery level and determine whether to charge or disconnect the power supply.
    Link: https://aws.amazon.com/lambda/
    Getting Started: Create a Lambda function and write the necessary code in your preferred language (e.g., Python, Node.js, Java).
  3. Amazon DynamoDB: Purpose: This fully managed NoSQL database service offers fast and predictable performance with seamless scalability.
    Link: https://aws.amazon.com/dynamodb/
  4. Amazon CloudWatch: Purpose: This monitoring and logging service helps you collect and analyze system and application performance metrics.
    How it’s used: We’ll use CloudWatch to log system health and generate alarms based on battery level or temperature threshold. Also it helps to monitor the performance of our Lambda functions and IoT Core devices, ensuring optimal system health.
    Link: https://aws.amazon.com/cloudwatch/
    Getting Started: Configure CloudWatch to monitor your AWS resources and set up alarms for critical events.

How it Works:

  1. Data Collection: My Ubuntu system with the help of a shell script uses aws cli to send real-time battery level data to the cloud watch logs.
  2. Data Processing: Cloud watch metric filter alarms will trigger lambda function which is set for appropriate actions.
  3. Action Execution: The Lambda function sends commands to the smart power outlet to control the charging process.
  4. Data Storage: Historical battery level data is stored in Cloud Watch logs for analysis using Athena and further optimization.
  5. Monitoring and Alerting: CloudWatch monitors the system’s health and sends alerts if any issues arise.

Benefits of Automated Charging:

Optimized Battery Health: Prevents overcharging and undercharging, extending battery life.
Reduced Heat Generation: Minimizes thermal stress on the laptop.
Improved Performance: Ensures optimal battery performance, leading to better system responsiveness.
Energy Efficiency: Reduces energy consumption by avoiding unnecessary charging.

Conclusion

By leveraging AWS services, a sophisticated automated charging system that safeguards the laptop’s battery health and enhances its overall performance is reached. This solution empowers you to take control of your device’s charging habits and enjoy a longer-lasting, cooler, and more efficient laptop.

Start Your AWS Journey Today, Signup for free !

Ready to embark on your cloud journey? Sign up for an AWS account and explore the vast possibilities of cloud computing. With AWS, you can build innovative solutions and transform your business.

Reference Architecture

Reference architecture for a generic interface for Cloud Search on AWS with a broker in any lambda-supported runtime. For the particular implementation, I chose and used Node.js. Hence any client request is authorized from an API key and hits the aws api gateway which in turn invokes the lambda function. In this function internally the code will do necessary normalization and pass it on to aws Cloud Search and if any response the same is reformatted for adapting as aws api gateway response. Along with this functionality, the lambda broker will write a human-readable version of the request as analyzed from the request with request method as verb keywords and sort direction with a prefix of JSON property names, etc into AWS cloud watch with simple console.log methods. Tried to make it as generic as possible.

An event bridge scheduler will trigger another lambda which will analyze these human readable messages and try to detect any missing indexes which will be auto-created into the Cloud Search and updated into a config file on aws S3. Lots of production testing and fine tuning is pending along with necessary documentation as well as the AWS sam template to deploy the same. As of now, this is just a blueprint and the components are lying in different locations and need orchestration there are no plans to open this on any public repository. But anyone who wants to adopt the design is free to pick this and do it on his own without any commitment to me. By creating this with the self-learning capabilities this system can be used literally by many applications even those that already depend on some kind of custom clumsy backend.

A few real-time use cases could be community member databases, hospital patient records, pet shops and many more. Generally, the request methods should work like POST create a new record, PUT updates a record, DELETE deletes ( or trash ) a referenced record, and GET fetch according with proper documentation the feature can be defined as the client software is designed and developed.

The reference architecture drawing is attached here and that is just my thoughts. Please share if you think this is good enough.

Architecture in a Serverless Mindset

Consider designing a simple serverless system to process orders within an e-commerce workflow. This is an architecture for a REST micro-service that is simple to implement.

Simple Rest Architecture

How an order will be processed by this e-commerce workflow is as follows.

  1. Amazon API Gateway handles requests and responses to those API calls.
  2. Lambda contains the business logic to process the calls.
  3. Amazon DynamoDB provides persistent JSON document storage.

Though this is simple to implement, this can cause bottlenecks and failures resulting in frustrated clients at the web front end. Analyze the flow and see the possible failure points. Amazon API Gateway integrates with AWS Lambda with a synchronous invocation method and expects AWS Lambda to respond within 30 seconds. As long as this happens, all is well and good. But what if a promo gets shared over social media and very large users pile up with orders, Scaling is built into the AWS Services, but can reach the throttling limits.

The configuration of Amazon DynamoDB, where capacity specifications do play a lot. AWS Lambda throttling as well as concurrency can also create failures. Large dynamic library linking which requires initializing time also affects the cold start time and eventually the latency of AWS Lambda which could get lost with the http request timeout of Amazon API Gateway. Getting deep into the system, the business logic could have some complications and in case one request cannot be processed due to the custom code written in AWS Lambda could fail without any trace of the request saved into any persistent storage. Considering all these factors as well as suggestions by veterans in this walk of life this architecture could be further expanded to something like the below.

Revised Order Processing Architecture

What is the revision and what do the additional components provide as advantage? Let’s discuss it now.

  • Order information comes in through an API call over HTTP into Amazon API Gateway
  • AWS Lambda validates and populates the request into Amazon Simple Queue Service (SQS)
  • SQS integrates with AWS Lambda asynchronously and automatic retries for failed requests as well as Dead Letter Queues (left out in illustration) could help out
  • Business logic Processed requests could be stored to DynamoDB
  • DynamoDB Streams could trigger another AWS Lambda to intimate through SNS about the order to Customer Support

Digging more into the illustrations and explanations there are more to be done to make this a full production-ready blueprint let’s leave those thoughts to upcoming Serverless enthusiasts.

Conclusion

I strongly believe that I have been loyal to the core thoughts of being in a Serverless Mindset. Further thoughts of cost optimizing and scaling can be considered with savings plans, AWS Lambda Concurrent provisioning, Amazon DynamoDB on-demand capacity setting and making sure to optimize business logic and reduce latency.

Rearchitecting an Old Solution

It was in 2016 that a friend approached me with a requirement of a solution. They were receiving video files with high resolution into an ftp server which was maintained by the media supplier. They had created an in-house locally hosted solution to show these to the news operators to preview video files and decide where to attach them. They were starting to spread out their news operational desk to multiple cities and wanted to migrate the solution to the cloud, which they did promptly by lift and shift and the whole solution was reconfigured on a large Ec2 instance which had custom scripts to automatically check the FTP location and copy any new media to their local folders. When I was approached, they were experiencing some sluggish streaming from the hosted Ec2 instance as the media files were being accessed from different cities at the same time. Also, the full high-definition videos had to be downloaded for the preview. They wanted to optimize bandwidth utilization and improve the operator’s response times.

Continue reading “Rearchitecting an Old Solution”

CloudFront with multiple origins

A while ago I had bragged about how this site is published – heavily customized WordPress deployed on S3. Though I had left out some of the indigenous parts which I would like to explain in this article. The main aspect explained would be the convergence of multiple origins with CloudFront and configuring behaviours for different cache settings.

Continue reading “CloudFront with multiple origins”