php command line – JijuTM.COM

Automating Church Membership Directory Creation: A Case Study in Workflow Efficiency

Maintaining and publishing a church membership directory is a meticulous process that often requires managing sensitive data and adhering to strict timelines. Traditionally, this would involve significant manual effort, often taking days to complete. In this blog post, I will share how I streamlined this process by automating the workflow using open-source tools. This approach not only reduced the time from several hours to under 13 minutes but also ensured accuracy and repeatability, setting a benchmark for efficiency in handling similar projects. Specifically it should be noted that the complicated sorting they needed for the final output could have taken the same time if done manually in case of last minute changes like addition or removal of a member that too if a head of the family expired and has to be updated before taking final output the whole prayer group sorting can affect. Consider the head of the family name was starting with Z and when removed automatic upgrade of the next member to head of family and the name starts with A the whole prayer group layout has a chance to take drastic change and manual layout would be herculian in this case. But with this implementation of automation, that is another 15 minutes to the maximum just a flag change in the xls and the command line “make directory” will run through the full process.

Workflow Overview

The project involves converting an xls file containing membership data into a print-ready PDF. The data and member photographs are maintained by a volunteer team on Google Sheets and google drive, these are shared via Google Drive. Each family has a unique register number, and members are assigned serial numbers for photo organization. The workflow is orchestrated using GNU Make, with specific tasks divided into stages for better manageability.

Stage 1: Photo Processing

Tools Used:

Bash Shell Scripts for automation
ImageMagick for photo dimension checking and resizing

The photo directory is processed using identify (ImageMagick) to determine the dimensions of each image. This ensures that all photos meet the required quality (300 DPI for print). Images that are too large or too small are adjusted using convert, ensuring consistency across all member profiles.

Stage 2: Importing Data into MySQL

Tools Used:

MySQL for data management
Libre Office Calc to export xls to csv
Bash and PHP Scripts for CSV import

The exported CSV data is imported into a MySQL database. This allows for sorting, filtering, and advanced layout calculations, providing a structured approach to organizing the data.

Stage 3: Data Sorting and Layout Preparation

Tools Used:

MySQL Queries for layout calculations

The data is grouped and sorted based on location and family register numbers. For each member, a layout height and page number are calculated and updated in the database. This ensures a consistent and visually appealing directory design.

Stage 4: PDF Generation

Tools Used:

PHP and FPDF Library

Using PHP and FPDF, the data is read from MySQL, and PDFs are generated for each of the 12 location-based groups. During this stage, indexes are also created to list register numbers and member names alongside their corresponding page numbers.

Stage 5: Final Assembly and Indexing

Tools Used:

GNU Make for orchestration
PDF Merge Tools

The 12 individual PDFs generated in the previous stage are stitched together into a single document. The two indexes (by register number and by member name) are combined and appended to the final PDF. This single document is then ready for print.

Efficiency Achieved

Running the entire workflow on an ASUS A17 with XUbuntu, the process completes in less than 13 minutes. By comparison, a traditional approach using desktop publishing (DTP) software could take 20–30 hours, even with a skilled team working in parallel. The automated workflow eliminates manual errors, ensures uniformity, and significantly improves productivity.

Key Advantages of the Automated Workflow

Time Efficiency: From 20–30 hours to 13 minutes.
Accuracy: Eliminates manual errors through automation.
Scalability: Easily accommodates future data updates or layout changes.
Cost-Effective: Utilizes free and open-source tools.
Repeatability: The process can be executed multiple times with minimal adjustments.

Tools and Technology Stack

Operating System: XUbuntu on ASUS A17
Photo Processing: ImageMagick (identify and convert)
Database Management: MySQL
Scripting and Automation: Bash Shell, GNU Make
PDF Generation: PHP, FPDF Library
File Management: Google Drive for data sharing

Conclusion

This project highlights the power of automation in handling repetitive and labor-intensive tasks. By leveraging open-source tools and orchestrating the workflow with GNU Make, the entire process became not only faster but also more reliable. This method can serve as a template for similar projects, inspiring others to embrace automation for efficiency gains.

Feel free to share your thoughts or ask questions in the comments below. If you’d like to adopt a similar workflow for your organization, I’d be happy to provide guidance!

Creating a Time-lapse effect Video from a Single Photo Using Command Line Tools on Ubuntu

In this tutorial, I’ll walk you through creating a timelapse effect video that transitions from dark to bright, all from a single high-resolution photo. Using a Samsung Galaxy M14 5G, I captured the original image, then manipulated it using Linux command-line tools like ImageMagick, PHP, and ffmpeg. This approach is perfect for academic purposes or for anyone interested in experimenting with video creation from static images. Here’s how you can achieve this effect. And note that this is just an academic exploration and to be used as a professional tool the values and frames should be defined with utmost care.

Basics was to find the perfect image, and crop it to 9:16 since I was targetting facebook reels and the 50 MP images taken on Samsung Galaxy M14 5G are at 4:3 with 8160×6120 and Facebook reels or YouTube shorts follow the format of 9:16 and 1080×1920 or proportionate dimensions. My final source image was 1700×3022 added here for reference. Had to scale it down to keep inside the blog aesthetics.

Step 1: Preparing the Frame Rate and Length
To begin, I decided on a 20-second video with a frame rate of 25 frames per second, resulting in a total of 500 frames. Manually creating the 500 frames was tedious and any professionals would use some kind of automation. Being a devops enthusiast and a linux fanatic since 1998 my choice was shell scripting. But addiction to php as an aftermath of usage since 2002 kicked up inside me and the following code nippet was the outcome.

Step 2: Generating Brightness and Contrast Values Using PHP
The next step was to create an array of brightness and contrast values to give the impression of a gradually brightening scene. Using PHP, I mapped each frame to an optimal brightness-contrast value. Here’s the PHP snippet I used:

<?php


$dur = 20;
$fps = 25;
$frames = $dur * $fps;
$plen = strlen(''.$frames) + 1;
$val = -50;
$incr = (60 / $frames);

for($i = 0; $i < $frames; $i++){
   $pfx =  str_pad($i, $plen, '0', STR_PAD_LEFT);

    echo $pfx, " ",round($val,2),"\n";

    $val += $incr;
}

?>

Being in ubuntu the above code saved as gen.php and after updating the values for duration and framerate this was executed from the cli and output redirected to a text file values.txt with the following command.

php -q gen.php > values.txt

Now to make things easy, the source file was copied as src.jpg into a temporary folder and a sub-folder ‘anim’ was created to hold the frames. Here I already had a script which will resume from where left off depending on the situation. the script is as follows.

#!/bin/bash


gdone=$(find ./anim/ -type f | grep -c '.jpg')
tcount=$(grep -c "^0" values.txt)
todo=$(( $tcount - $gdone))

echo "done $gdone of ${tcount}, to do $todo more "

tail -$todo values.txt | while read fnp val 
do 
    echo $fnp
    convert src.jpg -brightness-contrast ${val} anim/img_${fnp}.jpg
done

The process is quite simple, first code line defines a var gdone by counting ‘.jpg’ files in the ‘anim’ sub-directory and then taking total count from values.txt the difference is to be done the status is echoed to output and a loop is initiated with reading the last todo lines from values.txt and executing the conversion using the convert utility of imagemagick. In case this needs to be interrupted, I just close the terminal window from xwindows, as a subsequent execution will continue from where leftoff. Once this is completed, the frames are stitched together using ffmpeg using the following commad.

ffmpeg -i anim/img_%04d.jpg -an -y ../output.mp4

The filename pattern %04d is decided from the width of number of frames plus 1 as in the php code the var $plen on code line 4 is taken for the str_pad function input padd length.

The properties of final output generated by ffmpeg is as follows. Note the dimensions, duration and frame rate do comply as decided on startup.

Importance of event logging in server side scripting

Now a days script driven web applications are getting more and more complicated with background operations and triggered events. Debugging or event tracking is tough once the application is moved into production. Fresh and aspiring programmers are always too cautious to wade into deeper waters, and always go with line by line testing. Almost always in the course of debugging or code optimizations I see a lot of them using file_put_contents or echo to check variables at a particular point of execution.

I always gave the pressure to use a good logging system from the start itself, and to add level based message logging with debug_backtrace wherever needed. The most recent class abstraction for php programmers which is being used in our custom framework is attached to the downloads here. The file logging is being done after serializing, compressing and base64_encoding to keep logs in single lines, and to make sure they dont take up too much of the space.
Continue reading “Importance of event logging in server side scripting”

Using TagTheNet to generate tags on WordPress

Recently on Kerala News by Asianet, though the wp-simple-tags was there, the posts were not being tagged automatically. And on a detailed check I found it was due to a misconfiguration, and once the same was done properly the tagging started smoothly. But already a set of 8K posts were there with no or unrelated tags mistakes and un awareness of the operators who were posting to the site. Now I wanted these to be tagged properly.
I had already used the word-twit plugin and done some mods to the same, so it was more eaiser for me. Just went through the original code of word-twit and salvaged a small script which is attached here with as download.
require_once( dirname(__FILE__) . '/wp-load.php' );
The code above loads the wordpress system and initializes the wp variables and connects to the database.
global $wpdb;
Makesure we have the database abstraction object from wordpress, to select the posts, and do all the required manipulations on the same.
require_once( ABSPATH . 'wp-includes/class-snoopy.php' );
Include the inbuilt snoopy class, to mimic a web browser with the most simplicity.
Continue reading “Using TagTheNet to generate tags on WordPress”

User input in php command line

Ever wondered how to capture the user input when writing php command line scripts ?

<?php function getInput($msg){ fwrite(STDOUT, "$msg: "); $varin = trim(fgets(STDIN)); return $varin; } ?>

The function above is being used by me in certain command line scripts, where I need user responses.