Export Cloudwatch Logs to AWS S3 – Deploy using SAM

With due reference to the blog which helped me in the right direction, the tensult blogs article Exporting of AWS CloudWatch logs to S3 using Automation, though at some points I have deviated from the original author’s suggestion.

Some points are blindly my preference and some other due to the suggested best practices. I do agree that starters, would be better off with setting IAM policies with ‘*’ in resource field. But when you move things into production it is recommended to use least required permissions. Also some critical policies were missing from the assume role policy. Another unnecessary activity was the checking of existence of s3 bucket and attempt to create if not exists, at each repeated execution. Again for this purpose the lambda role needed create bucket permission. All these were over my head, and the out come is this article.

Well if you need cloudwatch logs to be exported to S3 for whatever reason, this could save your time a lot, though this needs to be run in every different region where you need to deploy the stack. Please excuse me as the whole article expects to have aws-cli and sam-cli pre installed.

First lets create the bucket and apply a policy that permits the cloud watch logs service to write into the bucket. Also the bucket owner has full permission on any created object.

aws s3 mb s3://<bucketname> --region <AWS_REGION>
aws s3api put-bucket-policy --bucket <bucketname> --policy file://policy.json

The first line creates the bucket and the second line adds the policy, for the sake of explanation the file is reproduced below.

{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.ap-south-1.amazonaws.com"
      },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::<bucketname>"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.ap-south-1.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::<bucketname>/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

Now we can deploy the template using

sam deploy -g

Then respond with the responses as expected.

To cut the story short, using the template we create two roles, one for the state machine, the second for lambda, since the state machine requires lambda:InvokeFunction permission on the lambda and actual lambda does not need the same. Continuing the Lambda Function, which actually does the whole job, is mostly a branch from the same function provided in the original article but logically those parts which does the bucket existance check and the sort was removed.

The whole template and its accessories can be downloaded here. Download Export Cloudwatch Logs to AWS S3 SAM Template.

Javascript API Credentials – Just a port

There is not much to write than to attribute the logic up to a post on online code generator. Well since this was written in php and I needed the same in javascript, there was some cutting corners, and finally the script which is attached came up.

randomstring

 

The vars defined hold the two array of strings, and the function generates the key. Calling the function with genKey(16, access_salt) will generate a 16 char random string from the defined array access_salt.

Download

Linux CPU usage and montioring using shell memcache and jquery

Recently in a project where the application was deployed across multiple servers, the Client QA as well as Support Team wanted a better monitoring of all the servers in the production. It was too much to provide everybody with shell access and ask them to monitor using top. Well after a lot of digging through the wonderful search index of Google. And with insights from Paul Colby vide his article Calculating CPU Usage from /proc/stat, and various comments of Memcache usage through telnet along with the /dev/tcp socket connections it was just a matter of using some nifty shell processing before I could store each machine cpu values, loadavg, and running tasks as a json encoded string into memcache on one of the hosts with hostname as the key.
Continue reading “Linux CPU usage and montioring using shell memcache and jquery”

Getting datetime type into JavaScript as Date object

When selecting datetime to be displayed in a JavaScript ui library, select the unix_timestamp * 1000 from the sql

This is not any new thing and may be discussed at different other places. But just as I came across like any other things, I just wanted to make a record of this.

When selecting datetime to be displayed in a JavaScript ui library, select the unix_timestamp * 1000 from the sql.
Continue reading “Getting datetime type into JavaScript as Date object”

Prototype based date picker

We had started optimizing drive for the projects of Reserway Technologies, and the work being done on the demo site. For the initial prototype, we used a dual datepicker developed in flex, which interacts with the html using javascript calls. We used this because the component was already available, and just adding a couple of handlers could do our job, and we could concentrate more on the abstraction of GDS integration. Well we are in a state where the backend is almost stable and we need more performance drive on the frontend. The first choice was ofcouse linking the libraries to google cdn. Then started cutting the curves. The biggest bottleneck was the whopping 220Kb for the flash control. I was particular to use a simple datepicker, which could be styled using css, and would be better if used the prototype library.
Continue reading “Prototype based date picker”

protoype.js: Deep Category Select; Ajax

I was experimenting with the prototype.js library, thanks to all who have contributed towards this, and the wonderful documentations avaliable as download, as well as online references.

For a multilevel hierarchical selector of category, where the top levels should not be selectable, the existing ui elements were not enough to show off, with out incurring ambiguity. This led me to do the basic tests, and finalized the said widget. It does not support much now, though I may be working on a extension which will support multi-select.

See the Deep Category Select, in action where it is embedded into an example.

Category Selector by jiju-saturn

How ’bout writing a JavaScript template engine ?

Yea! a minimalistic one, without much xBrowser checks, would be as follows,

function render(o){
	if(!o.o) return;
	var rv = document.createElement(o.o);
	if(o.p){
		var pC = o.p.length - 1;
		for(var i = 0; i < pC; i += 2){
			rv.setAttribute(o.p[i], o.p[i+1]);
		}
	}
	if(o.c){
		var cC = o.c.length;
		for(var i = 0; i < cC; i++)
			rv.appendChild(render(o.c[i]));
			}
	if(o.txt)
		rv.innerHTML = o.txt;
   return rv;
}


Now what the heck is written above ? no wonder, it is just a recursive function which can handle json page structures. Only thing is that the page structuring can go heck and complex, though the renderer is too simple with only just a handful of statements. The var names are intentionally kept at a minimum to make the function lighter.
Continue reading “How ’bout writing a JavaScript template engine ?”

JavaScript Object Oriented

It may be shocking news, but JavaScript is a very powerful object-based (or prototype-based, whatever you wish to call it) language. Yes, JavaScript is a powerful language, not just something that’s handy for image rollovers and other corny, flashy effects. However, very few people who have used JavaScript realize its capabilities. If you’re one of these people, this tutorial is aimed at you.

First of all, JavaScript is not a full-blown OOP (Object-Oriented Programming) language, such as Java, but it is an object-based language. So, why should you use objects? Not only do they help you better understand how JavaScript works, but in large scripts, you can create self-contained JavaScript objects, rather than the procedural code you may be using now. This also allows you to reuse code more often.

Read the full article by Ryan Frishberg

AJAXification by js graceful degradation

Now the question was about a suggestion to have a balance between best designs considering a friendly URL, Ajax, and SEO. It was implied about some contradictory indications. With the discussion passing on to friendly URLs are great if for no reason than log analysis, however, friendly URLs are supposedly better for SEO also (besides all
the other stuff for SEO).

Suppose if you would like to start moving over to AJAX for CMS-related, stuff, such as loading a news article when the user clicks on a headline, and stil make the site really search engine friendly.

Basically, you have a regular tag link that goes to the location you want, but you also have an onclick javascript event for that link that does the ajax stuff and returns false so that the browser doesn’t request the href part of the tag. Ideally, you’ll have an onload event to your page that attaches all these onclick events to your links so that you don’t even have inline javascript. (Rob Marscher at NYPHP).

The added benefit to backloading your ajaxification is that regardless of js enablement, the user could right click your link and open in new tab or window or even bookmark it and it’s still a plain link that will open normally. (Mark Armendariz at NYPHP).

The overall suggestion is to use prototype.js, and some implementation to use asynchronous fetch.

php wrapper for zapatec tree widget

To quote the zapatec site ” The Zapatec DHTML Tree is an easy, attractive, and versatile way to display information. You can use the DHTML tree as a menu, a site map, or a way to display your data. Our DHTML Tree works in many different browsers, but if visitors use older browsers, they will still be able to see and use the underlying HTML code. ”

I did find this very good, and in the sense that it provides a method for the Search Engines to see all the links. Recently in an administrative backend, I needed to provide a method to browse multilevel deep category subcategory, where they add new products, to their existing line, for which I did choose the lite version of zapatec suite.

The generation of nested ul/li tags for the tree was a real pain for me, from the existing database. And I thought about a wrapper which could do the job if provided with an array of data rows. Hence I made the class-tree.php. The included file is having an implementation sample which will need the lite version from zapatec site.

The sample implementation with the help of zapatec could generate a tree as shown from the code attached below.

tree screen shot

from

The Data

To start with the implementation, we create an instance of the zptree after including the class-tree.php, and passing the path of zapatec js files.

create instance

Then initiate the tree data using $catTree->treeData = array(); Then go along selecting the data from mysql using the select statement select category_id as id,category_name,parent_category_id from category order by parent_category_id, inside a while loop, the tree data is added to the member array, while($rd = $db->fetch_array($rs)) $catTree->treeData[] = $rd;

The attached class-tree.php also has an example implementation at the end, which should be removed before using in production.

Please excuse me about the sloppy documentation, since I am no good at explaining things.

class-tree.php