Node.js, Sys admin

Node.js script to make a zip archive

While working in a node.js project, I had an use case where user will have to query based on days and query result will be pdf filtered by  the date range. And I have to create a directory of pdf files and return it as a zip file for the user.

As there are many modules related to zip, I tried couple of the active modules but none was meeting my requirements. And some of those modules has different bugs/issues which were open at that time. After different try/error and going through most of the zip modules, I used archiver.  And below a sample how it worked.

 

After cloning the gist make sure you have files to zip and run the script like below:

node_archiver

Happy coding :)

About these ads
Standard
Python, python csv, Sys admin, Ubuntu, Unix

Python script to download Google spreadsheet.

I like to automate tasks, I think every software engineer like that, right? After all thats our job. I wrote the following script for downloading google spreadsheet as csv. Just got it when I was going through my old code base, hopefully it would help someone else too.

To run the script you have to install gdata python module.

 

You have to run the script like below:


python gdooc.py spread_sheet_id#gid=tab_id

For example check the following screenshot:

cmd

 

And after downloading you will have the csv file in the same directory, currently the document id is being used as name of the csv file, you can change it as you want.

Happy Coding :)

Standard
লিনাক্স, Linux, Sys admin, Ubuntu, Unix

Attaching new EBS volume in AWS EC2 instance.

Amazon AWS plus EC2 logo_scaled

In AWS, EC2 by default provide 8GB space, in a past project I had to extend the size of one of my development instance as the data was growing fast. From AWS console add new EBS volume. Then attach it to your instance by AWS console and log into you EC2 instance via ssh.

Run following command:


sudo fdisk -l

which will show list of volumes with the newly added volume as unpartitioned. Something like below:

Screen Shot 2012-09-19 at 1_up_2

Then next step is to build the file system of new EBS volume using unix command mkfs. Like below:


sudo mkfs -t ext4 /dev/xvdf

Screen Shot 2012-09-19 at 1_mkfs

Next you have to mount it in your desired path,  e.g. /mnt/ebs1. Run following command:


sudo mount /dev/xvdf /mnt/ebs1

Then add an entry into /etc/fstab. it would be something like this:


"/dev/xvdf  /mnt/ebs1 ext4 defaults 1 1"

There are facts if you add the EBS volume to your /etc/fstab and some how if there are  issue (like file system corruption, unavailability of zone  etc ) with the volume during booting the instance it will not be booted. Because while booting your system will look for the entry and when its not available the whole instance is down. Check AWS forum post for details.

And also check this whole SO discussion to resolve this issue in alternative way ( using a script for example).

Check following docs if you are more interested about the unix commands that used in this post.

fdiskmount and unmount  and mkfs.

Standard
লিনাক্স, Node.js

Serving static files using Restify

I was working with Node.js for building a REST API. For REST API module I was using restify. The restify is a simple and yet powerful node module. One of the use case of the API was, I had to serve static file for specific routing. I went through the docs and tried different things but couldn’t figure yet out at first. After hustling for hours,  me and Christian started to go deep into it and figured it out!

So in my case the configuration was  something like this


server.get(/.*/, restify.serveStatic({
    'directory': 'static_content',
    'default': 'index.html'
 }));

When restify internally resolve the path, it looks for ‘static_content/index.html’.

I have coded a very basic application to show it works, sample application looks like below:

In this sample application the static content (index.html) is in the same root as server.js as why ‘directory’: ‘.’.

Use package.json to install necessary module and start playing :)

Standard
লিনাক্স, Sys admin, Ubuntu, Unix

Counting line number of files recursively in Unix

I was working in a project for last couple of months, as the days are passing the codebase is getting larger. Suddenly I thought, It would be great if I can  know how many lines of code I have written so far for each module. And also in total. I know unix has a really awesome utils named wc.

After googling and trying different params and commands I managed to find it by merging to unix tool(wc and find), the full command for recursive line number counting is like below:

wc -l `find . -type f`

The command returns something like below:

Screen Shot 2013-09-25 at 1.39.13 AM

Using  find . -type f   listing all the files recursively and wc -l is counting the line numbers :)

For learning these tow unix command in details check wc and find manual.

Standard
Python, python basics

Python super and init explained with example

Python super :

Python super keyword is confusing some time to newbie or even for intermediate python programmers.

But the idea behind super is really simple. In OOP paradigm we often need to do implement inheritance like below:

CODE:

class A(object):

    def fancy_func(self):
        print 'Fancy Function Called from Class A'

class B(A):

    def fancy_func(self):
        return super(B, self).fancy_func()

b is an Object of class B and fancy_func is the method of B, super is returning the base classes method. If we don’t use super,we had declare an object of class A and then we had to call fancy_func. On the other hand,super returns proxy object. Super uses __mro__(method resolution order).

Super can be used:

  • For single inheritance using to refer parent classes
  • In multiple inheritance its very useful in during dynamic execution.

For real life coding when we need to enhance any module method we can easily super to get things done.

And we don’t even have to know details about the base class that we are extending from.

super is only applicable for python new style classes ( the classes derived from object ex. class A(object) )

For python3 the syntax is like below:


super().methoName(args)

Syntax of calling super is like below:

super(subClass, instance).method(args)

Python __init__ :

If you declare a __init__ in your python class, it will be run when you initialize an object from that class.

__init__ acts like constructors in other languages but actually its not. There is a basic major difference between from other methods and __init__,its you cant return anything from it.You can add properties to the current object using like self.myProperty = ‘TEST’ and you can use it in any other method by accessing like self.myProperty

Simply __init__ is used when we want to control the initialization of the class.

Lets build something real with these features:

In the above example we implement both the concept of __init__ and super. Here __init__ using for setting value of url while intilazing the object  and super is being used to call the crawlPyCentral’s getTitles.

To dig more deep into super check this blog post

Standard
লিনাক্স, MongoDB, Sys admin, Ubuntu, Unix

MongoDB backup script

Last year while I was working in a project, I needed to automate the whole backing up process from taking snapshot of the current db to saving it to AWS S3 buckets. At that time I took most of the staffs from this blog post.

mongo_logo

Couple of days ago, I started to code for making small backup script that will backup to another cloud machine rather than to AWS S3.  Instead of  coding it from scratch, I reused my previously coded script. All I need to implement a bash function(save_in_cloud) which runs a simple scp command :)

The whole script look like below:

I reused this script, all I did just added a new function which copy the current backup data to a remote server.  And also updated do_cleanup, now it works in any year.

The backup script depends on other two js (fsync_lock.js and fsync_unlock.js) functions which responsible for locking mongo during  db snapshots and releasing lock  after the  snapshots.

Happy Coding :)

Standard