Painless deployment with Fabric
Deployment of code in test/staging/production servers is one of the important part of modern web applications development cycle.
Deploying code were painful because its repetitive same tasks we have to do every time we want to push code, during deployment if something goes wrong the application will go down too. But the scenario has changed, now we have many tools to make the deployment easier and fun. I have used Capistrano and Fabric for deployment. Found Fabric really painless and as its a Python battery, it was easier for me to adopt and get things done.
I am going to cover fundamental operations and finally a simple fabric script(like boilerplate) for writing your own fabric script.
env = its a Python dictionary like subclass where we define specific settings like password,user etc
local = runs command in local host(where fabric script is being run)
run = runs command in a remote host
You can use these code tasks in many different ways, to do that check the Fabric Office Documentation from here.
First gist is a sample fabric script,second one is a bash script to install fabric in your ubuntu machine.
After setting username,password and host information into the script you cab check your server’s access log by running fab test_server latest_access_log
I am using fabric for around two years and used for different small,medium and large projects.
There are many interesting open source projects going on top of Fabric. I found these two projects really promising.
1.Fabtools
Search through github,you will find many advance level Fabric use.
Happy Coding!
Pythonic way to calculate Standard Deviation
If you are familiar with basic statistics, I think you know what is Standard Deviation, if you dont know what is Standard Deviation you can check wiki for details.
And if it seems yet hard to wrap the idea into your brain,check this thread. Hope you understand it now. Standard deviation is an efficient when you want to understand a set of data and widely used in different industries. I was working with an algorithm couple of months ago where I had to calculated standard deviation of a series of data. And the sets of data is large.
After coding couple of versions I code a small python class which calculates standard deviation of data. Check it out
Happy Coding!
Summary of 2012
As its weblog for myself,like every other blogger in the world I share my yearly summary for last couple of years.For 2012 it was …
- I switched my job,started to work in Tasawr Interactive since last September.
- Worked with NewsCred API, it was an amazing experience. Coz NewsCred API is a scaled solution and have large code base.I got real taste of working in a complex Python project while I worked with the API team.
- I had to brush up my Drupal knowledge and learned Drupal 7
- Started to work on my pet project which is completely based on Python/Django
- Developed a small financial API for one of my client(I used tastypie for the first time and loved it)
- Worked for Indexica as a remote developer, where my role was DevOps. I had to develop a portion of their API and also managed cloud infrastructure.
- Worked with SOLR and Nutch for the first time and loved both of this.
Daily Unix Commands
If you are a dev or a sys admin you must rely on unix command to get things done.
These are my daily unix tools list,I am putting these together for myself as a future reference,if it helps anyone else I will be happy 
Screen
I use ‘screen’ for keeping myself alive in my remote cloud machines. If you are new to screen,please check this docs.
Checking list of screen in current machine 'screen -ls'
Resuming 'screen -r screen_name' or 'screen -r pid'
Create new screen 'screen -S screen_name'
Detach from screen using 'ctrl+a+d'
Copy mode start 'ctrl+a+{'
Kill the current window 'ctrl+k'
Screen cheatscheet1 screen cheat sheet2
File Compression
Using zip and unzip
Zip command example is like below:
zip backup.zip filename1 filename2 zip -r backup.zip dir_name
Unzip command example is like below:
unzip backup.zip
tar,gzip and bzip2
tar cfv backup.tar filename1 filename2 tar xvf backup.tar
for specific directory
tar xvf backup.tar -C /dir_name/
to use gzip compression use -z and for bzip2 compressing use -j like this
for compression:
tar -zcfv backup.tar.gzip filename1 tar -jcfv backup.tbz2 filename
for uncompression
tar -zxvf backup.ta tar -jxvf backup.tbz2
Unix file compression cheat sheet
And if you want to learn details about this compression techniques please check this
Vim basics
I use vim for editing files in unix system(specially in remote machines).These are list of commands I use:
Basic Navigation:
j->down k->up l->left r->right H->first line of the file G->last line of the file $->end of the line
Changing Line:
yy->copy the current line p->paste the copied line
check Vim Cheat sheet .
I rarely use ‘sed’ command,and I only used it for easy find replace of strings(its really handy when you have to work on multiple files for same find replace ).The syntax is like below:
sed s/search_string/replace+string old_file.txt > new_file.txt
check sed refernce
…
Rank SQL query Result
I was playing with mysql client,suddenly found that it would be really great if I can rank the query results.Lets say we have a table like below:
Now we want to rank the students based on their Roll No.To do it,we have to define a variable with the initial value of 0 and in next query we will show it by as one of the resulted field query
like below:
set @counter=0; select @counter:=@counter+1 as Rank,LastName,Roll_no as Roll from Students order by Roll_no ASC;
and in next query we will show it by as one of the resulted field ,that results:
Check mysql docs from here to know details about user defined variable in mysql
Build Periodic crawler with Celery
Its a very common use case when you build a crawler and it will have to run periodically.And generally we set a unix Cron Job to handle the crawler periodically.
But its really pain when you add new task you have to login to the server and add new cron task into the crontab.Its only feasible when you have to run only few cron jobs.
I thought it would be great if I can handle it from my python code and do some interesting things.I have heard about Celery a lot as a meesage queue system. Truly speaking at first I couldn’t understand how it works or how I can integrate with my projects.After googling I understood what is Celery then I thought it will be really great if I can run crawlers around it and use Celery to scheduling periodic work.
Install Celery by following this link,then you will have to install and configure RabbitMQ from this link. BTW dont forget to add user,vhosts it is described on SettingUp RabbitMQ sections(ou can use mongodb,resddis as broker too).
And then you can clone my git repo,change the celeryconfig.py file as per as your configuration.Add a new task into tasks.py following the first method.
I have added a sample method which requests this site and print the HTTP response status code .
To run the project run “celerybeat”,then it will start celerybeat and start to send tasks to the broker like below:
Run “celeryd” into another terminal window to check the task output,you will see something like below:
It is printing the response status after every 5 seconds.
You can handle anything that you want todo after the crawling,like parsing the dom saving text,submitting form etc.
Btw dont forget to run
pip install -r requirements.txt
to install necessary packages for the projects.My Github Periodic Crawling Project
Comment notification plugin in wordpress
May be it was end of the last year, I was assigned to finish a wordpress project which was taking way too long time to deliver. I was away from wordpress development for more than two years. So after jumping into the project I found most of the user requirements are bit different than the general.
One of the requirements was to wordpress admin should be able to use “admin comment” section to reply the comments as email whether the user is subscribed to the post/comment or not. If some one comment into the content and if admin approved it with reply, the user will have to receive an email with the reply.
Worpdress provide a nice action call “comment_post“, I wrote a simple method which will be executed after every comment and check is it approved or not and will send an email. Check the full plugin from github
By the way, without subscriptions into the post and comment, sending email is not “ethical” so if you want to use it use in your own risk.
Also feel free to fork it if you want to add/customize
Share this:
Like this: