Get Parent Directory via Python

Get the parent directory of a folder/file in python. Using the "os" library its possible using python.

Lets try to understand with an example

>>> import os
>>> os.path.dirname('D:\Workspace\8.0.0.0 - INT')

Output will be as: 'D:\\Workspace'

Same if want to get the parent directory of file,

>>> os.path.dirname('D:\Workspace\8.0.0.0 - INT\AdapterFactory.py')
Out put will be as: 'D:\\Workspace\\8.0.0.0 - INT'

Thanks.

Read More

Merge Two CSV Into One CSV

Hello my so lovely friends,

So how are your days going on? Hope you guys doing some experimenting. NOOO.. not an issue.

During my office hours I met with a problem to merge two CSV's into one on the basis of the common header, no need to include those records with mis match headers with another CSV. After experimenting finally able to write a script in python. So here is the code for the same. Hope if any one is facing or will face same issue can use this script.

Link for the script: https://gist.github.com/anupamshakya7/60607c86a7dbe9853843

So here is the script. Enjoy my friends and have a good day :)

import csv
import os

if __name__ == '__main__':

    """This script will merge two CSV with common data as per the ID as a field
    into new CSV.

    ::NOTE:: This program tested on 2.7.6 version of python.

    For an example Test1.csv have this data:
    ----------------------------------------
    ID,name,add,school
    1,anupam,FHG,sch1
    2,aditya,DFR,sch2
    3,urmila,HJTY,sch3
    4,roop,GHG
    5,EASH,HJJ

    Test2.csv have this data:
    ------------------------
    ID,Key,Value
    1,x-jdkj,100
    2,k-djsh,200
    3,j-jdjd,300

    Resultant CSV have this data:
    -----------------------------
    add,school,ID,name,Value,Key
    FHG,sch1,1,anupam,100,x-jdkj
    HJTY,sch3,3,urmila,300,j-jdjd
    DFR,sch2,2,aditya,200,k-djsh


    How to run
    ----------
    On command promt go to the location where you placed your script(python script)
    And issue this command 'python CSV.py'

    After that it will ask for the absolute path where you have placed your two CSV with ID
    as a common field(in header part).

    ::NOTE:: Please provide absolute path for folder like - C:\Users\hv3775\Desktop\TEST

    That's. It will generate new CSV into the same folder that you have provided as a input path.
    """

    input_folder_location = raw_input('Enter location where your required two csv files placed. NOTE:: Please enter absolute path: ')
    infiles = []
   
    for csv_file in os.listdir(input_folder_location):
        if csv_file.endswith('.csv'):
            infiles.append(csv_file)
   
    data = {}
    fields = []

    for fname in infiles:
        with open(fname, 'rb') as df:
            reader = csv.DictReader(df)
            for line in reader:
                # assuming the field is called ID
                if line['ID'] not in data:
                    data[line['ID']] = line
                else:
                    for k,v in line.iteritems():
                        if k not in data[line['ID']]:
                            data[line['ID']][k] = v
                for k in line.iterkeys():
                    if k not in fields:
                        fields.append(k)
            del reader

    data_list = []
   
    for d in data.items():
        if len(d[1].values()) != len(fields):
            continue
        data_list.append(d[1])  
   
    csv_output_location = os.path.join(input_folder_location, 'Result.csv')
    with open(csv_output_location, 'wb') as f:
        w = csv.DictWriter(f, fields)
        w.writeheader()
        w.writerows(data_list)

Read More

Get list of installed libraries in a system using python

From lovable python terminal its possible to get list of all installed python libraries. Here is the piece of cake for you that I already tested. Try from yourself too :)

Open your python terminal doesn't matter whatever OS you are using(As of now I' using windows. Type following command

from pkg_resources import WorkingSet , DistributionNotFound
working_set = WorkingSet()

# Printing all installed modules
print tuple(working_set)

# Detecting if module is installed
try:
    dep = working_set.require('paramiko>=1.0')
except DistributionNotFound:
    pass

Output of above print will be as:
(astroid 1.0.1 (c:\python27\lib\site-packages), BeautifulSoup 3.2.1 (c:\python27\lib\site-packages), blinker 1.3 (c:\python27\lib\site-packages),....

Read More

Transform an XML file using XSLT in Python

Howdy,

Form last two months was so busy, so not able to write any new post. But in last two months while doing my project encountered with an issue. The issue was to generate CSV files from the given XML & XSLT file using python. But by doing some research able to find solution using the most popular library LXML. Lets try to explore how its is possible.

Go to your python terminal, and import lxml library
>>> from lxml import etree

1) Open xslt file and read its content.
>>> data = open('D:\Conversion\MACSXML_Parts.xslt')
>>> xslt_content = data.read()

2) Now parse xslt content to etree.XML method. "etree.XML" returns root node (or the result returned by a parser target).
>>> xslt_root = etree.XML(xslt_content)

3) Parse required XML file to etree.parse method, will return ElementTree object like this "<lxml.etree._ElementTree object at 0x01AB1F08>"
>>> dom = etree.parse('D:\Conversion\Cat2015UF.xml')

4) Parse xslt_root node to etree.XSLT(created above). "etree.XSLT" turn an XSL document into an XSLT object.
>>> transform = etree.XSLT(xslt_root)

5) Finally parse dom that has been created above to the new created transformed XSLT object.
>>> result = transform(dom)

Your result contained the appropriate data. You can store this into CSV file like:
>>> f = open('D:\Conversion\MACSXML_Parts.csv', 'w')
>>> f.write(str(result))
>>> f.close()

Full example is as:

 from lxml import etree

 data = open('D:\Conversion\MACSXML_Parts.xslt')
 xslt_content = data.read()
 xslt_root = etree.XML(xslt_content)
 dom = etree.parse('D:\Conversion\Cat2015UF.xml')
 transform = etree.XSLT(xslt_root)
 result = transform(dom)
 f = open('D:\Conversion\MACSXML_Parts.csv', 'w')
 f.write(str(result1))
 f.close()

You can view the same code on gist too. Link is as https://gist.github.com/roopsinghadi/11285898
Hope it is helpful. If it's don't forget to share :P

Read More

Get the Content-Type of an URL

Hey everybody,

Have you ever tried to get the content type of any website via python, If not try like this. I tried on python terminal.

OR

>>> import urllib
>>> res = urllib.urlopen("https://www.wslb2b.ford.com/login.cgi")
>>> http_message = res.info()
>>> http_message.type
'text/html'
>>> http_message.maintype
'text'

Thanku :)

Read More

Python Regex Expression(Regex!!)

Introduction
Matching text patterns

Support
Python module supports re module for regex support

Visit Of Regex

Search expression
Python regular expression can be written as
match = re.search(pattern, string)

Search pattern takes following parameters
a) pattern: A regular expression pattern
b) string: A string in which we wanna find regex pattern

Return by this method:
a) It will return a match object is search is successful
b) Else, None

Lets consider an example,  
>>> import re                          # Import re module
>>> pattern = r'word'                  # Define regex pattern
>>> string = 'Test word'               # Define string on which implementation of search we want
>>> match = re.search(pattern, string)  # Syntax of search in regex(regular expression)
>>> match                               # Return an object after searching
<_sre.SRE_Match object at 0x018A1B10>  
>>> if match:
...     print 'Matched, ', match.group() # If matched, print group of words
... else:
...     print 'Opss not found..'
...
>>> Matched,  word                      # Return matched words :)

Have you noted 'r' while defining the patterns. This is actually called raw string('r') which passes through backslashes without change. r prefix (e.g. r"\d+") is actually used for raw strings in Python. In a raw string you don't have to escape backslashes with \\ (e.g. "\\s+" equals r"\s+"). On python terminal
>>> "\\s+"
'\\s+'
>>> r'\s+'
'\\s+'
So while it's a very good convention to prefix all regular expressions with r, it's not technically necessary when an expression doesn't have any backslashes (e.g. "[0-9a-z]").

Escape sequences are:
     
Basic Patterns
Following are the basic patterns in regex:

• a, X, 9, < - Ordinary characters just match them selves. Meta characters dont match them selves.  11 metacharacters that must always be preceded by a backslash, \, to be used inside of the expression  
• . (a period) - matches any single character except newline '\\n' 
• \\w (lowercase w) - matches a "word" character: a letter or digit or underbar [a-zA-Z0-9_]
• \\b  -  boundary between word and non-word
• \\s (lowercase s) - matches a single whitespace character. space, newline, return, tab, form ['\\n \\r\\t' ]
• \\t, \\n, \\r - tab, newline, return(respectively) 
• \\d -  boundary between word and non-word
• ^ = start, $ = end  -  match the start or end of the string

Some basic ExampleSome basic examples on python terminal:

>>> match = re.search(r'.', 'Test word')
>>> match.group()
>>> 'T'
>>> match = re.search(r'..', 'Test word')
>>> match.group()
>>>'Te'
>>> match = re.search(r'...', 'Test word')
>>> match.group()
>>>'Tes'
>>> match = re.search(r'.w', 'Test word')
>>> match.group()
' w'
>>> match = re.search(r't.', 'Test word')
>>> match.group()
't '
>>> match = re.search(r'.w.', 'Test word')
>>> match.group()
' wo'
>>> match = re.search(r'wor..', 'Test word')
>>> match.group()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'group'
>>> match = re.search(r'wor.', 'Test word')
>>> match.group()
'word'
>>> match = re.search(r'\d', '1001 Test word')
>>> match.group()
'1'
>>> match = re.search(r'\d\d', '1001 Test word')
>>> match.group()
'10'
>>> match = re.search(r'\d\d\d', '1001 Test word')
>>> match.group()
'100'
>>> match = re.search(r'\w\w', '1001 Test word')
>>> match.group()
'10'
>>> match = re.search(r'\d\d\d', '1001 Test word @!#')
>>> match = re.search(r'\w\w', '1001 Test word @!#')
>>> match.group()
'10'
>>> match = re.search(r'\w\w\w', '@@abcd!!')
>>> match.group()
'abc'
>>> match = re.search(r'\w\w', '@#1$ 1001 Test word')
>>> match.group()
'10'

Repetition

• (+) - One(1) or more occurances of pattern from left
• (*) - Zero(0) or more occurances of the pattern from left
• (?) Zero(0) or One(1) of the pattern from left

So basically it means that (+, *), match string from left most and go far as possible.

Examples

>>> match = re.search(r'Te+', 'Teest Word')
>>> match.group()
'Tee'
>>> match = re.search(r'Te+', 'Teeeast Word')
>>> match.group()
'Teee'
>>> match.group()
'Te'
>>> match = re.search(r'\d\s*\d\s*\d', 'sfad1 2   3asdxx')
>>> match.group()
'1 2   3'
>>> match.group()
'1 2   3'
>>> match = re.search(r'\d\s*\d\s*\d\s*', '1 2   3    ')
>>> match.group()
>>> match = re.search(r'\d\s*\d\s*\d', '    1234         ')
>>> match.group()
'123'
>>> match = re.search(r'\d\s*\d\s*\d\s*', '1 2   3    ')
>>> match.group()
'1 2   3    '
>>> match = re.search(r'\d\s*\d\s*\d\s*', '     123             ')
>>> match.group()
'123             '
>>> match = re.search(r'\s*\d\s*\d\s*\d', '    1234         ')
>>> match.group()
'    123'

Find E-mails in an example

>>> match = re.search(r'\w+@\w+', 'This is the search for abc@domian example')
>>> match.group()
'abc@domian'

Usage of square brackets

• Used to indicate set of characters, i.e. [abc] matches a or b or c
• The codes \\w, \\s etc. work inside square brackets too with the one exception that dot (.) just means a literal dot. For the emails problem, the square brackets are an easy way to add '.' and '-' to the set of chars which can appear around the @ with the pattern r'[\\w.-]+@[\\w.-]+' to get the whole email address

>>> match = re.search(r'[\w.]+@[\w.]', 'This is the search for abc@domian example')
>>> match.group()
'abc@d'
>>> match = re.search(r'[\w.]+@[\w.]+', 'This is the search for abc@domian example')
>>> match.group()
'abc@domian'
>>> match = re.search(r'[\w.]+@[\w.]+', 'This is the search for abc@domian.com example')
>>> match.group()
'abc@domian.com'
>>> match = re.search(r'[\w.-]+@[\w.]+', 'This is the search for abc-def@domian.com example')
>>> match.group()
'abc-def@domian.com'

Usage Of Parenthesis()

• Allow to pick part of matching text.
• Do not change the pattern  text.
• Establish groups of matched text. like match.group(1), match.group(2) etc
• Plain match.group() match whole text as usual

For an example,

>>> match = re.search('([\w.-]+)@([\w.-]+)', 'This is an example of group matching abc-def@gmail.com')
>>> match.group()    # Whole plain match
'abc-def@gmail.com'
>>> match.group(1)   # First match of group
'abc-def'
>>> match.group(2)   # Second match of group
'gmail.com'

findall Function
Match all pattern in a text.

• re.search() - Find first match  for a pattern.
• findall() - Match all string and return list of matched content.

>>> match = re.findall('[\w.-]+@[\w.-]+', 'An exmapple of findall abc-def@gmail.com, xyz-uvw@yahoo.com')
>>> match
['abc-def@gmail.com', 'xyz-uvw@yahoo.com']

findall & Groups

• () can be combined with findall.
• If pattern have two or more (), it will return list of tuples. Each tuple represent matched pattern.

Lets take example to view this scenarion
>>> match = re.findall('([\w.-]+)@([\w.-]+)', 'An exmapple of findall abc-def@gmail.com, xyz-uvw@yahoo.com')
>>> match                  # Return list of tuples of matched content with separate groups in a tuple
[('abc-def', 'gmail.com'), ('xyz-uvw', 'yahoo.com')]
>>> match[0]        
('abc-def', 'gmail.com')
>>> match[0][0]
'abc-def'
>>> match[0][1]
'gmail.com'
>>> match[1][0]
'xyz-uvw'
>>> match[1][1]
'yahoo.com'

Thanks

Read More

Gotchas about __str__ and __repr__

Many of python developers confused with the gotcha of __str__ and __repr__ in-built function of python.

So lets's try to clear this.

For new beginners

• __str__ can be treated as informal string representation of an object.
• __repr__ can be taken as formal representation of an object

Confused!! :P

No need to worry, lets take help of lovable python terminal. 

Consider an example of number

As from the above result str() and repr() are same.

Now consider an example of string

From above results its clear that while evaluating result of str & repr are different and not valid in case of str representation of a string

Hope now its clear.

Read More

Absolute Path Vs Relative Path

Hey,

So my friends whats happening? From  last few days was facing an issue to understand the concept of Absolute Path and Relative Path.

So as per the R&D provinding a simple solution for the same.

Absolute Path, basically the location/address of file from root(/) to the specified location. Or in other words, full path of a file/directory from the root(/).
For an example,
/var/www/operx
/home/user/anupam/Books/PythonBooks/

Relative Path, is the location of file related to the present working directory(pwd). Suppose I am inside /home/user/anupam/Books and want to change directory to /home/user/anupam/Books/PythonBooks.
I can use relative path concept to change directory to PythonBooks by,
/home/user/anupam/Books
cd PythonBooks

Note: If you observe there is no / before PythonBooks which indicates it's a relative directory to present working directory.

Is it possible to change /home/user/anupam/Books/PythonBooks/ using absolute path concept via
/home/user/anupam/Books/PythonBooks/

Hope this is useful.

Thanks.

Read More

ElasticSearch Installation

Installation Of Elastic Search

Elastic search is real time distributed search engine. Its very fabulous to use elastic search for projects instead of creating own search strategies. 

So here are the steps to install elastic search:

• Download debian package of elastic search from official site.
• Install JDk by - sudo apt-get install openjdk-6-jdk
• Install elsatic search - sudo dpkg -i elasticsearch-0.90.2.deb (path of debian package of elasticsearch) 

How you can test whether elastic search is installed in your system. It is possible to test by putting this URL on browser http://localhost:9200/ , by default elastic search working on the port of 9200 and return following json on browser.

{

• ok: true,
• status: 200,
• name: "Lodestone",
• version: {
  

  • number: "0.90.2",
  • snapshot_build: false,
  • lucene_version: "4.3.1"

  },
• tagline: "You Know, for Search"

}

Number of options are avail for elastic search. Run following command on terminal to see options.
a) anupam@anupampc:~$ sudo service elasticsearch help
Output 
 * Usage: /etc/init.d/elasticsearch {start|stop|restart|force-reload|status}

b) anupam@anupampc:~$ sudo service elasticsearch start
Output
* Starting ElasticSearch Server                                        [ OK ]

c) anupam@anupampc:~$ sudo service elasticsearch status
Output
* ElasticSearch Server is running with pid 4996
 
d) anupam@anupampc:~$ sudo service elasticsearch restart
Output
* Stopping ElasticSearch Server                                        [ OK ]

Plugin avail for using elastic search in chrome

SENSE plug-in avail for google chrome. Add sense to your favorite google chrome and ready to use elasticsearch. Once you have installed it you'll find Sense's icon in the upper right corner in Chrome. The first time you click it and run Sense a very simple sample request is prepared for you. It is possible to run queries directly on sense and see results. 

Sense Interface On Google Chrome

Sense provides a simple user interface specifically for using ElasticSearch's REST API. It also has a number of convenient features such as autocomplete for ElasticSearch's query syntax and copying and pasting requests in curl format, making it easy to run examples from the documentation.

Read More

Implementation of PUT and DELETE in pycurl

Implementation of PUT and DELETE in pycurl

Through this post explaining about the usage of PUT and DELETE in pycurl.

import pycurl
import json

"""

Here is the sample example of curl which is using PUT and DELETE

curl -XPUT localhost:9200/tweets/tweet/1 -d '
{
user: "Test User",
message: "Test Message",
postDate: "20130201T02:00:00",
"mappings" : {
    "tweet" : {
        "_source" : {"enabled" : false}
    }
}
}
'

curl -XDELETE 'http://localhost:9200/twitter/tweet/1'

"""
page_index = {
user: "Test User",
message: "Test Message",
postDate: "20130201T02:00:00",
"mappings" : {
    "tweet" : {
        "_source" : {"enabled" : false}
    }
}
}

c = pycurl.Curl()

# Remove Index if alreday exists

c.setopt(pycurl.URL, "http://localhost:9200/twitter")
c.setopt(pycurl.CUSTOMREQUEST, "DELETE")

c.perform()
 

# Create new index

c.setopt(pycurl.URL, "http://localhost:9200/twitter")

c.setopt(pycurl.CUSTOMREQUEST, "PUT")
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.POSTFIELDS, '%s' % json.dumps(page_index ))

c.perform()

Read More

Create readonly permission for all models - Django

Create read only permission for all models - Django

Is it possible to create readonly /viewonly permission for all models in django. If admin provide read only permission to a particular user, an user can't edit record and can view only. So here is a very simple code for the same. Created an app and have the following features:

• If user have a permission to view only, user cant edit record or in others word can only view.
• Submit row will disappear, as an user dont have any benefits for - Save, Save & Continue, Save & Add buttons.
  

Follow these steps.
a. Create an app with the name of 'admin_hack' by this command -

"python manage.py startapp admin_hack" This will create following files.

admin_hack/
    __init__.py
    admin.py
    models.py
    tests.tpy
    views.py

b. Rename admin.py >> admin_hack.py

c. Write following code to admin_hack.py file

from django.contrib import admin
from django.db.models.signals import post_syncdb
from django.contrib.auth.models import Permission
from django.contrib.contenttypes.models import ContentType
from django.contrib.admin.util import flatten_fieldsets
from django.contrib.admin.templatetags.admin_modify import *
from django.contrib.admin.templatetags.admin_modify import submit_row as original_submit_row

def add_cannot_edit_record_permission(sender, **kwargs):
    """
    This syncdb hooks takes care of adding a view permission to all our
    content types.
    """
    for content_type in ContentType.objects.all():
        codename = "cannot_edit_record_for_%s" % content_type.model

        if not Permission.objects.filter(content_type=content_type, codename=codename):
            Permission.objects.create(
                content_type=content_type,
                codename=codename,
                name="Cannot edit record for %s" % content_type.name
            ) 

post_syncdb.connect(add_cannot_edit_record_permission)

class HackAdminModel(admin.ModelAdmin):
    def get_readonly_fields(self, request, obj=None):
        """Get readonly fields.

        :param request: HTTP request object.
        :param obj: An object.
        :return: Return readonly fields.
        """
        class_name = self.__class__.__name__.replace('Admin', '').lower()

        for permission in request.user.get_all_permissions():
            head, sep, tail = permission.partition('.')
            perm = "cannot_edit_record_for_%s" % (class_name)
            if str(perm) == str(tail):
                if request.user.has_perm(str(permission)) and not request.user.is_superuser:
                    if self.declared_fieldsets:
                        return flatten_fieldsets(self.declared_fieldsets)
                    else:
                        return list(set(
                            [field.name for field in self.opts.local_fields] +
                            [field.name for field in self.opts.local_many_to_many]
                        ))
        return self.readonly_fields

    @register.inclusion_tag('admin/submit_line.html', takes_context=True)
    def submit_row(context):
        """Sumbit row.

        :param context: Dictionary of required data.
        :return: Return update context.
        """
        ctx = original_submit_row(context)
        app_name, seprator, model_name = str(context.dicts[0]['opts']).partition('.')

        for permission in context['request'].user.get_all_permissions():
            head, sep, tail = permission.partition('.')
            perm = "cannot_edit_record_for_%s" % (model_name)
            if str(perm) == str(tail):
                if context['request'].user.has_perm(str(permission)) and \
                        not context['request'].user.is_superuser:
                    ctx.update({
                        'show_save_and_add_another': False,
                        'show_save_and_continue': False,
                        'show_save': False,
                    })
                return ctx
        return ctx
post_syncdb.connect(add_cannot_edit_record_permission) 

d. Add following code to models.py file

from admin_hack import *

e. Now go to your settings.py file and add this app to "INSTALLED_APPS"  like this

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.admin',
    'django_tables2',
    'admin_hack',

    'pages',

 )

NOTE: Add this app i.e 'admin_hack' at the top of other created apps.

Now your app is ready to use.

Now question raise, how to use this app for all models. Using an example, going to use this app i.e admin_hack.

Here in the installed apps there is an app after admin_hack i.e 'pages'. I am gonna use this app for pages

Follow these steps:

a.  Go to the admin.py file of this app and import 'admin_hack' like this

# Hack Model
from admin_hack.admin_hack import HackAdminModel

b. Pass this 'HackAdminModel' to admin class like this

class PagesAdmin(HackAdminModel)

Before passing HackAdminModel the class PagesAdmin was like 

class PagesAdmin(admin.ModelAdmin) 

Now run following command to create permissions - "python manage.py syncdb" 

Testing of Admin Hack App

Now login to the admin site and view permissions. Permissions created for all models with the 

name as"Cannot edit record for '*' (* - signifies model name)

NOTE: This permission will work only for those models if you  pass 'HackAdminModel' 

to class ModelAdmin(admin.ModelAdmin) like this class ModelAdmin(HackAdminModel)

Code is on git hub and can download, here is the link: https://github.com/anupamshakya7/django-admin-hack

Read More