Python and the interpreted/compiled misconceptions

May 9, 2013 – 8:31 pm

Python is a language, not an implementation. Python can be run as a scripting language internally to let your users script your application. Python can also be the main programming language that you use to write your entire application. The implementation of the language being the target platform for the language and not the language itself. Python is not directly compiled into a machine language, it is compiled into a byte code language that is either interpreted or further compiled into a machine language before it is run. Cython is compiled into C and then further compiled into a machine language before ever being run. Pypy compiles hot spots into machine language for optimized sections of code. Cpython interprets compiled byte code similar to CLR or JVM without the hot spot to machine language jit compilation that speeds up pypy, java, c# etc..

Here is a good quote from stack overflow on the topic http://stackoverflow.com/questions/2998215/if-python-is-interpreted-what-are-pyc-files

There is no such thing as an interpreted language. Whether an interpreter or a compiler is used is purely a trait of the implementation and has absolutely nothing whatsoever to do with the language.

Every language can be implemented by either an interpreter or a compiler. The vast majority of languages have at least one implementation of each type. (For example, there are interpreters for C and C++ and there are compilers for JavaScript, PHP, Perl, Python and Ruby.) Besides, the majority of modern language implementations actually combine both an interpreter and a compiler (or even multiple compilers).

A language is just a set of abstract mathematical rules. An interpreter is one of several concrete implementation strategies for a language. Those two live on completely different abstraction levels. If English were a typed language, the term “interpreted language” would be a type error. The statement “Python is an interpreted language” is not just false (because being false would imply that the statement even makes sense, even if it is wrong), it just plain doesn’t make sense, because a language can neverbe interpreted.

In particular, if you look at the currently existing Python implementations, these are the implementation strategies they are using:

IronPython: compiles to DLR trees which the DLR then compiles to CIL bytecode. What happens to the CIL bytecode depends upon which CLI VES you are running on, but Microsoft .NET, GNU Portable.NET and Novell Mono will eventually compile it to native machine code.

Jython: interprets Python sourcecode until it identifies the hot code paths, which it then compiles to JVML bytecode. What happens to the JVML bytecode depends upon which JVM you are running on. Maxine will directly compile it to un-optimized native code until it identifies the hot code paths, which it then recompiles to optimized native code. HotSpot will first interpret the JVML bytecode and then eventually compile the hot code paths to optimized machine code.

PyPy: compiles to PyPy bytecode, which then gets interpreted by the PyPy VM until it identifies the hot code paths which it then compiles into native code, JVML bytecode or CIL bytecode depending on which platform you are running on.

CPython: compiles to CPython bytecode which it then interprets.

Stackless Python: compiles to CPython bytecode which it then interprets.

Unladen Swallow: compiles to CPython bytecode which it then interprets until it identifies the hot code paths which it then compiles to LLVM IR which the LLVM compiler then compiles to native machine code.

You might notice that every single one of the implementations in that list (plus some others I didn’t mention, like tinypy, Shedskin or Psyco) has a compiler. In fact, as far as I know, there is currently no Python implementation which is purely interpreted, there is no such implementation planned and there never has been such an implementation.

Not only does the term “interpreted language” not make sense, even if you interpret it as meaning “language with interpreted implementation”, it is clearly not true. Whoever told you that, obviously doesn’t know what he is talking about.

Share and Enjoy

Permalink | Comment (0)

Python

Python producer – consumer queue

April 15, 2013 – 7:52 pm

import threading, time, random, Queue

q = Queue.Queue()

class Producer(threading.Thread):

    def __init__(self, q):
        super(Producer, self).__init__()
        print "Producer Started"
        self.q = q

    def run(self):
        for x in range(50000):
            self.q.put(50000)

class Consumer(threading.Thread):

    def __init__(self, q):
        super(Consumer, self).__init__()
        print "Consumer started "+self.name
        self.q = q

    def run(self):

        while not self.q.empty():
            numbers = self.q.get()            
            num = 0
            for x in range(numbers):
                num = x**2

            q.task_done()
            print numbers            

if __name__ == "__main__":    

    p = Producer(q)
    p.start()

    p1 = Producer(q)
    p1.start()

    for x in range(8):
        c = Consumer(q)
        c.start()

import threading, time, random, Queue

q = Queue.Queue()

class Producer(threading.Thread):

def __init__(self, q):

super(Producer, self).__init__()

print "Producer Started"

self.q = q

def run(self):

for x in range(50000):

self.q.put(50000)

class Consumer(threading.Thread):

def __init__(self, q):

super(Consumer, self).__init__()

print "Consumer started "+self.name

self.q = q

def run(self):

while not self.q.empty():

numbers = self.q.get()

num = 0

for x in range(numbers):

num = x**2

q.task_done()

print numbers

if __name__ == "__main__":

p = Producer(q)

p.start()

p1 = Producer(q)

p1.start()

for x in range(8):

c = Consumer(q)

c.start()

Share and Enjoy

Permalink | Comment (0)

PHP

Whats so great about PHP?

March 17, 2013 – 5:02 pm

The short answer – EVERYTHING

In a word php is the answer to web related questions like “what language should i code this in?”, “how can i make this thing?”, “how do i solve this problem?”. It’s syntax can be awkward but in my opinion it is close enough to Java and C# that it won’t force people to re think everything they ever knew about writing code. While I really love writing code in a more expressive language that gives a coder more flexibility in its syntax, I do feel strongly that php is a good candidate for the most important programming language to learn title.

I write php at work and many of my coworker also write php. We have a great team and I feel that if we were writing everything in a cumbersome language it would have a serious impact on productivity. My school has recently started teaching php in more classes and even teaching code ignighter in 1 class. If all schools started their programming courses by teaching their students php before languages like Java and C# I really think the world would be a much better place.

Out of all the students that are graduating with me and all of the previous graduates from my computer science program, I can definitely say that I know more people that went on the be php programmers than I do for any other language, and I believe this is in part due to the fact that more new companies are starting with linux as their foundation running apache web servers and php as their programming language. Gone are the days of Microsoft dominated work places.

Many of the most popular content management systems are written in php. I would even guess that 90% of all web site run on a back end of php code at this point, and the numbers are growing everyday! go php!

Share and Enjoy

Permalink | Comment (0)

Python

Django, Iron Python, Pygame for android

February 13, 2013 – 12:54 am

I am not usually the type of person that would promote the use of IIS over Apache for serving dynamic web content, but I do recognize that there are many work places out there that are stuck running exclusively on IIS.

For those people running IIS you are are actually in luck! Iron Python can be used to write all of you .NET applications.

We all know that .NET is better than Java. The .NET framework is huge and it comes with everything you could ever need for being a productive developer. HOWEVER… C# can be a bit of a pain to code in. When you need large scale applications built and you need them done fast I would recommend looking into Iron Python. Iron Python does’t have a global interpreter lock so it will let you do real multithreading. Iron Python has access to the FULL REAL .net framework. Iron Python will work with asp.net. There is absolutely no downside to using it.

Another really great thing you will want to do if you are running an IIS web server is to setup Hellicon ZOO! (google it) it is the absolute best thing ever invented. Using the Hellicon Zoo I was able to install Django on IIS using MSSQL as my database without ANY problems at all. The install is really easy, but I would recommend having a seasoned web server administrator perform the installation.

And one more really great python thing i started looking into recently that has nothing to do with IIS… Pygame subset for android!

Im not sure how much of pygame is actually implemented here, but I plan to find out really soon. I absolutely love pygame and I cant wait to get started with this!

Share and Enjoy

Permalink | Comment (0)

Movies

Raging Bull

January 2, 2013 – 2:43 am

Robert De Niro plays the part of the angry boxer fighting his way to the title in Raging Bull.

His role is very interesting in this one. De Niro plays the part of Jake LaMotta an angry man outside the ring and a beast inside. He seems to have some mental problems that make him out to be a dick head to his wife. He leaves his wife for a 15 year old girl that he meets at the local swimming pool. The girl is still in high school and he was a married professional boxer… what the hell is wrong with this guy?

So eventually Jake gets married to the 15 year old girl. Maybe she gets older in the movie but they never really say anything more about her age. The actress Cathy Moriarty was apparently graduating from high school when she was cast as Vicki LaMotta.

The movie really centers around the relationship between Vicki and Jake and Jake’s mistreatment of his wife. I really thought De Niro was great in this movie. The story wasn’t all that interesting, but De Niro brings the movie to life for me. His portrayal of Jake LaMotta is one that you won’t forget after the movie is over. I found myself randomly quoting Jake LaMotta when celebrating New Year’s Eve with Steph. The dialog sticks with you for some reason.

After Jake LaMotta fights Sugar Ray Robinson and wins, he is asked to throw his next fight by some guy who is paying his brother and helping Jake in his career. I think throwing the fight really put a mental strain on LaMotta and he takes it out on his brother and his wife. He seems paranoid for no reason. He believes that his wife is cheating on him and that his brother is out to get him. Jake even tries to say that his brother had sex with his wife.

Eventually Jake’s wife gets fed up with his complaining and she tells him that she is leaving, but only after Jake goes over to his brothers house and beats him up making a complete ass of himself.

The funny part is that when Jake’s wife threatens to leave him, Jake grabs here and kisses her and she gives in. They never break up and everything is suddenly fine again.

Towards the end of the movie we see Jake opening his own club, being arrested for an underage girl in his club and then going to jail for a while. At this point Jake is about 100 lbs over weight. I read that Robert De Niro gained 60 lbs for the role which was amazing to see.

Seeing the dedication to the role that De Niro has for the role and how well he does in the role really makes the movie great for me.

I give it 8/10

The beginning of the movie bored me to be honest. One of the first scenes of the movie Jake LaMotta (played by Robert De Niro) is yelling at his wife for overcooking his steak. The first thing I noticed is that he had an anger problem and probably beat his wife. I was right.

As the movie progresses, there is not much focus on Jake LaMotta boxing. The boxing seemed to be a side story. The movie did not have to hit the 2 hour mark. I was surprised to see that the movie came out in 1980. They did a good job of making it look very 1940′s, besides just shooting it in black and white.

I went into this movie thinking it was a boxing movie, but I soon found out that it was mostly about a crazy husband. LaMotta leaves his first wife (the one who overcooks the steak) when he meets Vicki, a local girl who sits at the pool all day. She is 15 years old, that did make me gag a bit considering LaMotta is at least 30 years old. So how ever it happens, LaMotta ends up marrying Vicki. He is seriously CRAZY about her. He is always convinced that she is cheating on him and then beats her for it. No matter what she does, this girl can’t do anything right in LaMotta’s eyes.

I do have to say that De Niro nailed the part as psycho husband. It was so good it was almost comical, because all you can do is shake your head and say “this guy is nuts!” It was so comical that Rich attempted to shock the hostess at dinner doing his best impersonation of De Niro from Raging Bull when we went out to dinner (which was comical in it’s own respect) let’s just say I played a horrible Vicki LaMotta- looking at Rich like he had two heads and smirking with laughter.

In one part of the movie, he asks his wife why she slept with his brother, she never did, says she never did until he beats her enough to say that she slept with him and a whole bunch of other guys. I mean how many times does she have to say no? So, she lied and said yes, oddly enough the beating stopped for a second until she took a K.O. punch to the floor at her brother-in-law’s house. Jake beat his brother Joey up pretty good for this too, which led to them never speaking again. Joey was played by Joe Pesci, I really liked this role for Pesci, it was (to my knowledge) the first time he played the character he usually plays, which was completely rehashed with a mafia twist in Goodfellas (1990).

Overall I liked the movie more than I expected. I was very impressed with everything that De Niro did for the role. He boxed and even won two matches as well as gained 60 pounds for the role. The irony of the movie was also striking. There was Jake LaMotta, a boxer who had it all but when he tried to take everything that made him a great boxer and apply that to his personal relationships he was left with nothing.

Rating: 7/10

Share and Enjoy

Permalink | Comment (0)

Ruby

First JRuby threading trials ( easy to follow for intro to jruby and threading )

December 20, 2012 – 1:21 am

Ruby programming is very new to me so I’ve decided to start out with some small examples to see how difficult it would be to do some things I do regularly in python and java.

Currently I’ve been working on a fast way to parse through giant libraries of images and store all of the image exif data, the locations to the images on disc, and various other data points.. in mysql.

Version 1 was done in CPython which turned out to be very slow. The code works 100% now but it takes forever to finish. The whole reason for writing it in python in the first place was because of how easy the language was to write utility programs like this. The problem came when trying to speed it all up with threading.

In python there is the GIL.. so I started porting this code over to Jython which speeds things up ALOT! In Cpython my cpu never reached over 150% on a 6 core cpu. That was around 25% maximum usage per core.

After porting the code to Jython the cpu now reaches its maximum potential

So after diving into this world of multithreading with python and jython I thought .. hey maybe this would be a good opportunity to write some ruby code! I have been wanting to write ruby code for a while but I just never get around to it. There are so many other languages that I code with everyday and I don’t have the chance to use Ruby at work.

Enter my first code snippet for a multithreaded JRuby number counter.

include Java

require 'thread'

class DirReader < Thread

     def initialize
          super &method(:run)
     end

     def run
          # puts Thread.current.object_id
          num = 0
          (1..5**10).each do |x|
               num += 1
          end
     end
end

threads = []
(0..5).each do
     threads.push DirReader.new
end

threads.each do |t|
     t.join
     puts t.object_id
end

include Java

require 'thread'

class DirReader < Thread

def initialize

super &method(:run)

end

def run

# puts Thread.current.object_id

num = 0

(1..5**10).each do |x|

num += 1

end

threads = []

(0..5).each do

threads.push DirReader.new

end

threads.each do |t|

t.join

puts t.object_id

end

So far I really like the fact that Jython and JRuby let me use the standard python and ruby threading modules without forcing me to implement any java interfaces. This is great!

I am also going to finish the Jython and JRuby mysql data blaster code soon and I will post some benchmarks when finished.

Share and Enjoy

Permalink | Comments (2)

Python

Python Threaded Image Resizing

December 17, 2012 – 3:18 am

Threading in python

can be somewhat of a sour topic. On one hand, yes python does support threading using operating system level threads. The threading module will allow you to create threads that will be fully independent of each other and managed by the host operating system to run on different cores. On the other hand the global interpreter lock will prevent the execution of all other threads while a single thread is being executed. This interpreter lock prevents errors in garbage collection. Basically pythons garbage collection system works by doing something called “reference counting” on variables used throughout your program. When a pointer to an object is lost the reference count is decremented. When the reference count is decremented to zero the memory is freed and the interpreter will go about it’s day.

What all of this means is that if python does not have a lock on the interpreter level the reference count could be corrupted and python would suffer from memory leaks.

So does this impact performance? Yes… There are cases when a threaded program in python will work slower than a single threaded program due to the GIL. (something like 80% of all cases)

Is there any real reason to use threading in python? Yes… I have read a lot on this topic and done my own benchmarks that reveal things like web servers, web readers, and disk reading operations will benefit from addition of threading into your program as I/O bound operations do typically release the GIL.

Other reasons to use python for threading

1. Teaching synchronization techniques

python is a simple language that lets you remove all of the “fluff” associated with programming. The language is so simple that even I can understand thread synchronization

2. Asynchronous operations.

asynchronous operations can be anything that does not require a response to happen before the execution of the rest of a thread. These types of operations are typicaly things like database update queries, responding to events without blocking user interfaces etc..

Is there any way I can use threading in python without the GIL? Yes… the GIL is an implementation specific design pattern specific to cPython. There is no GIL for Jython or IronPython as these two versions of python run on other platforms that have there own threading interfaces.

So lets do some image resizing using only 6 threads that are re-used when they finished. First we will start by importing all of the modules that will be used in this program. I want to read all of the exif information from each image, resize the image, parse the folder names to attempt to read a dated named folder if there is one, and insert all of this good stuff into mysql! (kind of a big task here, but no sweat python can handle this)

#import all of the modules that we 
#will be using throughout this code

import os, dateutil.parser, PIL, hashlib, json, re, threading, sys
from PIL import Image

import time as _time
import MySQLdb as mysql

#we will be reading exif information from the images as we go :)
from PIL.ExifTags import TAGS
from datetime import *

#import all of the modules that we

#will be using throughout this code

import os, dateutil.parser, PIL, hashlib, json, re, threading, sys

from PIL import Image

import time as _time

import MySQLdb as mysql

#we will be reading exif information from the images as we go :)

from PIL.ExifTags import TAGS

from datetime import *

After the import statements we will be creating a class that will inherit from threading.Thread

###
    next_thread is an object that will be set to the next thread that is to be a consumer
###
next_thread = None

#dir reader inherits from threading.Thread
class dir_reader(threading.Thread): 

    ###
        Setdir function is used to set the dir that will be parsed by this instance
        of the dir_reader class
    ###
    def setDir(self, path):
        self.path = path

    ###
        readFiles will be used to return a list of all files in the current dir
    ###
    def readFiles(self, dir):
        return os.listdir(dir)    

    ###
        getExif is used to read all of the exif info available for an image.
        This function will return a dictionary will key value pairs
    ###
    def getExif(self, image):
        ret = {}
        info = image._getexif()
        for tag, value in info.items():
            decoded = TAGS.get(tag, tag)
            ret[decoded] = value
        return ret

    ###
         createThumbnail will create a thumbnail out of an image and save 
         it to the thumbnails folder
    ###
    def createThumbnail(self, path, h):
        im = Image.open(path)
        im.thumbnail((100, 150), Image.ANTIALIAS)
        im.save("thumbnaildir"+h+".jpg", "JPEG")
        return im

###

next_thread is an object that will be set to the next thread that is to be a consumer

###

next_thread = None

#dir reader inherits from threading.Thread

class dir_reader(threading.Thread):

###

Setdir function is used to set the dir that will be parsed by this instance

of the dir_reader class

###

def setDir(self, path):

self.path = path

###

readFiles will be used to return a list of all files in the current dir

###

def readFiles(self, dir):

return os.listdir(dir)

###

getExif is used to read all of the exif info available for an image.

This function will return a dictionary will key value pairs

###

def getExif(self, image):

ret = {}

info = image._getexif()

for tag, value in info.items():

decoded = TAGS.get(tag, tag)

ret[decoded] = value

return ret

###

createThumbnail will create a thumbnail out of an image and save

it to the thumbnails folder

###

def createThumbnail(self, path, h):

im = Image.open(path)

im.thumbnail((100, 150), Image.ANTIALIAS)

im.save("thumbnaildir"+h+".jpg", "JPEG")

return im

The next method is a little bit trickier. We have to think about what is going on when using a thread. First the thread is started in the “run” method when using python’s threading module. The run method and the threading module are built to mimic the threading classes available for Java and I think most java programmers that have used the Runnable class would agree that coding python threads looks very similar.

The idea of our run method will be to look forever until a stopping flag is triggered. The thread will signal the main thread which is also looping forever and waiting on signals. When a worker thread signals the main thread, the worker thread will be assigned a new folder to parse through and the main thread will signal back to the worker.

    def run(self):
        ###
            This gives each thread a reference to the next_thread object so that it can
            aquire the lock and set itself as the next available thread when it finishes execution
        ###
        global next_thread

        while not self.done:
            dir = self.readFiles(self.path)     

            ###
                This is wear we begin to build up our bulk insert queries for mysql
                Bulk inserts are limited to 1000 files per insert which can translate to about
                80000 rows of exif information into the exif information table
            ###
            file_info_query = "insert into file_info(`id`, `folder_path`, `folder_date`, `size`, `last_mod`, `last_change`) values "
            file_info_holder = []

            ###
                The exif info bulk insert query will be significantly larger than the file
                info bulk insert due to the fact that each file can have up to 80 different
                pieces of exif information and we are saving all of it as key value pairs
            ###
            exif_info_query = "insert into exif_info (`file_id`, `key`, `value`) values "
            exif_info_holder = []

            print self.name+" Now indexing: %s" % self.path
            for x in dir:

                ###
                    If we run into a folder, we aquire the lock and add the folder to our
                    folder stack.
                ###
                if os.path.isdir(self.path+"/"+x):
                    print self.name+" Found folder: %s" % self.path+"/"+x
                    with self.lock:
                        self.all_folders.append(self.path+"/"+x)
                else:    
                    h = hashlib.md5(self.path+"/"+x).hexdigest()
                    if not os.path.exists("thumbnails"+h+".jpg"):    

                        ###
                              In the next section, we will gather the file information from the os.stat function
                              and we will try to run our date regular expressions on the file
                              and folder name to determine if a date has been applied to 
                              one of the folder names
                        ###

                        (mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime) = os.stat(self.path+"/"+x)

                        atime = datetime.fromtimestamp(atime)
                        mtime = datetime.fromtimestamp(mtime)
                        ctime = datetime.fromtimestamp(ctime)

                        de = (
                            r'(0[1-9]|[12][0-9]|3[01])[- /.](0[1-9]|1[012])[- /.](19|20)\d\d',
                            r'(0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])[- /.](19|20)\d\d',
                            r'(19|20)\d\d[- /.](0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])'
                        )

                        for exp in de:
                            folder_date = re.search(exp, self.path)
                            if folder_date is not None:
                                folder_date = folder_date.group()
                            else:
                                folder_date = "0000-00-00 00:00:00"

                        image = None
                        try:
                            image = self.createThumbnail(self.path+"/"+x, h)
                            file_info_holder.append(
                                "('%s', '%s', '%s', '%s', '%s', '%s')" % (
                                    h, 
                                    (self.path+"/"+x).replace("'", "''"),
                                    folder_date,
                                    size,
                                    mtime,
                                    ctime
                                )
                            )
                        except IOError as e:
                            pass
                        except Exception as e:
                            raise e
                        if image is not None:
                            exif = None
                            try:
                                exif = self.getExif(image)
                            except Exception as e:
                                pass
                            if exif is not None:
                                for x in exif:
                                    if exif[x] != '':
                                        exif_info_holder.append(
                                            "('%s', '%s', '%s')" % (h, str(x), 
                                                str(exif[x]).replace("'", r'\'')
                                            )
                                        )

                        ###
                              If we have read through 1000 files its time to blast into mysql
                              or if we have reached the end of the dir
                        ###    
                        if len(file_info_holder) == 1000 or (dir[len(dir)-1] == x and len(file_info_holder) > 0):
                            #connect to mysql
                            self.con = mysql.connect('host', 'user', 'pass', 'db')
                            self.cur = self.con.cursor()

                            #join file query with file info list
                            try:
                                self.cur.execute(file_info_query+",".join(file_info_holder)+";")
                            except Exception as e:
                                print e

                            #join exif info query with exif info list
                            try:
                                self.cur.execute(exif_info_query+",".join(exif_info_holder)+";")
                            except Exception as e:
                                print e

                            self.con.commit()
                            self.con.close()
                            #print "%s files indexed in folder %s" % (str(len(file_info_holder)), self.path)
                            file_info_holder = []
                            exif_info_holder = []
            ###
                Now that were done reading through the folder, we aquire the lock
                Then we notify our condition variable and wait on our condition
                When we are notified that we can continue, our folder will have been 
                updated by the main thread and we loop again :)
            ###
            with self.lock: 
                next_thread = self
                self.cnd.notify()  
                self.cnd.wait()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

def run(self):

###

This gives each thread a reference to the next_thread object so that it can

aquire the lock and set itself as the next available thread when it finishes execution

###

global next_thread

while not self.done:

dir = self.readFiles(self.path)

###

This is wear we begin to build up our bulk insert queries for mysql

Bulk inserts are limited to 1000 files per insert which can translate to about

80000 rows of exif information into the exif information table

###

file_info_query = "insert into file_info(`id`, `folder_path`, `folder_date`, `size`, `last_mod`, `last_change`) values "

file_info_holder = []

###

The exif info bulk insert query will be significantly larger than the file

info bulk insert due to the fact that each file can have up to 80 different

pieces of exif information and we are saving all of it as key value pairs

###

exif_info_query = "insert into exif_info (`file_id`, `key`, `value`) values "

exif_info_holder = []

print self.name+" Now indexing: %s" % self.path

for x in dir:

###

If we run into a folder, we aquire the lock and add the folder to our

folder stack.

###

if os.path.isdir(self.path+"/"+x):

print self.name+" Found folder: %s" % self.path+"/"+x

with self.lock:

self.all_folders.append(self.path+"/"+x)

else:

h = hashlib.md5(self.path+"/"+x).hexdigest()

if not os.path.exists("thumbnails"+h+".jpg"):

###

In the next section, we will gather the file information from the os.stat function

and we will try to run our date regular expressions on the file

and folder name to determine if a date has been applied to

one of the folder names

###

(mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime) = os.stat(self.path+"/"+x)

atime = datetime.fromtimestamp(atime)

mtime = datetime.fromtimestamp(mtime)

ctime = datetime.fromtimestamp(ctime)

de = (

r'(0[1-9]|[12][0-9]|3[01])[- /.](0[1-9]|1[012])[- /.](19|20)\d\d',

r'(0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])[- /.](19|20)\d\d',

r'(19|20)\d\d[- /.](0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])'

)

for exp in de:

folder_date = re.search(exp, self.path)

if folder_date is not None:

folder_date = folder_date.group()

else:

folder_date = "0000-00-00 00:00:00"

image = None

try:

image = self.createThumbnail(self.path+"/"+x, h)

file_info_holder.append(

"('%s', '%s', '%s', '%s', '%s', '%s')" % (

(self.path+"/"+x).replace("'", "''"),

folder_date,

size,

mtime,

ctime

)

except IOError as e:

pass

except Exception as e:

raise e

if image is not None:

exif = None

try:

exif = self.getExif(image)

except Exception as e:

pass

if exif is not None:

for x in exif:

if exif[x] != '':

exif_info_holder.append(

"('%s', '%s', '%s')" % (h, str(x),

str(exif[x]).replace("'", r'\'')

)

###

If we have read through 1000 files its time to blast into mysql

or if we have reached the end of the dir

###

if len(file_info_holder) == 1000 or (dir[len(dir)-1] == x and len(file_info_holder) > 0):

#connect to mysql

self.con = mysql.connect('host', 'user', 'pass', 'db')

self.cur = self.con.cursor()

#join file query with file info list

try:

self.cur.execute(file_info_query+",".join(file_info_holder)+";")

except Exception as e:

print e

#join exif info query with exif info list

try:

self.cur.execute(exif_info_query+",".join(exif_info_holder)+";")

except Exception as e:

print e

self.con.commit()

self.con.close()

#print "%s files indexed in folder %s" % (str(len(file_info_holder)), self.path)

file_info_holder = []

exif_info_holder = []

###

Now that were done reading through the folder, we aquire the lock

Then we notify our condition variable and wait on our condition

When we are notified that we can continue, our folder will have been

updated by the main thread and we loop again :)

###

with self.lock:

next_thread = self

self.cnd.notify()

self.cnd.wait()

The idea behind all of this is that we will have a main thread that is holding a list of all folders that are waiting to be parsed. Each worker thread will add to the folder stack as it finds new folders and each time a thread asks for a new folder it will be given a folder from the top of the stack. This will create a system of folder crawling that will force our program to read through all sub folders before the program can exit and to be able to re-use threads instead of creating new ones.

if __name__ == "__main__":
    start = _time.time()

    ###
        Clean out our tables first
    ###
    print("Re-Indexing database..")
    con = mysql.connect('host', 'user', 'pass', 'db')
    cur = con.cursor()
    cur.execute("truncate table exif_info;")
    cur.execute("truncate table file_info;")
    con.commit()
    con.close()

    #Create a lock
    lock = threading.Lock()

    #Create our condition variable using our lock
    cnd = threading.Condition(lock)

    all_folders = []
    all_threads = []

    ###
        This is the root folder for all of your photos
        We will first read through the root folder and save the path to all folders
    ###
    start_folder = "/somefolder"
    for thing in os.listdir(start_folder):
        if os.path.isdir(start_folder+"/"+thing):
            all_folders.append(start_folder+"/"+thing)

    ###
        Create some threads to use and assign some variables to them so they can
        Have access to certain things

        I am creating 12 threads because my cpu has 6 cores and I wanted to experiment
        With choosing the optimal number. 6 vs. 12 seems to have no effect and I am
        Convinced that my only problem is due to the GIL
    ###
    for x in range(12):        
        d = dir_reader()
        d.setDir(all_folders.pop())
        d.lock = lock 
        d.cnd = cnd  
        d.done = False
        d.all_folders = all_folders
        d.start()
        all_threads.append(d)

    ###
       Aquire the lock and loop until we have cleared out our all_folders list
    ###
    with lock:
        while len(all_folders) > 0:
            #wait for a signal from a thread
            cnd.wait()
            print len(all_folders)
            ###
                 next_thread will already be set by now from the signaling thread
                 we just have to give it a new folder to parse
            ###
            next_thread.setDir(all_folders.pop())
            #notify the waiting thread that it is ok to go
            cnd.notify()

        sys.exit(0)

if __name__ == "__main__":

start = _time.time()

###

Clean out our tables first

###

print("Re-Indexing database..")

con = mysql.connect('host', 'user', 'pass', 'db')

cur = con.cursor()

cur.execute("truncate table exif_info;")

cur.execute("truncate table file_info;")

con.commit()

con.close()

#Create a lock

lock = threading.Lock()

#Create our condition variable using our lock

cnd = threading.Condition(lock)

all_folders = []

all_threads = []

###

This is the root folder for all of your photos

We will first read through the root folder and save the path to all folders

###

start_folder = "/somefolder"

for thing in os.listdir(start_folder):

if os.path.isdir(start_folder+"/"+thing):

all_folders.append(start_folder+"/"+thing)

###

Create some threads to use and assign some variables to them so they can

Have access to certain things

I am creating 12 threads because my cpu has 6 cores and I wanted to experiment

With choosing the optimal number. 6 vs. 12 seems to have no effect and I am

Convinced that my only problem is due to the GIL

###

for x in range(12):

d = dir_reader()

d.setDir(all_folders.pop())

d.lock = lock

d.cnd = cnd

d.done = False

d.all_folders = all_folders

d.start()

all_threads.append(d)

###

Aquire the lock and loop until we have cleared out our all_folders list

###

with lock:

while len(all_folders) > 0:

#wait for a signal from a thread

cnd.wait()

print len(all_folders)

###

next_thread will already be set by now from the signaling thread

we just have to give it a new folder to parse

###

next_thread.setDir(all_folders.pop())

#notify the waiting thread that it is ok to go

cnd.notify()

sys.exit(0)

If anyone has any comments for optimizing this solution please let me know. I am considering moving the entire thing into Jython to take advantage of the threading freedom but I wanted to flesh out the code design first and have it all working before moving it over.

I haven’t been able to find any Java based image library similar to PIL yet so if anyone knows of one that would work in a similar way to PIL without a steep learning curve please let me know. Thanks for reading

Share and Enjoy

Permalink | Comment (0)

Python

Simple 2D transformations in python

November 27, 2012 – 3:41 am

Here is a fun code snippet that will get you thinking about translation, rotation, and scale. (I called these functions move, resize, and rotate)

Run this code and edit to play around with some simple 2d transformations. Click and drag to draw some lines, then click right mouse button to draw scaled, rotated, and translated

from tkinter import Tk, Canvas
from math import sin, cos

lines = [
    [[0,0],[10, 100]],
    [[10, 100], [10, 120]],
    [[10, 120], [50, 120]],
    [[50,120], [60, 100]]
]

t = Tk()

c = Canvas(t, width=800, height=600)
c.pack()
mouse_loc = []
def mousemove(event):
    mouse_loc.append((event.x, event.y))
    c.delete("all")
    for p, y in enumerate(mouse_loc):
        if p % 2 == 0 and p != 0:
            c.create_line(
                mouse_loc[p-1][0],
                mouse_loc[p-1][1],
                mouse_loc[p][0],
                mouse_loc[p][1]
            )
def mouse2(event):
    c.delete("all")
    lines = []
    for p, y in enumerate(mouse_loc):
        if p % 2 == 0 and p != 0:
            lines.append([
                [mouse_loc[p-1][0],
                mouse_loc[p-1][1]],
                [mouse_loc[p][0],
                mouse_loc[p][1]]
            ])
    for line in lines: c.create_line(*line)
    for line in lines: c.create_line(*move(line, 500, 500))
    for line in lines: c.create_line(*resize(move(line, 300, 200), .5))
    for line in lines: c.create_line(*move(rotate(line, -5), 500, 50))

c.bind('<B1-Motion>', mousemove)
c.bind('<Button-3>', mouse2)

def move(l, dx, dy): return list( (po[0] + dx, po[1] + dy) for po in l)
def resize(l, p): return list( (po[0] * p, po[1] * p) for po in l)
def rotate(l, d): return (( l[0][0] * cos(d) - l[0][1] * sin(d), l[0][1] * cos(d) + l[0][0] * sin(d) ),( l[1][0] * cos(d) - l[1][1] * sin(d), l[1][1] * cos(d) + l[1][0] * sin(d) ))

for line in lines: c.create_line(*line)
for line in lines: c.create_line(*move(line, 500, 500))
for line in lines: c.create_line(*resize(move(line, 300, 200), .5))
for line in lines: c.create_line(*move(rotate(line, -5), 500, 50))

t.mainloop()

from tkinter import Tk, Canvas

from math import sin, cos

lines = [

[[0,0],[10, 100]],

[[10, 100], [10, 120]],

[[10, 120], [50, 120]],

[[50,120], [60, 100]]

]

t = Tk()

c = Canvas(t, width=800, height=600)

c.pack()

mouse_loc = []

def mousemove(event):

mouse_loc.append((event.x, event.y))

c.delete("all")

for p, y in enumerate(mouse_loc):

if p % 2 == 0 and p != 0:

c.create_line(

mouse_loc[p-1][0],

mouse_loc[p-1][1],

mouse_loc[p][0],

mouse_loc[p][1]

)

def mouse2(event):

c.delete("all")

lines = []

for p, y in enumerate(mouse_loc):

if p % 2 == 0 and p != 0:

lines.append([

[mouse_loc[p-1][0],

mouse_loc[p-1][1]],

[mouse_loc[p][0],

mouse_loc[p][1]]

])

for line in lines: c.create_line(*line)

for line in lines: c.create_line(*move(line, 500, 500))

for line in lines: c.create_line(*resize(move(line, 300, 200), .5))

for line in lines: c.create_line(*move(rotate(line, -5), 500, 50))

c.bind('<B1-Motion>', mousemove)

c.bind('<Button-3>', mouse2)

def move(l, dx, dy): return list( (po[0] + dx, po[1] + dy) for po in l)

def resize(l, p): return list( (po[0] * p, po[1] * p) for po in l)

def rotate(l, d): return (( l[0][0] * cos(d) - l[0][1] * sin(d), l[0][1] * cos(d) + l[0][0] * sin(d) ),( l[1][0] * cos(d) - l[1][1] * sin(d), l[1][1] * cos(d) + l[1][0] * sin(d) ))

for line in lines: c.create_line(*line)

for line in lines: c.create_line(*move(line, 500, 500))

for line in lines: c.create_line(*resize(move(line, 300, 200), .5))

for line in lines: c.create_line(*move(rotate(line, -5), 500, 50))

t.mainloop()

Share and Enjoy

Permalink | Comment (0)

Python

Getting started with Matplotlib

November 26, 2012 – 11:23 pm

Welcome to the wonderful world of scientific programming with Python

SO LETS GET STARTED!

First we will need to download python. I recommend downloading python version 3.2 from here. Why 3.2? Because python 3 has significant speed improvements and syntax updates. Also there is a python 3.3 but you won’t find a pre built version of numpy and scipy for it.

After installing python 3.2 you should install numpy from here. Choose the latest version of numpy that is compatible with python 3.2 and your type of processor (32 or 64bit).

Next you will need to download and install scipy from here. Once again make sure that you choose the correct version of python and the appropriate version for your cpu type.

Finally we can install matplotlib from here .

Once you have successfully installed all of the requirements, you can begin writing some code. I would strongly recommend using an IDE with intellisense that will give you functions with auto completion etc.. so that you can see which modules contain which methods. I prefer Aptana Studio.

Ok so now for some code!

If everything goes as planned, this should show you the graph of x squared.

Share and Enjoy

Permalink | Comment (0)

Games

2500 – A Space Shooter

November 25, 2012 – 5:50 pm

2500

Is a space shooter that I started working on as a demo for a kinect application I have been writing for my capstone project class at SCSU.
The game mechanics are similar to other side scrolling space shooter games with the added twist of Kinect integration. So far there is only a demo of a first level that has been started, but I wanted to upload it here to maybe get some feedback and to test out the Burster web version of Blender!!

If anyone out there is interested in playing with the .blend file or helping out with building things into this game please let me know. I would love to hear any feedback.

I am embedding the burster playing version (minus kinect integration) into this post so if you would like to try out the game, simply download burster player and restart your browser. Come back to this post and the game will start!

If you dont have the burster web player visit Geta3D.com to download.

Share and Enjoy

Permalink | Comment (0)

Rich Wandell's Blog

Python and the interpreted/compiled misconceptions

Share and Enjoy

Python producer – consumer queue

Share and Enjoy

Whats so great about PHP?

Share and Enjoy

Django, Iron Python, Pygame for android

Share and Enjoy

Raging Bull

Share and Enjoy

First JRuby threading trials ( easy to follow for intro to jruby and threading )

Share and Enjoy

Python Threaded Image Resizing

Threading in python

Share and Enjoy

Simple 2D transformations in python

Share and Enjoy

Getting started with Matplotlib

Welcome to the wonderful world of scientific programming with Python

Share and Enjoy

2500 – A Space Shooter

2500

Share and Enjoy

Tweetz

Share and Enjoy

Share and Enjoy

Share and Enjoy

Share and Enjoy

Share and Enjoy

Share and Enjoy

Threading in python

Share and Enjoy

Share and Enjoy

Welcome to the wonderful world of scientific programming with Python

Share and Enjoy

2500

Share and Enjoy

Categories

Tweetz