Monday, November 16, 2009
PaiMei Framework Installation Woes, or Why Some Open Source Projects Give Free Software a Bad Rep
I am a big fan of Open Source Software, or Free Software as the GNU Foundation terms it. But occasionally, it can be extremely frustrating. Here's an example. Yesterday, I decided not to work on an online IT Security course I've been taking for the last two weeks. Instead, I decided that I would try to install a reverse engineering framework called Paimei on a Windows system. What would have been trivial for a Windows, Mac, or Linux install became a nightmare for this particular Windows installation. The first major gotcha is that the installation instructions are not up to date. They call for Python 2.4 to be installed, but the current version of pydbg in Paimei requires the python25.dll which means Python version 2.5 needs to be installed (more on that later). With Python25, one doesn't need the ctypes package. The other software requirements seem to be accurate.
The second gotcha was due to file permissions. My Windows systems would not see the contents of certain folders due to the read-only permissions from the read-only Subversion repository. Now admittedly, the developers set read-only file permissions as a security precaution, but would it hurt to tell Windows users that they might have problems installing the software due to file permission issues since this framework is used almost exclusively on Windows systems? I had downloaded the code to my Linux system previously and it saw all the files. The only way I found that I could get the files to be read by Windows was to burn them to a CD on my Linux system and copy them from the CD to the hard drive. Downloading them directly via a browser from the Google repository is futile. Copying them from Linux to Windows via an SMB share was just as futile even after changing file permissions to read-write. There might be a way to avoid this issue via the Windows version of Subversion, but I am not aware of it.
The third gotcha was the __install_requirements.py installation script. It had not been updated. The first issue with the script is that it downloads the old software whose links are now stale and it doesn't necessarily detect the newer required software even when the packages are installed. The second issue is that the script suppresses standard errors to the console. It would have been nice to know that the reason it couldn't find pydbg even though all the install files were there was that the failure was due to a missing python25.dll, and by implication, the use of the wrong version of Python that was specified by the developers themselves. The third issue was that the script builds version 1.2 of the Windows installer (via a batch file), but looks for and invokes version 1.1 which doesn't exist. Once you fix the batch file and that line in the script, things tend to be easier.
So, here's some tips on installing the Paimei Reverse Engineering Framework (as of November 2009):
1. Install Python version 2.5 instead of Python 2.4 as they state. Make sure you set the PythonPath environmental variable and modify your Path statement accordingly.
2. After you download the Paimei svn files, make sure your Windows system sees all the files. I noticed that the console folder was empty and this led me to realize that the file permissions were causing a problem on Windows. If you don't have a Linux system handy, boot your Windows box off of a Linux Live CD set to "use ram" (load the OS into the ramdisk) so that you can access your CD/DVD burner. Then download Paimei and burn it to CD. Then reboot into Windows, and copy the files to your Windows system.
3. Install all of the required software packages as listed in the installation instructions with the exception of Python24.
4. Fix the error in the __build_installer.bat file from
c:\python\python.exe setup.py bdist_wininst --bitmap=logos\installer.bmp --title=PaiMei
to
c:\python25\python.exe setup.py bdist_wininst --bitmap=logos\installer.bmp --title=PaiMei.
5. Fix the __install_requirements.py script or use the one below. I have not documented my changes, so run a diff on this one and the original if needbe. They may fix these errors some day soon, so check first.
6. Change directories into console and invoke the console like so:
python .\PAIMEIconsole.pyw
Good luck!
#!c:\python\python.exe
# $Id: __install_requirements.py 194 2007-04-05 15:31:53Z cameron $
import urllib
import os
import shutil
# globals.
downloaded = 0
########################################################################################################################
def urllib_hook (idx, slice, total):
global downloaded
downloaded += slice
completed = int(float(downloaded) / float(total) * 100)
if completed > 100:
completed = 100
print "\tdownloading ... %d%%\r" % completed,
def get_it (url, file_name):
global downloaded
downloaded = 0
u = urllib.urlretrieve(url, reporthook=urllib_hook)
print
shutil.move(u[0], file_name)
os.system("start " + file_name)
########################################################################################################################
try:
print "looking for ctypes ...",
import ctypes
print "FOUND"
except:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://superb-east.dl.sourceforge.net/sourceforge/ctypes/ctypes-0.9.9.6.win32-py2.4.exe", "installers/ctypes-0.9.9.6.win32-py2.4.exe")
try:
print "looking for pydot ...",
import pydot
print "FOUND"
except:
print "NOT FOUND"
try:
print "looking for wxPython ...",
import wx
print "FOUND"
except:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://umn.dl.sourceforge.net/sourceforge/wxpython/wxPython2.6-win32-ansi-2.6.3.2-py24.exe", "installers/wxPython2.6-win32-ansi-2.6.3.2-py24.exe")
try:
print "looking for MySQLdb ...",
import MySQLdb
print "FOUND"
except:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://superb-east.dl.sourceforge.net/sourceforge/mysql-python/MySQL-python.exe-1.2.1_p2.win32-py2.4.exe", "installers/MySQL-python.exe-1.2.1_p2.win32-py2.4.exe")
try:
print "looking for GraphViz in default directory ...",
fh = open("c:\\program files\\graphviz2.24")
close(fh)
except IOError, e:
if e.errno == 2:
print "NOT FOUND"
else:
print "FOUND"
try:
print "looking for Oreas GDE in default directory ...",
fh = open("c:\\program files\\govisual diagram editor")
close(fh)
except IOError, e:
if e.errno == 2:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://www.oreas.com/download/get_gde_win.php", "installers/gde-win.exe")
else:
print "FOUND"
try:
print "looking for uDraw(Graph) in default directory ...",
fh = open("c:\\program files\\udraw(graph)")
close(fh)
except IOError, e:
if e.errno == 2:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://www.informatik.uni-bremen.de/uDrawGraph/download/uDrawGraph-3.1.1-0-win32-en.exe", "installers/uDrawGraph-3.1.1-0-win32-en.exe")
else:
print "FOUND"
try:
print "looking for PaiMei -> PyDbg ...",
import pydbg
print "FOUND"
except:
print "NOT FOUND"
try:
print "looking for PaiMei -> PIDA ...",
import pida
print "FOUND"
except:
print "NOT FOUND"
try:
print "looking for PaiMei -> pGRAPH ...",
import pgraph
print "FOUND"
except:
print "NOT FOUND"
try:
print "looking for PaiMei -> Utilities ...",
import utils
print "FOUND"
except:
print "NOT FOUND"
choice = raw_input("\nInstall PaiMei framework libraries to Python site packages? ").lower()
if choice.startswith("y"):
os.system("start C:\\paimei\\installers\\PaiMei-1.2.win32.exe")
print "\nRun __setup_mysql.py to setup database and complete installation. Then run console\PAIMEIconsole.py"
raw_input("\nHit enter to exit installer.")
#---------end of file---------------
The second gotcha was due to file permissions. My Windows systems would not see the contents of certain folders due to the read-only permissions from the read-only Subversion repository. Now admittedly, the developers set read-only file permissions as a security precaution, but would it hurt to tell Windows users that they might have problems installing the software due to file permission issues since this framework is used almost exclusively on Windows systems? I had downloaded the code to my Linux system previously and it saw all the files. The only way I found that I could get the files to be read by Windows was to burn them to a CD on my Linux system and copy them from the CD to the hard drive. Downloading them directly via a browser from the Google repository is futile. Copying them from Linux to Windows via an SMB share was just as futile even after changing file permissions to read-write. There might be a way to avoid this issue via the Windows version of Subversion, but I am not aware of it.
The third gotcha was the __install_requirements.py installation script. It had not been updated. The first issue with the script is that it downloads the old software whose links are now stale and it doesn't necessarily detect the newer required software even when the packages are installed. The second issue is that the script suppresses standard errors to the console. It would have been nice to know that the reason it couldn't find pydbg even though all the install files were there was that the failure was due to a missing python25.dll, and by implication, the use of the wrong version of Python that was specified by the developers themselves. The third issue was that the script builds version 1.2 of the Windows installer (via a batch file), but looks for and invokes version 1.1 which doesn't exist. Once you fix the batch file and that line in the script, things tend to be easier.
So, here's some tips on installing the Paimei Reverse Engineering Framework (as of November 2009):
1. Install Python version 2.5 instead of Python 2.4 as they state. Make sure you set the PythonPath environmental variable and modify your Path statement accordingly.
2. After you download the Paimei svn files, make sure your Windows system sees all the files. I noticed that the console folder was empty and this led me to realize that the file permissions were causing a problem on Windows. If you don't have a Linux system handy, boot your Windows box off of a Linux Live CD set to "use ram" (load the OS into the ramdisk) so that you can access your CD/DVD burner. Then download Paimei and burn it to CD. Then reboot into Windows, and copy the files to your Windows system.
3. Install all of the required software packages as listed in the installation instructions with the exception of Python24.
4. Fix the error in the __build_installer.bat file from
c:\python\python.exe setup.py bdist_wininst --bitmap=logos\installer.bmp --title=PaiMei
to
c:\python25\python.exe setup.py bdist_wininst --bitmap=logos\installer.bmp --title=PaiMei.
5. Fix the __install_requirements.py script or use the one below. I have not documented my changes, so run a diff on this one and the original if needbe. They may fix these errors some day soon, so check first.
6. Change directories into console and invoke the console like so:
python .\PAIMEIconsole.pyw
Good luck!
#!c:\python\python.exe
# $Id: __install_requirements.py 194 2007-04-05 15:31:53Z cameron $
import urllib
import os
import shutil
# globals.
downloaded = 0
########################################################################################################################
def urllib_hook (idx, slice, total):
global downloaded
downloaded += slice
completed = int(float(downloaded) / float(total) * 100)
if completed > 100:
completed = 100
print "\tdownloading ... %d%%\r" % completed,
def get_it (url, file_name):
global downloaded
downloaded = 0
u = urllib.urlretrieve(url, reporthook=urllib_hook)
shutil.move(u[0], file_name)
os.system("start " + file_name)
########################################################################################################################
try:
print "looking for ctypes ...",
import ctypes
print "FOUND"
except:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://superb-east.dl.sourceforge.net/sourceforge/ctypes/ctypes-0.9.9.6.win32-py2.4.exe", "installers/ctypes-0.9.9.6.win32-py2.4.exe")
try:
print "looking for pydot ...",
import pydot
print "FOUND"
except:
print "NOT FOUND"
try:
print "looking for wxPython ...",
import wx
print "FOUND"
except:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://umn.dl.sourceforge.net/sourceforge/wxpython/wxPython2.6-win32-ansi-2.6.3.2-py24.exe", "installers/wxPython2.6-win32-ansi-2.6.3.2-py24.exe")
try:
print "looking for MySQLdb ...",
import MySQLdb
print "FOUND"
except:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://superb-east.dl.sourceforge.net/sourceforge/mysql-python/MySQL-python.exe-1.2.1_p2.win32-py2.4.exe", "installers/MySQL-python.exe-1.2.1_p2.win32-py2.4.exe")
try:
print "looking for GraphViz in default directory ...",
fh = open("c:\\program files\\graphviz2.24")
close(fh)
except IOError, e:
if e.errno == 2:
print "NOT FOUND"
else:
print "FOUND"
try:
print "looking for Oreas GDE in default directory ...",
fh = open("c:\\program files\\govisual diagram editor")
close(fh)
except IOError, e:
if e.errno == 2:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://www.oreas.com/download/get_gde_win.php", "installers/gde-win.exe")
else:
print "FOUND"
try:
print "looking for uDraw(Graph) in default directory ...",
fh = open("c:\\program files\\udraw(graph)")
close(fh)
except IOError, e:
if e.errno == 2:
print "NOT FOUND"
choice = raw_input("\tWant me to get it? ").lower()
if choice.startswith("y"):
get_it("http://www.informatik.uni-bremen.de/uDrawGraph/download/uDrawGraph-3.1.1-0-win32-en.exe", "installers/uDrawGraph-3.1.1-0-win32-en.exe")
else:
print "FOUND"
try:
print "looking for PaiMei -> PyDbg ...",
import pydbg
print "FOUND"
except:
print "NOT FOUND"
try:
print "looking for PaiMei -> PIDA ...",
import pida
print "FOUND"
except:
print "NOT FOUND"
try:
print "looking for PaiMei -> pGRAPH ...",
import pgraph
print "FOUND"
except:
print "NOT FOUND"
try:
print "looking for PaiMei -> Utilities ...",
import utils
print "FOUND"
except:
print "NOT FOUND"
choice = raw_input("\nInstall PaiMei framework libraries to Python site packages? ").lower()
if choice.startswith("y"):
os.system("start C:\\paimei\\installers\\PaiMei-1.2.win32.exe")
print "\nRun __setup_mysql.py to setup database and complete installation. Then run console\PAIMEIconsole.py"
raw_input("\nHit enter to exit installer.")
#---------end of file---------------
Labels: Paimei RCE frustrations
Comments:
<< Home
i think you put the original script there instead of the modified one, at least the links still point to python 2.4 stuff.
I did not change the install URLs for the first three packages because I'd already downloaded them. Besides, the URLs will change over time. But it's not the original that I can tell.
Thanks for the article; I actually just went through this process before running across your post, and your fixes will definitely help people.
I also note that with mysql > 5.0.25 the __setup_mysql.py script will fail because of bug 21532 (http://bugs.mysql.com/bug.php?id=21532).
The error looks like:
"BLOB/TEXT column 'eax_deref' can't have a default value"
Essentially the "default ''" syntax used in the CREATE TABLE statements is no longer supported, and probably won't be in the future. Anyone trying to use PaiMei with recent versions of MySQL should find/replace "default ''" with empty string.
Cheers,
PST
I also note that with mysql > 5.0.25 the __setup_mysql.py script will fail because of bug 21532 (http://bugs.mysql.com/bug.php?id=21532).
The error looks like:
"BLOB/TEXT column 'eax_deref' can't have a default value"
Essentially the "default ''" syntax used in the CREATE TABLE statements is no longer supported, and probably won't be in the future. Anyone trying to use PaiMei with recent versions of MySQL should find/replace "default ''" with empty string.
Cheers,
PST
Thanks for this informative article. Indeed, Paimei installation scripts/documents requires lot of changes. Beside what has been listed in this post, I found that pydasm is also a source of annoyance. The supplied one assumes pytohn24.dll, whereas python25.dll is required. So, we need to download pydasm and compile it with python25 and replace the new pydasm.pyd in pydbg folder under site-packages.
Occasionally, my_ctypes.py also requires some changes that can be found in paimei forum.
regards
-Sanjay
Post a Comment
Occasionally, my_ctypes.py also requires some changes that can be found in paimei forum.
regards
-Sanjay
<< Home