Thursday, October 29, 2009

Data Recovery Tips and Tricks

If your hard drive is dying, go and buy a new hard drive and a USB enclosure for it. If you have a copy of Norton Ghost, you can clone the failing hard drive over to the new hard drive. Alternatively, if you have the courage, build a WinPE CD using either PEBuilder or UBCD. If you wish to go the Linux route, there's System Rescue CD, although any Linux Live CD will work. If you build a WinPE clone, put Norton Ghost32.exe or the Windows version of ddrescue.exe along with the cygwin1.dll on the CD. When using cygwin or its tools, you have to use the POSIX device names for the drives.

Here's some tips on using ddrescue:
1. Always use a log file
2. Start ddrescue like this the first time:
ddrescue /dev/sda1 /dev/sdb1 logfile.log (sda1 will be the C: drive, sdb1 will be
the replacement drive.) Do not get them backwards in order.
3. Once you notice a ddrescue struggling to read through bad sectors, kill the job.
In Linux, you can try pkill ddrescue, or ps -ef | grep ddrescue . Find the PID or
process id number. Then type kill -9 pid, i.e. kill -9 1234.
4. Rerun ddrescue using the -i and -s switches, i.e. ddrescue -i75G -s125G logfile.log
(start copying 125GB after the first 75GB). The -s switch is size or amount to copy.
The -i switch is the initial position to copy from. You want to copy as much of the good
parts of the drive as quickly as possible, so skip the bad parts if you can.
5. Once you have most of the drive copied, then kill the job and type the following:
ddrescue -n -d /dev/sda1 /dev/sdb1 logfile.log . Ddrescue will read the log file and
only copy what it hasn't copied. With the above command, it will try to trim the bad
sectors and recover as much data as it can without recovering the bad sectors.
6. You can then try to recover the bad sectors, but this will be VERY time consuming and
problematic. Rerun ddrescue like so: ddrescue -r3 -d /dev/sda1 /dev/sdb1 logfile.log .
7. You can then rerun ddrescue using the -n -d switches to get anything you missed, but
by this stage it is likely you've recovered all the data that you are going to get.
Google "man ddrescue" and "info ddrescue" for more information about the program.

If you need to recover an audio CD, you can use cdrdao. Ddrescue is only good for recovering digital media. Audio CDs use a different format and can't be read by ddrescue. If the audio CD has scratches, you will get skips using a normal CD or DVD burning program. I have built a GUI front end for ddrescue that works on Windows. If any one wishes to use it, you can email me at jbmoore61 [AT] gmail[DOT]com. I hope to have it on or some other convenient web site shortly.

Using an Ubuntu liveCD is easiest. Just boot your system with the liveCD. Under accessories, choose terminal. Then type sudo su. The prompt will change to # which means you are essentially root. Then type dhclient to grab an IP address, or you can click on the adapter icon in the upper righthand corner of the screen. Then type apt-get update ; apt-get install ddrescue. Choose y if prompted and ddrescue will be installed into the ramdisk. You can then use ddrescue to recover and copy your Windows drive to the new hard drive. Try to get a replacement drive that is the same size as your old drive. If you get a larger drive, you may have to run qpartd or gpartd to make the new copy use all of the new drive's disk space.


Why Windows Users Suffer

I have spent the last few days repairing a friend's system. He had put Vista on the system when he had problems, but Vista didn't recognize some of his hardware. It turned out that his 200 GB hard drive had 4,000MB of bad sectors. 4GB/200GB equals roughly 2% of his hard drive. Initially, he didn't know the drive was bad, and I assumed that it was good. When I booted the system up with a Linux Live CD and started moving files around, to get his Windows XP back, the system started a file integrity check when I booted back to Windows. That's when I started to notice problems. To make matters worse, I couldn't see his subfolders under Documents and Settings after I had tried to move the folder to its proper location. At first, I tried to be conservative and repair the drive and the operating system. The drive was unrepairable and Windows XP would freeze at 34 minutes into the repair setup routine because it couldn't find a specific driver. I didn't have this driver discovery freeze error if I did a clean install. To make a long story short, I spent over 25 hours recovering his drive with ddrescue by copying it to an identical size hard drive. I recovered all but the missing 4GB of bad sectors. I also recovered his Product keys using a program called Keyfinder. Once I had rebuilt the system, I reinstalled his software whose install files he kept on another drive on his computer. I then copied his data from the pristine copy to a pristine drive with an NTFS partition. I then copied those copies to his computer.

I realized that a lot of this pain could be avoided if Windows ran chkdsk more often. Windows only checks the hard drive's integrity when it is setup the first time. After that, it is up to the user to check the hard drive. If the hard drive starts becoming corrupted, Windows will do the best it can until a critical file is damaged at which point the operating system will likely fail entirely. I've seen a system lock up and freeze with no errors logged. When a chkdsk c: /f/r was run, bad sectors were found and marked and the computer acted normally again. But Windows will never complain about a dying hard drive, it just tries to function until it dies. I haven't had a major data loss in several years because I run Linux and the designers decided several years ago to include an automatic file system test after 25 reboots of the system. I have had a hard drive start to fail, but the damage was minimized by the file system check routine. If Windows had this feature, a lot of people wouldn't lose their important data and have such heartache. You can google "schedule chkdsk" to find ways to configure a Windows system to check itself, but when I tried to set something up, Task Scheduler balked because my normal user account didn't have enough privilege to run it, which means only an administrator account can schedule a chkdsk to run. The easiest way to run chkdsk is manually. Just go to your menu, Start -> Run -> enter the word cmd in the box and click OK. You'll get a command prompt. At the prompt, type chkdsk c: /f. You'll be asked to enter Y or N, enter y (yes). The next time you start up, a disk check will be performed. Get a cup of coffee while your computer is doing the disk check. If you get errors and it notices bad sectors, rerun the chkdsk command like this chkdsk c: /f/r . Then go to the store or doing something else for a half hour or an hour depending upon your hard drive size.


Sunday, October 25, 2009

I Am Sick and Tired of People Complaining About Our Educational System

Thomas Friedman has an op-ed entitled The New Untouchables. He states that the decline of our educational system is to blame for our troubles. His op-ed is based on that fallacy. The reason our educational system has declined is because our local and county governments gave property tax breaks and other concessions to companies to attract jobs to their areas. Our business leaders negotiated all these tax exemptions for themselves and then have the chutzpah to declare that the school systems are not up to snuff when they were the ones responsible for lack of funding for the schools in the first place. University science and technology graduates have declined the last 20 years because it was easier and more lucrative to go to Business School than Engineering or Natural Sciences. Sales and management jobs don't tend to be either insourced (H1-B visa workers) or outsourced unlike engineering and science jobs. Therefore, the former professions' pay and career advancement don't suffer unlike the tech workers. The latter two schools had to accept more graduate students from overseas to keep up enrollments, and the majority of those graduates with advanced degrees stay in the United States because there are no jobs for them back home. I have a doctorate and I made $25,000/year for my first postdoc in Los Angeles. I was 32 years old. I was trying to get out of debt and live on $25K a year. It was hard, but I managed. It took me ten years to pay off my graduate student loans and my Mother insisted that she pay for my undergraduate loans for which I was grateful to her. I eventually had to leave my Science career and start over. It was the most painful decision I ever had to make. However, I earned more my first year in computer tech support than I ever made as a scientist. How insane is that?

The pay for postdocs has gotten better in recent years, but the problem of career advancement remains. There are too many people competing for too few jobs because the universities need the cheap graduate student labor to keep research and development costs down. That is why the universities howled when the student visa program was changed after 9/11. Businesses are no better. I was mandated to take 40 hours training per year at my last job. My performance review depended upon it as did a professional certification. Yet, my managers seldom ever let us use the new skills we learned to make our jobs easier. They certainly didn't give us raises for getting more and better certifications in IT Security. In fact, the smarter young people generally got taken advantage of by management asking them to take on more tasks for the same amount of pay and sometimes less pay. Any suggestions we made to improve operations were either ignored or languished because they fell on the deaf ears of management. Now companies import H1-B visa workers because they are cheaper than American graduates to employ, or they outsource jobs to India or China because it's even cheaper. What do economic decisions such as insourcing or outsourcing do to an educational system's graduates and the educational system itself? When a job leaves the country, the tax income from that job is lost forever. Depressed wages lead to less tax income. It's a negative feedback loop. So, economic decisions made by businesses have ramifications beyond their business spreadsheets. If our so called political leaders don't have the backbones to say no to our so called business leaders when the latter try to extort concessions and subsidies from our governments, then the rest of us who are good taxpaying citizens suffer the consequences. If we haven't learned that lesson from the latest financial crisis, then we deserve the fate awaiting us because of our misguided economic beliefs, ignorance, and shortsightedness.


Wednesday, October 21, 2009

Symptoms of a Much Bigger Problem

Here are some stories.

Kajakai Dam supplies power to southern Afghanistan, but unfortunately, only one turbine is currently working. A major joint British and American operation delivered a new turbine to the dam last year, but according to Michael Yon, all that money and effort may be wasted. Without the dam's powerplant coming fully online before 2014, aid money that was contingent on the dam providing needed power for Helmand Province will not materialize.

Then, there is the US Intel Community using open (public) sources for information. They've always used such sources, but now they are shooting themselves in the foot because of analysts giving themselves away on the Internet.

There are perils to the process. One source here said that analysts who engage in searches without masking their origin can lead to foreign governments or companies cutting off access to web sites or to people involved. The problem? Some analysts at NSA, CIA and other alphabet soup agencies forget to mask their IP addresses and the times at which they are searching. Chinese, Russian and other savvy operators can check time stamps, for example. If a search occurs during American working hours, it’s a pretty good bet that it’s an American source looking for the information.

Haven't these people heard of open proxies? If Russian cybercriminals can buy and use a colocation facility in the U.S., how difficult would it be for our Intel Community to buy or create open proxies in neutral or foreign countries for their analysts to use, and then train them to use them. There may be technical reasons why this is difficult, such as the use of a mandatory proxy server at work, but that's why the Three Initial Agencies have big black budgets. There are workarounds for these sorts of issues. But, that doesn't let them off the hook for letting their analysts get sloppy doing their research and compromising their intelligence gathering.

This last blog post by Tom Ricks is from a Canadian Military Intelligence officer who recently came back from Afghanistan. His report is illuminating and would be hilarious, verging on hysterically funny, if it wasn't symptomatic of the underlying problems NATO faces training the indigenous forces in Afghanistan.

Afghan National Army military intelligence officers brought an interesting perspective to signals interception: "rather than passively listening [to enemy radio traffic], the ANA had a tendency to get into arguments with insurgents."

Yet, when someone believed in us and did their job, we abandoned them to the Taliban.

In one remote village, strong Afghan commanders worked hard to deny the area to the Taliban, and also gained a remarkable amount of intelligence. But then the outpost "was closed just after the end of our tour due to its sustainment difficulties, in all likelihood dooming many of the locals who had collaborated with us there." This is the opposite of protecting the population -- it is endangering them.

These issues shouldn't have happened. They are all easily preventable. That they have been allowed to happen and continue unchecked shows a lack of attention to details that matter. All of these issues are symptoms of a much deeper and worrisome problem. With Afghanistan, they indicate that we will fail because we are not carrying through on our strategies and objectives. If we don't care about the Afghans and their country, then what are we doing there wasting Afghan, American, and others' lives, money, and resources? In the case of intelligence analysts being allowed to compromise their own online research, it implies that the American Intelligence Community isn't nearly as smart as it thinks it is. I hope the Chinese and Russians aren't laughing too hard while they collect reams of intelligence (and money) from our own military and civilian networks while denying us the same.


Monday, October 19, 2009

Confusing the Nmap Scanner

Nmap is a powerful port scanning tool. Like all tools, it can be used for good or ill. I have used it for years, but in a straight forward and trivial way. While doing some online research, I discovered that the -v -sV combination enumerates services quite well. Indeed, it will show you the banner or give you other information. So, I was a bit surprised to find that the version of openSSH running on my systems gave the version number and the OS version. It was likely that nmap was just matching strings in its signatures file. There are a number of ways to throw off a port scanner:

1. Create a firewall rule that drops all packets recognized as coming from a port scanner.
2. Change the strings in the source code and recompile the application.
3. Change the strings in the binary file itself.

In my case, option three was the easiest option because the system I was interested in protecting is a honeypot in a DMZ. Option 2 may or may not work depending upon how old the source code is and what compiler and other build software it requires. Anyway, I fired up khexedit and replaced the version number and OS version characters within the sshd binary. You'll want to try to avoid adding or subtracting characters and just replace them with different characters. After testing, OpenSSH appears to be working fine. I also did the same thing for the module for nepenthes. The first time I tried to change the string in the module it broke nepenthes, so you might want to back up the file you are working on before you try this. When I ran a nmap scan on the honeypot, nmap failed to recognize my openSSH version or that nepenthes was running on the box. This trick won't fool a sophisticated attacker, but it might confuse a script kiddie or an automated nmap scan.

Labels: ,

Interesting Times

I am currently trying to collect unemployment benefits from the state of Texas. I was released from my job back in August. I filed for benefits immediately. Although the state says that I am eligible to collect unemployment, my status is now listed as "under review". When I tried to call the Texas Workforce Commission this morning, the automated system stated that I could not be accommodated at this time and shuttled me into the automated equivalent of oblivion. By automating services they've made things cheaper, and not just in the sense of providing services, but in the sense of not providing services. The benefits come in the form of a JP Morgan Chase debit card. What a scam!


Saturday, October 10, 2009

Simple Gimp Tricks

The Gimp is a free photo manipulation tool as powerful as Photoshop. I just found a blurb entitled 5 Super Simple Photo Fixes with the GIMP. So, I tried one of them. I was never happy with the dark snapshot I had of that opossum I found on my landing. I had deleted that snapshot after I uploaded it here. So, I tried the GIMP level correction trick.

Here's another still taken from the Flip video mp4 footage I shot:

opossum in the dark - an original snapshot

Here's the level corrected version:

altered opossum.jpg

That's a bit better. The corrected jpeg picture is closer to the light levels I see in the actual digital movie. Uh-oh! This posting looks fine on my old Linux workstation, but on my laptop, the upper photo is fine, but the lower photo is too light. Guess I should have checked the posting on all of my systems to see if there was any difference in light levels.


Friday, October 09, 2009

Obama Wins the 2009 Nobel Peace Prize

President Obama has won the 2009 Nobel Peace Prize. For doing what exactly? I used to believe that the Nobel Peace Prize was awarded to politicians such as President Jimmy Carter and Secretary of State Henry Kissinger for putting a great deal of effort in negotiating peace deals. President Bill Clinton failed to win one because he failed to broker a deal like Carter had. Former Vice-President Al Gore got his prize for making a movie and educating the public about the risks of climate change. Obama has just outlined his diplomatic goals, but it's too early to tell if his foreign policies are having any effect or bearing fruit. So, this seems to be more of a PR prize and an attempt to give him more legitimacy by the Royal Swedish Academy of Sciences, than an award for accomplishing a major diplomatic treaty among factions or states. All that said, congratulations President Obama! I'm glad you won the Nobel Peace Prize. Now, please do your job and earn the peace (and arms reductions) you say you desire. With the Great Depression II in effect, the world will likely be less stable and peaceful for the foreseeable future. A depressed global economy only makes your job harder. I wonder what the average Afghan tribesman thinks about this piece of news caught between the Taliban and ISF?

This blogger has a point. All the wingnuts on Fox News and elsewhere are going to explode, so there is a humorous side to this announcement. Considering the bombastic vitriol on Fox News and talk radio though, this isn't a major insight.


Web Page Text Smeared When Scrolling Down

I've had this annoying issue for some time. When I scroll down a web page displayed in Firefox lines of text disappear or are duplicated. The pages were not rendering correctly in Firefox on either my laptop or my home Linux workstation. It was the ATI Radeon display driver for Linux. If I set the nomodeset bootup parameter for either the 2.6.30 or 2.6.31 kernel in either system the problem goes away. I should have done something about this problem a lot sooner, but I thought it was an issue with KDE4 or Firefox, not the damn display driver.

Still having problems with the X-server crashing. I had to remove the nomodeset boot parameter in menu.lst. Performing the following:

apt-get install fglrx-driver

likely resolved the issue since the fglrx-driver is for 2-D graphics for ATI cards. Nothing else seemed to work.


Thursday, October 08, 2009

Online F.A.C.E. Training

Paul Ekman is offering F.A.C.E. training for those needing to spot lying. It seems rather inexpensive training for such a valuable skill.



The following was inspired by a Letter to a Dying Man and indirectly by George Breed over at Embodying Spirit.

You are a snowflake. Arising from nothing, perfect in form, evaporating in the bright sunshine. An endless end since you will return as a snowflake. Your essence is Stillness. It is not born and cannot die. That is the reality. What other snowflakes do, say, believe, or imagine is largely beyond your control or grasp. Snowflakes are best born and live in silence. Their silence inspires an appreciation and wonder of beauty. Snowflakes born in the fury of the blizzard inspire fear of the cold and of the elements. Each snowflake thinking itself flawed wishes to be beautiful, despite the fact that it was born beautiful, but many mistake noise, sound, and other nonsense with beauty instead of realizing that all beauty arises from silent awareness.


Wednesday, October 07, 2009

Learning Qt4.5 - Qt4 SDK and Qtcreator 1.2.9

** Warning: this is a very geeky posting for developers and other IT professionals. **

I've been learning the Qt4 version of C++ programming for the last few weeks. I started playing with qtdesigner last spring and then dropped it. A newer version called Qtcreator came out recently, so I picked up where I left off. The relevant links are Qt4 download page for SDK, Qtcreator, etc. and bleeding edge Qtcreator 1.2.9 snapshot. I'm using The Book of Qt4 as a reference. It's not the best manual for learning an SDK, but it's pretty much all that's available. It's rather a shame since the Qt4 toolkit is well designed and it is easy for even novices with some help to code a small decent graphical C++ program or GUI front-end for a console application. There are some decent tutorials at One of the tutorials, the media player, leaves out some key information that's in the code, and I could not get the two media based programs to function properly on my x86_64 Linux laptop due to the fact that the tutorials were created on a Mac. The ffmpeg program failed because it didn't recognize a program option which was not the fault of the Qt4 application. GTK failed on the media player which was also not the fault of the Qt4 application. Qt4 is supposed to be multiplatform, but if the underlying software fails the developer is screwed unless he can find a workaround. The tutorials are still quite useful, more so than some of the book examples which are scattered piecemeal throughout the book and therefore difficult to assimilate or understand. It's difficult to capture a visual workflow or process in a book, and the toolkit is intensively visual.

With Windows, one will need to install the SDK first, then Qtcreator. It is also best to install the SDK twice, and custom install the second copy. One will need to bring up the qtconsole on the second copy and run the following commands:

configure -static
mingw32-make or mingw32-make sub-src

(Info from and trolltech, and from

The first make command statically compiles all of the SDK source code and examples while the second version mingw32-make sub-src only statically compiles the SDK source code and takes significantly less time to build if one is in a hurry. However, the latter command still takes 30 or more minutes on a dual-core system. You will then need to point Qtcreator (via Tools-> Options) to the static version of the SDK, though it's pretty good at autodetecting the needed files. I still had to include the mingw32.dll with my Windows programs to get them to work, but the statically compiled versions were significantly smaller (10-fold) and more portable than the dynamically linked and compiled versions.

I primarily develop on Linux and then port the code to Windows. The problem with Windows is that the application paths and environmental variables are different which means that one has to change some code (see below) unless one is smart enough to create portable code. (Conditional if or switch statements which check to see which platform the code is being compiled on can be used.) I spent 2-3 hours last night trying different variations of code portability and failed due to variable scope limitations in the conditional statements which was a bummer indeed. I ended up putting Windows specific code in comments. I will likely have to post something on the Qt specific forums to find the answers I need.

Coding differences with Linux and Windows:

On Linux/Unix:

QString program = "program name";

On Windows:
QString program = "absolute path of program";

e.g. QString program = "C:/Tools/scalpel/scalpel.exe";

args << "-c";
args << "C:/Tools/scalpel/scalpel.conf";

So, I have to add two extra lines and modify a third line of code to get the same program to work on Windows. This is not even taking into account differences in the Windows version of the program my program is accessing.

Addendum: Found instructions for installing and running the latest version of ffmpeg and x264 on a Debian based systems. There is also this ffmpeg cheat sheet. The ffmpeg front-end program now works when built on my AMD64 laptop.


Sunday, October 04, 2009

How Many People Will Fall for this Phish Email?

Not from Bank of America

I received this in my inbox tonight. I wonder how many people will fall for this ruse? If you look, it seems to come from Likely's email server was used as a relay. This is

Administrative Contact, Technical Contact:
Vatturi, Sujan, Inc.
1900 Seaport Blvd
3rd Floor
Redwood CIty, CA 94063

Any real email would come from a Bank of America address, and they would not have you fill out a form to send them via email or post via the Worldwide Web unless you were a business account. (Yes, business to business transactions are a lot more lax. You'd think the security would be tighter, but it's actually almost nonexistent in many instances.) I generally don't comment about these things, but these scams are becoming so common and banks are losing people's money to these scams with greater frequency, yet no one knows the losses. What's worse is that your helpful Windows computer may be the link in the chain that allows some bad guy to rob your bank account some day. I don't use Windows for any banking transactions and I've had one credit card account compromised this year which means that either a merchant I've used, the bank that issued the credit card, or the company handling credit card transactions had its systems compromised. My guess is the latter since I didn't use that card for very many purchases.

But back to the scam at hand, buried in the web form they have you download is this bit:

...="Ps" action="" method="post" is in Lima, Peru.

IP Telefonica del Peru
Calle San Felipe 1144, 1144,
LI34 - Lima - LI

So, you are submitting your information to a server in Peru, but when you connect to it, it immediately redirects your browser to Bank of America:

wget -c
--2009-10-04 23:57:42--
Connecting to connected.
HTTP request sent, awaiting response... 302 Found
Location: [following]
--2009-10-04 23:57:42--
Connecting to||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

The 302 code is an HTML redirect. They don't want their site indexed by search engines because this is the easiest way to find them and shut them down. However, if Bank of America was clever, they would be correlating their customers' IP addresses accessing their web server with redirections from to gain a rough idea who is being fooled. They should also put in a complaint with that Peruvian telecom to shut down that server. But all the customer sees is the following web page while the bad guys abscond with the financial information the customers have given them.

Redirection to Bank of America from Peruvian server

Pretty clever, huh? It's an easy way to make a living for someone in a poor country. And, it costs all of us money in the form of bank fees.


Personal Experience?

If I understand Krishnamurti, most religions emphasize personal experience. You will be saved if you do this or that thing, follow that person, or belief. But if we are to believe modern science, Krishnamurti, the Buddha, and other deep thinkers, there is no such thing as personal experience. What I mean is that if every being is an observer participating in this reality or world we live on, then every observer is only perceiving a fragment of the whole of reality. When two or more observers observe the same event, the result is a shared experience. This is just as true for two human beings, a human being and a pet cat, dog, or bird, or any combination of human and other animals. We humans tend to think of ourselves as unique, but I know my cats have their needs and wants and their own ideas about how their world should be. Dogs are not as fussy, but even dogs have their own personalities and talents, and wants and needs.

So, if shared experiences are the most common events between human beings, how can a religious belief system work by emphasizing that people have personal experiences? Personal experience seems to be an oxymoron. Krishnamurti uses the ocean as an example and says that the ocean is there for any one to look at and admire, but it is not YOUR ocean. You don't own it, so it can't be a personal experience. But the same holds true for most everything else on the planet. The only people who may have had a personal experience were the command module pilots on the Apollo Moon missions who orbited around the Moon alone while their colleagues were on the lunar surface, and they weren't entirely alone except when their spacecraft went into the radio shadow of the Moon.

I believe that what Krishnamurti and other people are emphasizing is that spiritual growth is the lessening of the "me", the "me" identity of the mind. Others call it the loss of self. When I was born, there was no "me" - no conscious "me". There was a baby, but there was no "me" yet. When this body dies, there will be no "me" as well. The mind will have died with the brain. So, what is it that strives to understand whatever it is? What is it that loves, knows joy, laughs, cries, and tries to find happiness and contentment in the here and now? What is it that recognises itself in others, but is not the "me"?


How to Write a Chinese Poem

How to write a Chinese poem.


This page is powered by Blogger. Isn't yours?