Fortunately one of my users created a search that returned parsing errors of this nature and allowed me to dive into a solution to this problem. First of all, considering Craigslist will not be modifying the user input mechanism used to builds its feeds, and certainly will ignore my request for better formatting, i took it apon myself to do a little research. By submitting the feed for validation on http://validator.w3.org/feed I was able to get the exact issue causing the feed to break:

I couldn’t tell exactly what the issue was (aside from ‘not well-formed (invalid token)‘) until I pasted the contents into Notepad++, which displayed a strangely encoded character within the string. To fix the issue I needed to cleanse the extracted RSS feed just before it gets passed to PHP’s xml_parser, and after tracing the execution back I was able to pin-point it within rss_parse.inc right around line 130 or so.

$status = xml_parse( $this->parser, $source );

The contents of $source is the data within the rss feed and the point just before the parsing is conducted. The solution was simple - str_replace() the contents of $source before passing it to the parser.

function MagpieRSS ($source, $output_encoding='ISO-8859-1',
                        $input_encoding=null, $detect_encoding=true)
    {
		/*
		 * Feed cleansing (encoding replacement stuff) here CLF - 10/25/2008
		 * Should execute it on $source (the content extracted from feed)
		 */	
 
		 $source = str_replace('[bad data]','', $source);
 
		 /*
		  * END - feed cleansing
		  */
 
...

I wasn’t able to show the actual character I needed to replace, but you get the idea - replace [bad char] with the character(s) you want to remove before parsing. If you can’t seem to find it try pasting the section of bad data into Notepad++ and it will display it for you - notepad and wordpad will not. Invalid encoding/chars is only one way that feeds can be mal-formed, so if you run into other problems you would need to modify $source accordingly.

As some of you already know, I’ve been working on a site that lets you automate searching on websites such as Craigslist. I recently needed to purchase a vehicle and was repeatedly beaten to the punch by others who were able to monitor the ads much closer than I. AdRavage lets you create search terms, how often to search those terms, and ultimately sends out emails and/or text messages when matching results are found. The site is in beta as of now, and will soon feature other websites besides Craigslist.

Check it out! http://adravage.com

I think this happened after upgrading a few versions back, but had never actively used categories until now. All of my posts should have been uncategorized by default, but for some reason a huge number of them simply did not have a corresponding record in wp_term_relationships and as a result were drifting along in non-categorized (not in the default category ‘uncategorized’ mind you) purgatory.

Rather than sift through each and every post to see if it has an entry in the table, the script below will

1. Loop through every post and check if post id exists in wp_term_relationships table
2. If the post id does have a corresponding object id in wp_term_relationships its skipped
3. If the post id does not have a corresponding object id in table then add an entry using defined CATEGORYID

Make sure that you define a valid CATEGORYID in the script (default is 1). Look in the table wp_terms and find the term_id for the category you want each missing post to have, then set CATEGORYID appropriately.

<?php
/*
 * cleanseWordpressPosts.php
 * Script used to re-insert posts information missing in wp_term_relationships to fix category issues
 * Written by: Chris Fay
 *9/15/2008
 *
 * To Use: First backup your database!
 * Copy the contents of this script to a file called updateCat.php
 * Upload updateCat.php to your web server
 * Verify the script is set to run in test mode first to see what data would be inserted
 * Browse to the script on the web server
 * Change testmode to false to allow script to update database   
 */
 
$dbhost = 'hostname'; //usually localhost
$dbuser = 'dbUserNameHere'; //replace dbUserNameHere with your username
$dbpass = 'dbPasswordHere'; //replace dbPasswordHere with your user's password
$dbname = 'dbName'; //replace dbName with the database name you want to connect to
define('TESTMODE', true); //leave true to get information on what would be updated - the database
						  //will not be touched if set to true - change to false to allow the script to modify the datbase
define ('CATEGORYID', 1); //the category id you would like all posts missing category info to have
 
$conn = mysql_connect($dbhost, $dbuser, $dbpass) or die                      ('Error connecting to mysql');
mysql_select_db($dbname);
 
//loop through each record, cleanse it, then re-insert it
$query  = "SELECT id FROM wp_posts";
$query2 = "SELECT object_id FROM wp_term_relationships";
$totalUpdates = 0;
 
$result = mysql_query($query);
 
if(TESTMODE) {
	echo "**** TEST MODE: No data will be modified in your database ****<br /><br />";
}
 
while($row = mysql_fetch_array($result, MYSQL_ASSOC))
{	
	$id = $row['id']; 
	echo "Checking id: $id<br />";	
	$match = false;	
	$result2 =  mysql_query($query2);	
 
	while($row2 = mysql_fetch_array($result2, MYSQL_ASSOC))
	{
		if($id === $row2['object_id'])					
		{
			echo "Found match for id: " . $id . " - skipping. <br /><br />";
			$match = true;
			//no need to keep looping through since an item already exists
			break;						
		}
	}  
 
	if(!$match)
	{
		echo "No match found for id: ". $id . " - processing new category information<br />";
		$totalUpdates++;
 
		if(TESTMODE) 
		{
			$query3 = "INSERT into wp_term_relationships VALUES ($id,".CATEGORYID.",0)";	  	
			echo "The insert query we'd be using:<br /> $query3<br /><br />";				
		} 
		else
		{
			//update wp_term_relationships with missing id info
			echo "Creating category info for ID: $id. <br>";
 
			$query3 = "INSERT into wp_term_relationships VALUES ($id,".CATEGORYID."5,0)";	  					
			mysql_query($query3) or die('Error, insert query failed');
			echo "Category info updated successfully<br /><br />";	    				
		}
	}
 
} //end while
 
echo "Total updates: <strong>$totalUpdates</strong> <br/>All finished...<br />";
 
mysql_close($conn);
?>

We the people say “Over our dead bodies!!”

NWO replies ” Ok fine with us”

And police forces all around the globe light the night with tasered bodies EVERYWHERE.

WTB the ultimate undeniably inevitable collapse of this FARCE of a world society in which the cornerstone of its foundation is the way of get/greed once and for all plz. And no,I don’t want any fucking fries with that.

Going from a Suzuki gsxr 600 to a 125CC scooter will surely be demoralizing - as will be the glares I receive on the road to work in the morning - but I’m left with no choice and ultimately find it a better option than buying a second gas gusler.

A message to those driving in the KC area: please don’t kill the little guy on the red scooter, he will haunt you in your sleep.

I was a bit worried about our appliances though, considering neither Jennie nor I thought to measure the space (the fridge already in there was frick’n huge!), and fortunately most of them went in without a hitch - that is except the fridge. Fortunately for us, and less than fortunately for dad, the trim and a bit of the counter top was all that need to be modded for it to slide gleefully into its final resting place. Dad, you’re an Egyptian magician.

I’ll leave you with this picture of us, post unload into the new house, and I think Jennie’s stance says it all…

With Hayden growing older and a new baby on the way it became evident that the apartment lifestyle, as wonderful as it were in my bachelor and early marriage years, was simply not going to cut it for us Fay’s. After a few weeks of digging, searching, looking, complaining, searching, hopelessness, and more searching we finally found a house a block away from the house I grew up in before moving to AZ. I can see my old elementary school from the front porch (not sure if that’s a good or bad thing), and all in all I’m looking forward to the transition - though I’m not excited about the additional dollar dollar bills I’ll have to throw down. We’ll have some pictures up soon I’m sure…

Examples:

’
…

After browsing through the posts it looks like those characters represent things like quotations, dashes and double periods. I read through various forums that said this is caused due to the encoding and collation, however after following the recommendations of sites like http://www.mydigitallife.info/2007/0…to-version-22/ I was still unable to get rid of the huge amount of garbled characters. Ultimately, I spent some time mapping each set of characters to the appropriate one (quotes, commas, etc…) and built a php script to basically:

1. Loop through every post in the Wordpress database
2. Find and replace all the garbled characters
3. Re-insert the data

I have attached the script below (very simple) in case others find this the only way to cleanse out this character issue. One other thing I should mention - this only works if the garbled characters are actually garbled in the database - if the issue is simply how your browser is rendering the text, and not how the strings are actually stored in the database, then this wont help you. Although feel free to give it a shot.

One other thing - you may need to add additional characters to replace in the script. It should be self explanatory how based on what I have presented already. If you need help. simply post in here.

<?php
/*
 * cleanseWordpressPosts.php
 * Script used to replace various characters displayed in posts with appropriate values
 * Written by: Chris Fay
 * 6/14/2008
 *
 * To Use: First backup your database!
 * Run the script in test mode to see what data would be cleansed
 * Change testmode to false to allow script to cleanse database 
 *
 * If you need other characters clensed then simply duplicte one of the lines that look like:
 *
 * $cleansed = str_replace("“", "", $cleansed, $count4); 
 *
 * .. and if using the example above replace the “ with the string you want to replace
 * and inside the double quotes add the value you want to replace with. 
 */
 
$dbhost = 'localhost';
$dbuser = 'dbUserNameHere'; //replace dbUserNameHere with your username
$dbpass = 'dbPasswordHere'; //replace dbPasswordHere with your user's password
$dbname = 'dbNam'; //replace dbName with the database name you want to connect to
define('TESTMODE', true); //leave true to get information on what would be updated - the database
						  //will not be touched if set to true - change to false to allow the script to modify the datbase
 
$conn = mysql_connect($dbhost, $dbuser, $dbpass) or die                      ('Error connecting to mysql');
mysql_select_db($dbname);
 
//loop through each record, cleanse it, then re-insert it
$query  = "SELECT id, post_content FROM wp_posts";
$result = mysql_query($query);
 
if(TESTMODE) {
	echo "**** TEST MODE: No data will be modified in your database ****<br /><br />";
}
 
while($row = mysql_fetch_array($result, MYSQL_ASSOC))
{	
	$id = $row['id'];
	$unclensed = $row['post_content'];
 
	echo "Cleansing id: $id<br />";	
 
    $cleansed = str_replace("’", "'", $unclensed, $count1); 
	$cleansed = str_replace("ÂÃ", "", $cleansed, $count2); 
	$cleansed = str_replace("â€", "", $cleansed, $count3); 		
	$cleansed = str_replace("“", "", $cleansed, $count4);
	$cleansed = str_replace("…", ".", $cleansed, $count5);
	$cleansed = str_replace("’", "'", $cleansed, $count6);
	$cleansed = str_replace("â„¢", "'", $cleansed, $count7);
	$cleansed = str_replace("Å“", "", $cleansed, $count8);
	$cleansed = str_replace("˜", "'", $cleansed, $count9);
	$cleansed = str_replace("¦¦", "", $cleansed, $count10);
	$cleansed = str_replace("¦", "", $cleansed, $count11);
	$cleansed = str_replace("â€Â", " ", $cleansed, $count12);
 
	$escaped = mysql_escape_string($cleansed);
 
	if(TESTMODE) {
		echo "ID: $id would have been cleansed. <br>";
		echo "Cleanse count for each string:<br />"
			."’: $count1<br />"
			."ÂÃ: $count2<br />"
			."â€: $count3<br />"
			."“: $count4<br />"
			."…: $count5<br />"
			."’: $count6<br />"
			."â„¢: $count7<br />"
			."Å“: $count8<br />"
			."˜: $count9<br />"
			."¦¦: $count10<br />"
			."¦: $count11<br />"					  
			."â€Â: $count12<br />";
 
		$query2 = "update wp_posts set post_content = '".$escaped."' where id = '".$id."'";	  	
		echo "The insert query we'd be using:<br /> $query2<br /><br />";
	} else {
		echo "ID: $id clensed. <br>"
			."Inserting data into table<br />";
 
		$query2 = "update wp_posts set post_content = '".$escaped."' where id = '".$id."'";	  	
		mysql_query($query2) or die('Error, insert query failed');
		echo "Post cleansed and updated<br /><br />";	    
	}
} //end while
 
echo "All finished...<br />";
 
mysql_close($conn);
?>

I was not however, spared from a frigid dip in said river. We had come so far - almost the entire trip had blissfully passed without so much as an innocent splash of river warf on our clothes and inside our canoe. We had become uber men, quasi canoe super beasts, ready to battle any and all rapids that mother earth could toss our direction. It was in the peak of our vainglorious narcissism that she catapulted the head of our man canoe into a stump, then our glorious tail into yet another stump. In less than 10 seconds we were sunk, bobbing down the river in 40 degree water, clutching to the canoe, a wetbag, and our dignity. 

After finally escaping the clutches of the river we had so dominated earlier, our frustration was quickly replaced by yet another realization that our pullout was less than a mile down river. Not only had we capsized, but we did so in the final stretch of the 24 mile trip. Oh how close we came to total domination – congrats to G&B for achieving such canoe mastery and avoiding a fate of humiliation and noobitude. 

For a more visual representation of the trip see our home page for pics and the videos section for some live footage of the debauchery.

All of these magical components bundle together in order to serve one higher purpose – canoe trippage. That’s right - I’m embarking on a mystical journey with three other primates into the nefarious jungle in order to heed the man yell grumbling deep within us all. Or, if you’re like me, you got suckered into the idea that floating down a river would clench your burning desire to escape reality for one small moment. 

My original disposition was inaccurate after realizing that death is surely imminent – the river has flooded and we will surely peril under its mighty wrath. <sarcasm>Fortunately for us, the water level is just ‘barely’ low enough for the rangers to keep the river open – great, so we will just ‘barely’ survive the trip.</sarcasm>