Archive | Programming  

How to create a Python or Django logger to log to a string

8 Jan

I had to allow some of our admin users to view the logic and results of a parsing module to help find and pinpoint improvements. Since my backend Python Celery module already uses Python logging, I needed a way to log to a string to display the results on a Web page.

Here’s a function that will return a StringIO variable and a Python logger. Once you are done logging or need the logger results, you can easily get the output logger in a string. Also, I found a Stack Overflow article to remove Django database debug info from being logged so I added this as a default parameter that can be changed.

Here’s the Gist:

Email recipes and templates for nicely formatted HTML messages

29 Aug

A few geeks may still prefer text emails but when building a site or an app, you know you need HTML emails. It just still works. What doesn’t always work, thought, is the myriad of mail clients that all have their quirks and may break some of your formatting. What you need is a recipe or a template to display your stuff the way you want and ensure it works across most rich clients.

MailChimp HTML templates

While reading Hacker News this morning, I stumbled on this thread about HTML Email Boilerplate and discovered two great resources that deserve a place in your toolbox:

Creating a Generic jQuery Countdown Timer

7 Sep

I needed to create many HTML countdown timers for a little pet project of mine. So, I wrote a jQuery selector function to create countdown timers on demand. These timers are really simple, they countdown for a number of seconds to zero.

Add this to your scripts:

(function($) {
  $.fn.createCountdownTimer = function(selector,time,tick,cb) {
    var seconds = time;
    var el = this;
    tick = tick || 1;
    var timer = function() {
      seconds -= tick;
      el.find(selector).text(String(seconds));
      if (seconds > 0)
         setTimeout(timer,tick * 1000);
      else
         cb && cb();
    };
    setTimeout(timer,tick * 1000);
    return this;
 };
})(jQuery);

Parameters tick and cb are optional. tick defaults to 1 as you usually tick down 1 second at a time.

Suppose you have a HTML countdown timer such as:

<div id="my_timer">Starting in <span class="timer">10</span> seconds...</div>

then, all you need to do to start a countdown from 10 to 0 is:

$("#my_timer").createCountdownTimer(".timer",10);

You can also supply a callback as a fourth parameter that will be called when the timer is done counting to zero. For example:

$("#my_timer").createCountdownTimer(".timer",10,1,function() {
   alert('Done!');
});

Integrating bit.ly with WordPress 3.0 Shortlinks Support

2 Jul

The official release of WordPress 3.0 being a bit recent, I’ve had a hell of a time trying to find bit.ly plugin that supports WordPress 3.0. The idea is to get your bit.ly short URL instead of the WordPress 3.0 short URL that uses your own domain name when you press the “Get Shortlink” button of a post editor.

bit.ly support in WordPress 3.0

I found the “WP Bit.ly” plugin which looks pretty good but does not yet support WordPress 3.0.

So what does an unsatisfied geek do on a Friday evening instead of having dinner? He finds a solution.

1. Download the WP Bit.ly plugin

2. Fix the plugin code

In file wp.bitly.php, put lines 47-48 in comment:

/*add_action( 'wp',      'wpbitly_shortlink_header' );
add_action( 'wp_head', 'wpbitly_shortlink_wp_head' ); */

If you too are a geek and you need to know, this will make sure WordPress 3.0 continues to take care itself of the HTML response header and the Web page HTML header as per the rel=”shortlink” proposed standard. Otherwise, the WP Bit.ly plugin would add these twice to a page which wouldn’t be clean.

3. Modify your theme’s function.php

Add the following code to your theme’s functions.php file:

/* bit.ly shortlink */

function get_bitly_shortlink($shortlink, $id, $context, $allow_slugs) {
   if (function_exists('wpbitly_get_shortlink')) {
      $slink = wpbitly_get_shortlink($id);
      if (empty($slink)) {
         wpbitly_generate_shortlink($id);
         $slink = wpbitly_get_shortlink($id);
         }
      return !empty($slink) ? $slink : $shortlink;
      }
   return $shortlink;
}
add_filter( 'get_shortlink', 'get_bitly_shortlink',10,4);

This code links the plugin code to the new WP 3.0 shortlink feature. It also makes sure that WordPress 3.0 will continue to operate as before if you do not enable the WP bit.ly plugin or you disable it.

Generate the bit.ly links from the plugin

So that the “Get Shortlink” button gets you a bit.ly link (instead of the default permalink), these should be generated once from the plugin’s settings. For new posts new bit.ly links should be created automatically.

Be aware that visits and crawls to your site will generate bit.ly links for each visited posts if no link already existed for each visited post. This is due to the fact that the short links are included in the code of each of your page to support new standards.

Fix for Facebook URL Linter / Open Graph Protocol Bug with WordPress 3.0

2 Jul

I’ve been helping At Home with Kim Vallee support the Open Graph Protocol promoted by Facebook. It’s easy to implement and great research that we’ll be adding immediately to Shwowp. If you want to learn a bit more why it’s important for your site to implement the Open Graph Protocol, read this article: “Facebook Unleashes Open Graph Search Engine, Declares War On Google”.

URL Linter is a handy Facebook Developer tool that allows you to analyze a Web page meta data to see how Facebook understands it.

The problem is that URL Linter was working for the home page but not for the article pages. After much lost sleep and a call for help on Facebook Developer Forums, Paul from Facebook confirmed it was a bug:

  • WordPress 3.0 added a shortlink feature
  • If you changed your WordPress permalinks to friendly names instead of the default, it adds an HTTP header variable in the form
    • Link    <http://domain.com/?p=12345>; rel=shortlink
    • This new header variable is part of a proposed microformat standard.
    • The Facebook URL Linter and crawler currently has trouble parsing the page when it sees this HTTP header

The Fix

The bug should be fixed by Facebook in about a week. In the meanwhile, if you run WordPress 3.0 and are implementing Open Graph Protocol support to your site, you can add the following code in your theme’s functions.php to disable the shortlink in the HTTP header output:


/* Remove WP 3.0 shortlink */
function empty_shortlink($shortlink, $id, $context, $allow_slugs) {
return NULL;
}
add_filter( 'get_shortlink', 'empty_shortlink',10,4);

How to fix the WordPress plugin WP-SpamFree to avoid the “Your location has been identified as part of a reported spam network” error

11 Mar

My wife Kim is the successful editor behind At Home with Kim Vallee. She uses the WordPress platform.

With success and a popular platform, problems with comment and contact form spam becomes quickly a pain to manage. That’s why you need a good toolset to keep your sanity intact. Good WordPress plugins to protect against spamming and keep your installation secure include:

  • Akismet – protects against spam
  • Login LockDown – adds extra security to your login form
  • WP Security Scan – security
  • WP-SpamFree – extra spam protection

WP-SpamFree is quite invaluable against robots and contact form spam. However, it came to our attention that some people would see the following message instead of the contact form:

Your location has been identified as part of a reported spam network. Contact form has been disabled to prevent spam.

See the message when it would appear instead of Kim’s contact form:

image

This was really annoying. People were telling Kim they couldn’t send comments.

By chance, one PC at our office had this bug. The unique IP address assigned to our network by Videotron seemed clean. So I dug further and looked at the plugin code to understand and debug what’s happening. It seems some browsers in some configurations (in our case a particular Firefox installation on Windows XP) do not transmit the HTTP_ACCEPT_LANGUAGE variable to servers. There is a check in the WP-SpamFree plugin that identifies the visitor as a spammer if this variable is empty.

Once the problem identified, the fix is easy: simply disable this verification. The quick fix to the plugin is to modify the wp-spamfree.php file in the plugin directory to put this condition in comment. See my changes in red:

/*
            $user_http_accept_language = trim($_SERVER['HTTP_ACCEPT_LANGUAGE']);
            if ( !$user_http_accept_language ) {
                $contact_form_blacklist_status = ’2′;
                $spamfree_error_code .= ‘ CF-HAL1001′;
                }
*/
            // Add blacklist check – IP’s only though.

            if ( $contact_form_blacklist_status ) {
                $spamfree_contact_form_content = ‘<strong>Your location has been identified as part of a reported spam network. Contact form has been disabled to prevent spam.</strong>’;
                }
            $content_new = str_replace(‘<!–spamfree-contact–>’, $spamfree_contact_form_content, $content);

That’s it. No more false positives for Kim’s visitors!

I’ll send this issues to the WP-SpamFree devs.

Debugging an ASP.Net FBML Facebook Application using Visual Studio 2005

1 Dec

I finally managed to get my setup fine-tuned to debug Facebook FBML applications using Visual Studio 2005. Chances are that the same procedure will work under Visual Studio 2008. Since I struggled a bit to get my setup right, I thought I should share my findings.

I am using Nikhil Kothari’s excellent Facebook.NET API because I think it’s very good for FBML applications. I also tried the Facebook Developer’s Toolkit and I found it easy to use for IFrame applications, but a bit buggy. I also had to make corrections to the source code to improve performances and solve some bugs. In any case the following recipe to debug applications would work with any API.

Getting your PC to serve HTTP request over the Internet

Since Facebook FBML applications work though REST requests that are incoming from Facebook’s servers, you will need to setup your PC to respond to HTTP request from the outside.

The first thing in order, is to setup your rooter and or firewall so that your PC can respond to external HTTP requests. In general, you need to:

  1. Open the HTTP (80) port on your router to forward requests to your PC’s LAN address
  2. Configure your firewall to let HTTP requests through

In my case getting this to work was a bit of a struggle, because my Bell Canada service supplied me with a dreaded SpeedStream 6300 router/modem. To make a long story short, I had to use a second router, plug it in the WAN port and setup the DMZ address to the second router’s address on the SpeedStream.

You will also need to setup a DNS Server so that the name server reaches the public IP address your are connected to. To do this, you either need a fixed IP address and setup a DNS server or need to use a dynamic DNS service. In my case, I’m lucky, I have a fixed IP address and have my own DNS servers.

To test correctly that you can serve external HTTP requests, the following is recommended:

  1. If not already installed, install IIS on your PC (you will need it later, anyway)
  2. Configure IIS to serve a static HTML page, start the IIS service and test your page on localhost
  3. Test your HTML page using your external IP or fully qualified domain name. Here, chances are you will need to test the page from outside the network. Ask a friend to try the URL from a remote location.
  4. As a basic security measure, make sure to configure the IIS service to start manually and start it only when you need it or to change the port forwarding on your router to make sure your PC is not constantly available to respond to external HTTP requests.
  5. If your Web page is not responding, check the IIS logs. It will tell you if your Web server is being hit or not. If it’s not being hit, there’s surely a problem in routing external HTTP requests to your local Web server.
  6. In order to resolve external URLs that point directly to your domain name, you will need to add an entry for your fully qualidfied domain name in windowssystem32etchosts.

Testing your Facebook FBML application

Once you know your local Web server can answer external requests, make sure your Facebook FBML application is working. If you have access to a hosting service (you will need one if you wish to publish your application), publish your application, configure your Facebook Developer account for the application and test it.

Once you know your “Hello World” application works, you can now try and setup you debugging environment.

Debugging the FBML application

To debug a FBML application, you will need to setup your project as a Web Application Project. You might not need to, but I was never able to allow external HTTP requests to be served by the Web server included in Visual Studio 2005, even when configuring it on port 80 and disabling IIS. With Web Application Projects, you can use IIS. You can learn more about Web Application Projects here. To migrate a Web Site Project to a Web Application Project, I recommend this excellent article by Scott Guthrie.

Once your Web Application Project is ready for your Facebook application, go in your project’s properties and set up your project to use IIS:

WebAppFacebookFBML

If you start the debugger for your project you will be asked to log in your application. Instead, go to http://www.facebook.com, log into your account and install your application. You will probably get an error because it did not respond to the installation URL. It does not really matter, if there was an error while installing your application in Facebook, it will be installed anyway for your profile and your will see a link for your application in the left menu. If you browse to your application, you will probably get the following error (if you set up your Web.Config to show errors):

WebAppFacebookFBML2 

If you don’t get this error, but instead get a default Facebook application error, check your IIS logs to see if Facebook is reaching your local copy of IIS. If not, recheck your routing configuration, your firewall and your application’s configuration in the Facebook Developer application.

The above error happens because the debugger is not attached to the right process. Here’s how to solve it:

  1. Stop your debugger
  2. Restart the IIS Service.
  3. Use any browser to browse your application, you should finally see it served from your PC!
  4. Then, start your Visual Studio Debug to attach to the ASP.Net worker process (aspnet_wp.exe) using the “Debug/Attach to process” menu item:

WebAppFacebookFBML3

Redo the above 4 steps anytime you get an HTTP 500 error.

It should now work. If you refresh your application’s page in your browser, it should still show correctly and if you add a breakpoint in your code, you should see that you are debugging Facebook’s requests!

Facebook Developer: how to solve the “Page requested not found” error when developing a Facebook application

10 Sep

I’m developing a Facebook application for fun in my free time and found the solution to a problem that nagged me for a while.

To develop a Facebook application, you use the Facebook Developer application and create a new application. You can even define a localhost application so that you can debug your application locally. Once all the required parameters are well defined for you application and you begin to test the Facebook integration, you may encounter the following error when following you Canvas Page URL:

FacebookDeveloper2

When this was happening to me, my application wasn’t being called at all by Facebook. If you have a real error in your application, your application should be called and you should be able to debug it or you should be at least getting a 404 error from your server.

The solutionFacebookDeveloper1

With trial and error, incomprehension and a lot of swearing, I finally found out that this is a bug in Facebook and pinpointed exactly how it happens for me.

Ensure that the Canvas Page URL and that your Side Nav URL are both in lowercase! There is something in the Facebook folks code that forbids you to add any uppercase letters to the Canvas Page URL and Side Nav URL. So, to be on the safe side, for every URL defined in your application’s settings in Facebook, use lowercase letters!

By the way, it now looks like this problem is now specific to the Side Nav URL being in lowercase. I am pretty sure that there was a similar Facebook error when the Canvas Page URL was in uppercase.

United Nation Web site hacked through amateur SQL injection security hole

13 Aug

You might have heard that the United Nation Web site was hacked Sunday morning. As of Sunday evening was still open to attacks.

What is surprising (or is it?), is that it was easily hacked through a SQL injection attack. If any self-called Web developer were to tell me they don’t know what SQL injection is, I would tell them to change career.

A typical SQL injection attack is made possible when an application does not filter SQL escape characters for character strings. When SQL statements are constructed, string delimiters are inserted in the statement to change the statement or terminate the statement and execute other SQL statements.

For example (typical examples from Wikipedia):

sql = "SELECT * FROM users WHERE name = '" + userName + "';"

This statement checks for a user name (it could also check for a password). As you see, the userName variable is concatenated to the string and surrounded by quotes, the usual literal string escape character in SQL. However, such code is very poor and open to attack because if userName comes from a user input, the user can inject additional quotes by supplying something like:

a' or 1=1

When reconstructed by the deficient code, the SQL statement becomes:

SELECT * FROM users WHERE name = 'a' or 1 = 1;

If that statement was used for some kind of authentication mechanism, it would always be true and the system might open content that was not intended for the current user.

In the case of the UN web site, one of the original Web pages that was hacked can be viewed here.image

The titles for the latest speeches were changed to “Hacked By kerem125 M0sted and Gsy
That is CyberProtest Hey Ýsrail and Usa
dont kill children and other people
Peace for ever
No war”.

The Web site was probably hacked through one of the speeches Web page, which have a URL as follows:

http://www.un.org/apps/news/infocus/sgspeeches/statments_full.asp?statID=105

As you can see, this kind of hyperlink takes the parameter:

statID=105

Well, by simply appending a quote to this URL, anyone was able to access their database with the security permissions of the Web application. So, the hackers were able to insert (or update) different speeches that are supplied from the database by concocting their own SQL statements.

It now seems that a filter has been installed on their server, because if you try this URL, you connection will be reset and you will not reach their Web server. Some basic Web security filter will filter out URLs that seem suspect and URL with quotes in them are rarely used and stopping these can mitigate these sort of attacks.image

This might stop this attack, but if they had data entry forms with similar holes, they would still be open to attack. URL filtering would not stop these. You might put a plaster to try to protect a Web site, but in this case, the plaster has not been applied to other Web sites. For example, at this very moment, other Web sites are open to the same basic amateur holes. For example, the UNEP (United Nations Environment Programme), has also been defaced, probably through the same techniques.

Good security implies different layers of security. At the entry point of the network, you need good firewalls with filtering and intrusion detection systems and fully patched systems. But if the application opens up the database through programming holes, these are not worth much.

What’s more damning for an institution or a company that faces humiliation through such public attacks is that basic programming techniques would easily have avoided these holes in the first place:

  1. For performance and security reasons, it is recommended to use prepared SQL statements instead of dynamically building strings. All modern database engines support prepared statements.
  2. As an alternative to point #1, string inputs could filter escape characters. However, since most databases support multi-byte character sets and different character set settings, it can be quite easy to still leave some character escaping holes opened. If you need to do this instead (you don’t!), you would need a model that ensures that all input strings are correctly filtered. Leaving this task to the Web page developers will certainly leave some holes opened.
  3. There is a debate about using stored procedures, but by parameterizing inputs and forcing types, user inputs can be correctly filtered. Evidently, to avoid most holes, you would still need to use prepared statements to call stored procedures.
  4. A good object-oriented approach will also close some holes. In the example of the UN, the parameter is an integer corresponding to the identifier of a record. A typical entity class would necessitate to use integer properties and lookup methods that only accept integers to search by identifier. Therefore, by using strong types and forcing type conversions, this particular attack would not have happened.
  5. 3-tier architecture will isolate the database access functions in separate layer. It does not stops vulnerabilities, but having the database facing code isolated helps to avoid adding new vulnerabilities.
  6. A model-view controller architectural pattern can further isolate the interface from the database.

In Web applications and Web sites, many more vulnerabilities can be made possible though amateurish programming:

  1. Incorrect use of hidden form fields. In HTML forms, hidden fields in forms can be quite useful. When used by amateurish programmers, they are dangerous. For example, I once stumbled on a shopping cart that used hidden fields for the prices of the products sold. By simply saving the HTML of the page to my desktop and changing the prices, I could have ordered the products with my own user-defined pricing injection! Fortunately for the company, it was before publishing of the web site and they called me to check the Web site. When it took me 5 minutes to find the hole and came back to them with my findings. The next thing I heard was that the developing company was fired and I would doubt they got paid.
  2. Incorrect use of cookies. When a Web application relies on cookies, it must be very careful about what is stored in the cookie. For example, a poorly developed web site could simply store a user id in the cookies for a “remember me” option. Then, a malicious user might change that identifier to something else and even maybe change his access to a user that has administrative access to the application. In that case, storing the identifier directly is a very bad idea. To at least avoid impersonating other users (without access to their computers), a secret random-like key could be stored. Still, someone with access to the computer could still do damage. Because of this, pages that give access to or allow to update sensitive information should be protected by asking the user to be authenticated, independently of a “remember me” option. There are a lot of different strategies to mitigate the risks of cookie usage.
  3. Code injection. Even if your application prevents SQL injection holes, it might still be vulnerable to certain types of code injection attacks if you allow user inputs. For example, with community features like discussion forums, the application needs to filter inputs to avoid Web site defacement through injection of HTML or Javascript code.

Application security is a serious matter and is more complex than what I can go in depth in this blog. I’ve noted a couple of beginner’s mistakes, but there is much more than this.

More often than not, especially for small businesses, Web projects will go to the lowest bidders that are not necessarily professionals in the domain. This approach can be risky. I wonder how much the UN paid for their Web sites… I wouldn’t think they would go to the lowest bidder. But, even when paying the right price, you can get more than what you bargain for. It is important to correctly research the credentials and experience of the developers that will do the coding.

An institution like the UN would at least need to get an independent consultant that know the right questions to ask the developers to verify that they do apply the best practices in their development.

SIMILE Timeline: Manually Adding Events

28 Jun

Introducing SIMILE Timeline’s documentation

A few months ago, I began to use the SIMILE Timeline JavaScript control for a project. It’s really a handy control to visually show – you guessed it – a timeline of events.SIMILETimeline

However, there’s not a lot of documentation for it. As you can see on the documentation page, in pure Open Source spirit, you will see many “for now, read the code” references for parts of the API where the documentation is the source code. There’s also a Wiki documentation page where you can find a bit more information, which is handy to find a few more things, like how to change themes.

Because of this, I though I would write a few posts with a few tips and tricks I discovered. Now, that the introduction is made, let’s start with my first post…

Why would you want to manually add events in the Timeline?

One thing you might want to do at some point, is to use Javascript to manually add events to the Timeline.

There are methods to load data in a Timeline through XML with the Timeline.loadXML method, as described in the How to Create Timelines tutorial. Also, for more efficiency over the wire, you might discover and want to use the Timeline.loadJSON method. So, why would you want to add events manually?

Well, the are many reasons why you might deem this necessary and most of these are for efficiency reasons:

  1. You already loaded the Timeline through XML or JSON, but you want to add new events to the Timeline without refreshing the Web page. You want to use your AJAX skills, don’t you?
  2. You are already using a JSON (or XML) Web service for other things on the Web page and you don’t want to make a second call to load the Timeline.
  3. You want to process your JSON’s Web service data to, for example, generate the HTML for the pop-up through JavaScript from the data instead of sending the whole HTML thru the wire.

How to manually add events in the Timeline?

 It’s not that hard to add an event manually. When browsing the source code, you will see the definition for a new event in sources.js. The constructor is Timeline.DefaultEventSource.Event.  It accepts the following parameters: start, end, latestStart, earliestEnd, instant,text, description, image, link,icon, color, textColor.

Knowing this, here’s some code to generate 50 random events up to 20 days in the past or the future. Compared to the code on the official tutorial page, the important differences are in bold.

<html>
  <head>
    <script src=”http://simile.mit.edu/timeline/api/timeline-api.js” type=”text/javascript”></script>
    <script>
var tl;
function onLoad() {

  var eventSource = new Timeline.DefaultEventSource();
  //Generate 50 random events up to 20 days in the past or the future
  for(var i=0;i<50;i++) {
      var dateEvent = new Date();
      dateEvent.setTime(dateEvent.getTime() + ((Math.floor(Math.random()*41) – 20) * 24 * 60 * 60 * 1000));
      var evt = new Timeline.DefaultEventSource.Event(

         dateEvent, //start

         dateEvent, //end

         dateEvent, //latestStart

         dateEvent, //earliestEnd

         true, //instant

         “Event ” + i, //text

         “Description for Event ” + i //description

      );
      eventSource.add(evt);
  }
  //create the timeline
  var bandInfos = [
    Timeline.createBandInfo({
        trackGap:       0.2,
        width:          "70%",
        intervalUnit:   Timeline.DateTime.DAY,
        intervalPixels: 50,
        eventSource: eventSource
    }),
    Timeline.createBandInfo({
        showEventText:  false,
        trackHeight:    0.5,
        trackGap:       0.2,
        width:          "30%",
        intervalUnit:   Timeline.DateTime.MONTH,
        intervalPixels: 150,
        eventSource: eventSource
    })
  ];
  bandInfos[1].syncWith = 0;
  bandInfos[1].highlight = true;
  tl = Timeline.create(document.getElementById(“my-timeline”), bandInfos);

}

var resizeTimerID = null;
function onResize() {
    if (resizeTimerID == null) {
        resizeTimerID = window.setTimeout(function() {
            resizeTimerID = null;
            tl.layout();
        }, 500);
    }
}
    </script>
  </head>
  <body onload=”onLoad();” onresize=”onResize();”>
     <h1>Creating Events Manually in a SIMILE Timiline</h1>
     <div id=”my-timeline” style=”height: 150px; border: 1px solid #aaa”></div>
  </body>
</html>

You can see a working example here.