Hierarchichal representation under Postgres

Oracle has implemented hierarchical representation using the non-sql conformant "connect by" clause. Postgres has declined to accept this implementation and the team is currently working on a better approach.

In the meantime, the contrib folder contains a somewhat similar implementation to that of Oracle's, the connectby() function.

On-the-fly Language Translation

In my project I am in need of a system to dynamically translate natural text passages into multiple languages (basically and MT, or CAT system). Since the content source is not known, the first thing I need is a language identifier tool. There are many available tools (try this cool online version from Xerox).

I tried text_cat, which is written in Perl, and is used by spamassassin. The results where disappointing to me when I found out confusing results between Spanish and Catalan, even when using ñ, so I need to invest a bit more research on that.
Once the original text language has been identified, I need a translation service. Altavista's Babelfish is probably the most common one, but the programmatic interface is limited (although a Python-based client is fairly easy), it would demand too much bandwidth, and there is a limit on the number of translation requests per client on a given timeframe.

Google's language tools, and google toolbar offer an interface to translating text to multiple languages (be it complete websites, or text passages).

The Spanish newspaper 'el Periodico' uses some propietary software to publish their daily editions in both Catalan and Spanish.

Systran is a commercial solution which offers online and local translation services, it's currently being used by Altavista, Google and many others (Try translating some difficult, slangy passage using Babelfish then Systran and see for yourself).

Since there are a lot of different resources available, I will leave this post as an introduction to this fascinating subject, until my next writing with hopefully some clear and promising results.

Open source search engines

Nutch (wikipage) is an open source project that offers a set of tools to setup a search engine. There's a bunch of projects based on Nutch.

Deploying advanced text searches

Following along the line on a recent post, I have decided to use tsearch2 for postgres as a text search solution to my database. Installation can get tricky, especially when dealing with uncommon locales, but once it is up and running it is very flexible, fast, and customizable.

For instance, here is a select statement that I use to fetch data from my tables based on text keywords:

• select name,email,description from mytable, to_tsquery('free&as&in&beer') as q where description_idx @@ q;

First thing to note: tsearch2 will discard 'as' and 'in' from the query, as instructed by a file called locale.stop. This is interesting to obtain more accurate results out of the search request. Also the dictionary handles the way similar words are grouped in indexes, thus one can say that both 'beers' and beering' are indexed as 'beer' hence refining the expected search results. This is all part of the dictionary setup, which one should setup properly if intends to have a complete tsearch2 installation.

Tsearch2 comes with C and russian locales after a fresh install, so you'll likely run into problems if your system's locale is set to something different, or your database was created using a different encoding (something that cannot be changed afterwards, unless you dump your database, recreate it using a new encoding, and dump data back in). A typical example of such a problem is this message returned by a tsearch2 query:

• ERROR: could not find tsearch config by locale

As I discussed recently on another post, the prefered way of encoding your data in the backend is Unicode. So in my case I decided to create a utf-8 dictionary, tell postgres that's the default encoding/decoding from the database, then force tsearch2 to use this one instead of its default 'english'. There are prebuilt dictionaries here.

Download the Wikipedia

I randomly came across a page on the wikipedia which tells me that it is possible to download it on your local computer. All at once in one single file. Yes! It is available in a variety of digital formats. SQL dumps, html zipped packages (in 7zip format), and soon to be at the reach of everyone, in dvd format.

How cool would it be to release the wikipedia in DVD, updated every month, at the reach of everybody through a specially crafted dvd edition "wiki-pe-dvd", every Sunday of each month, at your local news shop!

Auditing Postgres activity from HTTP users

The goal of auditing who is updating what data in a postgres database is coming now to a need.

The idea is having an audit of all changes into any of the tables in a postgres database. The solution I need should be independent of wether the frontend is a web application or the command-line or a remote ssh connection. The extra trick is where to grab the user identification when updates come through the web application.

There is a gret Lorenzo has come with a very nice approach I still have to try. It is based on triggers applied to each of the tables you want to audit, and a global function which gets called on every update, insert or delete.

When the frontend is the web application, the design must allow for a user authentication, then the database connection needs to be done on behalf of that user. That way, the postgres procedures will already know who the user is.

Python dates and times

There are currently three ways of dealing with dates and times from within python. time, datetime and mxDateTime (a third party library).

time and datetime are to me a bit confusing and poor intuitive. You can find a good tutorial here.

Contrary to that I find mxDateTime much richer and straightforward. The project has a comprehensive documentation on pdf as well.

SSH Keys Authentication

I have come accross it a thousand times, and I still forget some key details when I try to set it up again, so I will post it here for everyone. It is very convenient to use ssh with key authentication. It saves lots! of typing. The following example is for version 2 DSA keys.

Step 1: From the client system generate your dsa keys

• ssh-keygen -t dsa (no passphrase)
  

Step 2: Let the server give you authorization for your key authentication

• copy your local public key into the server's "authorized_keys" file, in you homedir's .ssh directory
  

Step 3: Make sure your client is using the newly generated key (if you accepted the default key filename (id_dsa) this shouldn't be necessary)

• ssh -i my_private_key_file me@myhost, or...
• add this to your ~/.ssh/config file:
• "Host mysshservername" and "IdentityFile my_key_filename"
  

That's it!

Who am I?

The linux kernel has a wonderful piece of magic called the virtual proc filesystem. It is basically a directory on which the kernel dumps a bunch of low level system details.

It is also an interface for the applications to request special actions be taken by the kernel (these are called sysctl calls). For instance, enabling a running kernel to activate network routing between two network interfaces can be controlled by the file /proc/sys/net/ipv4/ip_forward.

I am currently being faced by a challenge. The organization I am working at the moment has a bunch of linux servers in their network. There are lots of variants of the linux kernel, customizations, distributions, let alone the speed at which they get updated by many different sysops, it is virtually impossible to keep track of what is running on each of those systems.

I am in the process of writing a tool which based on the /proc filesystem will collect as much data as possible from the system to come up with a report.

The linux kernel has an excellent document about the proc filesystem (Documents/filesystems/proc.txt). I can't seem to find the latest version online, it seems it is not being maintained any longer.

The lspci is also another valuable tool which combined with the -vv flag emits just everything that the PCI devices have to say from their eproms.

use cases

Alistair cockburn's excellent tutorial on use cases.

Web pages that suck

http://www.webpagesthatsuck.com

Internet Trends

I read three interesting threads on current state of Internet trends:

• http://it.slashdot.org/article.pl?sid=07/08/01/1938215
• http://slashdot.org/article.pl?sid=07/08/01/1822241
• http://books.slashdot.org/article.pl?sid=07/08/01/149236

HTMLHelp is down!

htmlhelp.org and htmlhelp.com (both owned by Web Design Group in Texas, US) are currently down for the last 4 days. What a pest!

I got addicted to start.gotapi.com as a development aid for its clear and agile interface, but it's now useless as it (apparently) feeds all tagging and CSS syntax help in real-time from htmlhelp's website.

I have also received a reply back from the sysadmin of that site and it turns out the problem is related to their service provider's data center (read more). It will hopefully be back online soon.

Linux routing (Debian)

I had to setup a debian multihomed box as a router, so here goes the receipt. In my example the internal network is 10.1.1.0 and the as its IP address is set to 10.1.1.1, you should direct your clients to use that as the gateway. eth0 is connected to the outside network.

• Add network interface in file /etc/network/interfaces:

1. auto eth1
2. iface eth1 inet static
3. address 10.1.1.1
4. netmask 255.255.255.0
5. broadcast 10.0.0.255

• echo 1 > /proc/sys/net/ipv4/ip_forward
• route add -net 10.1.1.0 netmask 255.255.255.0 dev eth1
• iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
• iptables --append FORWARD --in-interface eth1 -j ACCEPT

That should be it. Note that only the iptables and echo statements should be automated through init.d scripts.

Where do people go?

It's not very reliable, but give it a try.

Postgres Text Searches

Sounds easy, but what is the best design to implement a keyword search function on a very big database table's column?

Postgres' pattern matching functions look very powerful, but I am concerned about performance issues here, and it turns out I'm not the only one facing this problem, of course.

Duck suggests using a second index table (id, keyword) with a row for each useful keyword that links back to the main table, and momjian came across an implementation using triggers, although that was 7 years ago now.

Another solution is called TSearch2. This implementation seems to be heavily based on Duck's suggestions and it's bundled into Postgres' official distribution as of version 7.4. It has to be manually installed from sources (from contrib/tsearch2 make && make install, then dump tsearch2.sql into your database). There's a good review on tsearch2 here.

On Debian:

• apt-get install postgresql-contrib-7.4
• psql mydb < /usr/share/postgres/7.4/contrib/tsearch2.sql (as postgres)
• 4 new tables should be created. Set proper permissions on them
• try: SELECT set_curcfg('default') ;
• then, try: SELECT to_tsvector('I live in Sydney, Australia.')

Read this if you intend on using TSearch2 on Postgres up to 7.4.8, as the original version comes with a highly dangerous security flaw that needs to be manually patched.

International Web

While working on web development, I got stuck for a few hours trying to uncover the magic behind Internationalization, and how I am supposed to deal with it from the client (the web browser) down to the database backend engine, through the code that handles the application logic.

Some basic definitions worth having clear:

• Unicode is not an encoding. It is a system to represent any character in any writing system in use today: go here and ask your browser what encoding this page is using.
• Localized web applications might use any of the many character encoding systems: for instance, this Spanish website uses ISO8859-1.
• UTF-8 is a Unicode encoding system and is the preferred way of sending data back and forth from web browsers for its widespread use.

The safest and easiest approach then, seems to rely on the following rules:

• In your application, decode the data to Unicode as soon as it gets to your hands, and work with it all throughout your application processing (including Database backends).
  
• Once your output data is ready, encode it into UTF-8 and send it back to the web browser client as such.
  

Read this great tutorial from Python's Pylons website.

More Python Pylons

More pylons key information sources:

• a very useful cookbook
• first pylons book (to be released online by end of July 2007)
• Dave Kuhlman's quick site tutorial
  

Programmer's Editor

Looking at a pylons screencast I discovered ActiveStates' Komodo, a very advanced programmer's editor. It's free software and available on multiple platforms.

IELTS

the International English Language Testing System is a worldwide certificate of proof that you are proficient in the English language. It has branches all over the world.

Attending the test takes one day for the reading, writing and listening parts, and an extra day for the speaking part, which takes only about 11 to 15 minutes. Fee in Australia is $300.

There are sample tests available.

Australian Migration Agent

An Australian Migration Agent is a person or institution who helps, informs and supports individuals who need a Visa on visiting Australia. Mara is the official authority for registering and keeping a centralized record for Individual (not company-based) Agents.

Searching through their register, reveals the following figures:

55 agents in all United Kingdom
13 agents in London
No agents in Madrid or Barcelona
1 agent in Hong Kong
10 agents in Singapore

Mes receptes de cuina

http://www.mercacarn.com/

Patates estofades

Comensals

• 4 persones

Temps de preparacio

• 1 hora

Ingredients

• 1 quilo de patates
• 50 grs.d'avellanes i ametlles torrades
• 1 ceba gran
• 2 grans d'all
• 2 tomaquets mitjans
• oli
• sal
• julivert
• pebre vermell dolç
• Brou de verdures o aigua

Preparacio

Posar una cassola al foc amb oli i afegir-hi la ceba tallada fina.
Quan la ceba comenci a enrossir afegir l'all picat i, tot seguit, els tomaquets sense pell i tallats a trossos.

Deixar sofregir una mica el tomaquet i afegir una cullerada de pebre vermell remenant be i ràpid perquè no es cremi i enretirar la cassola del foc. Afegir les patates pelades i tallades a rodanxes o escantellades, tornar la cassola al foc i remenar les patates fins que agafin color. Cobrir amb el brou de verdures, afegir sal i pebre i deixar coure fins que les patates siguin toves.

Mentre acaba la coció, en un morter, preparar una picada amb les avellanes, les ametlles, el julivert i una mica del suc de cocció.
Afegir la picada al guisat i servir.

Comentari del cuiner

Escantellar les patates vol dir tallar-les a trossos irregulars i mig trencats, és a dir, que el ganivet no acabi de tallar el tros sinò que, a mig tall, s'estira de costat i es trenca.

Tallant-les així s'aconsegueix millor unió de sabors del sofregit i la patata i millor textura de la salsa.

Yet another CMS system

Skeletonz seems to be a promising CMS system for its easy and compact design.

Unfortunately there is no postgres support, but since it relies on amidb, and amidb has postgres support, it should be easy to implement it (basically by providing a wrapper around cursor.py that interfaces to psycopg2).

These websites are powered by skeletonz:

• http://www.chloecello.net/Cours%20de%20violoncelle/

The first caveat is that Internet Explorer cannot be used for managing Skeleton's content since it relies on a Firefox add-on, still have to research a workaround on this.

Goobdye table tags!

After realizing that html table tags are a thing from the past in terms of layout design, I searched for some good, very basic design tutorial using CSS style sheets.

Carmen Mardiros has an excellent step-by-step friendly website that shows you how to use CSS to design your website layout.

Maxdesign has a very good set of tutorials for designing layouts using different CSS techniques.

You will sooner or later bump into IE nastiness, as many of the W3C standards are not respected. One of a such is the div section box expansion problem.

A brief list of sites which offer CSS layouts and tutorials:

• http://blog.html.it/layoutgala/
• http://tutorials.alsacreations.com/modeles/
• http://pmob.co.uk/
• http://www.glish.com/css/

Firefox Add-ons

I recently came across a Firefox add-on for developers, it's a a toolbar that sits on your browser's menu which helps you unassemble a website into forms, fields, areas, etc. A must have if you are developing websites!

Surfing with anonymity

Tor and JanusVM

The Tor project is devoted to provide a free censorship-less data channel in the Internet. It is composed of a growing number of transit servers. When a user wishes to initiate an anonymous connection to a server, the Tor network creates a connection highway from the client to the server using a randomly, dynamically changed group of servers that setup the connection channel.

This allows the client to surf the Internet with anonymity: The server logs will never have your actual IP on their log files, and your ISP will never know your final destination. Each connected channel is re-established every few minutes using a different route. Tor does not provide for anonymity if you login or provide personal identification to the target system, of course.

You can picture it clearly in this diagram. Using the network as a client is free. If you want to extend the network you can also offer some of your bandwidth to the network and be part of the server farm.

JanusVM is a virtual machine based on VMWare which eases the setup and installation of the software. It is also freely available through the use of VMWare's free Server Edition product.

Spanish Lessons

The Instituto Cervantes is seeing Brasil as the new focus for Spanish schools. Spanish is the second national language in Brasil.

Instituto Cervantes has recently opened a branch in Sydney - Australia.

CMS Systems

Drupal, Joomla

I have been playing a bit with Drupal on my test server. I decided to go with Postgres instead of mysql, and overall it has been quite easy to setup:

1. Downloaded Drupal 5.1
2. Installed php4, postgres, php4-postgres
3. Created postgres user and database, customized drupal's correct dburi.
   
4. Configured a virtual host on apache and hit the main page

Once you successfully hit the main page, Drupal will invoke install.php to create database tables, setup the environment and get ready to setup the website.

A bunch of Drupal themes are available:

• Drupal's homesite themes
• Siteground themes
• Theme garden
  

Tango Resources

http://www.guitarfestivalmikulov.com/GFM06/bandoneon.htm

Graphic designs

A collection of 3dstudio renders:

• http://personal.auna.com/electric

Webs, Wikis, and Shared collaboration

I am looking for a wiki engine flexible enough to build a very customized website while allowing the owner to modify the contents easily. I want it to be python based and of course as independent as possible from libraries and third-party engines. I will have linux underneath with apache.

A first list reveals lots of options. MoinMoin seems to be the mostly spread option.

This website is being served by MoinMoin, and the author explains how handy it becomes to customize it to your needs.

Now, to start hands-on with customizing MoinMoin to drive a regular websites, you want to look at the following things:

• learn themes to customize your look
• setup a wiki farm (some examples) if you are running a virtual web server
• administer user accounts, authentication and access control

There is a minimalistic yet very powerful web framework called web.py. I love its simplicity, but pay attention when setting it up behind your apache, it's tricky!

Unwired

One of the wireless internet connection option in australia is called Unwired. A modem (known as wabbit in the jerk community) operates on the 3.5GHz spectrum.

There's a good forum related to this an other internet providers throughout australia, here.

It is possible to improve reception on Unwired modems by using an external antenna

Australia tax statements

Resources:

• You can do it online (download e-tax software, windows only): http://ato.gov.au/
• The medicare tax statement: http://www.medicareaustralia.gov.au/yourhealth/our_services/medicare/mc_tax_statement/index.htm
• Deductions checklist: http://ato.gov.au/individuals/pathway.asp?pc=001/002/013&mfp=001&mnu=45#001_002_013

Medicare: The government is interested in you paying for a private health insurance. If you are not paying for either one of those nor Medicare, you might get taxed higher than if you were paying to Medicare. Watch out!

Looking for home made antennas to a usb wifi adapter I have found this cool website from a group of kiwis. Don't forget to track your reception with NetStumbler

USB cable extensions come in passive and active forms. Passive cables should not exceed 16ft (4.8 meters). Active cables come with a repeater and some with a builtin buffer.

Winlogon and Gina

I read in the wikipedia about Winlogon, and following links I get to know there's an open source project on a Gina module called pGina. It offers a plugin architecture.

Meanwhile, Vista seems to have killed msgina, and it's offering a complete rewrite towards a new framework to extend a similar architecture to that of pGina, the so called Credential Providers. CPs have the opportunity to provide GUI objects like domain lists, titles, notices, entry fields. The user logon "experience" is visually seamless amongst multiple CPs as the interactive GUI objects are all rendered by Vista's new framework.

Linux Infrared

Two different approaches to join the Infrared world from linux :

• The IRDA project offers linux kernel drivers (including support for some usb based infrared transceivers), and a set of user level tools. It is included in the 2.2.x and 2.4.x linux kernels.
  
• The LIRC project is based on hand-made devices using serial, parallel, and sound ports. It covers many more scenarious while it lacks support for usb based transceivers. If you want to hack around with hardware and build your own infrared devices, this is your way to go.
  

It is possible to sniff infrared traffic, for the adventorous ones.

This magic hardware gadget is an Infrared to/from Sound converter. You sniff on an infrared device (your TV controller), save the converted sound file in an iPod, and when you hit play, the gadget (which plugs into your Earphone jack) converts the sound file back into its original Infrared signal. cool!

This is an excellent source to many IR related links.

Computer fairs

http://www.computerfairs.com.au/

Google Web tools

• Google analytics: https://www.google.com/analytics/home/provision
• Google webmaster tools: https://www.google.com/webmasters

HTML tricks

• How to refresh a webpage: <meta http-equiv="refresh" content="5">
• How to create a redirect page: Use same meta tag with uri after seconds specified.

HTML and JavaScript

I am collecting a set of sources for html related documents for the web designer, so here goes a list:

• http://www.w3.org is the World Wide Consortium on html standards.
• Htmlhelp offers html reference guides packed in html form.
• Here you can find quick reference cards on multiple html technologies.
• This pdf quickguide nicely shrinks most hmtl elements in a 2-page doc.
• A nice tutorial on how to embed Javascript into your html.
• Interactive forms which change layout based on user input are explained here.
• w3c guide on how to properly design and organize Web documents.
• An interactive multi-technology reference site: http://start.gotapi.com
• More on Web forms here.

Image Spamming

I came accross an article which explains the various new techniques used by spammers these days. It is amazing to see how the spam-antispam war it's a never ending story which just climbs up to higer and higher levels of sophistication.

Worms and anti-virus: The fight

Today I have been approached by a challenge to intercept and control win32 file system operations at the file system level. I have been roughly thinking about it and have so far come accross a few resources to help me refresh the scenario, that I will expose here:

The now defunct Phrack site is a good resource to look at. This article shows a way of hooking into the native api from kernel space. In this other article techniques are explained on how to install malicious code from userland space. This other one details techniques on how to surpass registry and file system operations by mangling with kernel land code. Fooling stack backtrace detectors (used in some intrussion detection systems) is exlained here.

Vidocq

This morning I've been watching Vidocq, a 2001 movie starring Gerard Depardieu. The movie is quite fantastic with exciting visual effects. It was the first movie filmed on digital media from start to end. Overall the movie is maybe a bit too heavy on special-effects, but it depicts a magical fabulous story in an absoribng way, about a monster called the 'Alchemist' who steals souls from virgin girls to keep his eternal youth. Vidocq is a detective working for the french police in the help to unmask the alchemist.

Vidocq's complete name was Eugène François Vidocq and was a french criminal with an amazing life, who became a detective for the french police. Amongst many of his stunning chapters throughout his life, he managed to escape prison endless times, survived 15 duels, and he also gained amnesty from the french police when he was challenged to escape from the guards and present himself to the inspector as a sign of honesty.

Life and miracles of Vidocq probably inspired Victor Hugo as well on his novel 'Les Miserables'.

International Worker's Day

I have been surprised to realize that May the 1st is not a public holiday in Australia. Most European countries celebrate that day as a rememorance on the achievements gained by the international labor movement, which ended up in the establishment of a regulated 8-hours work day.The May Day in Australia is not a public holiday, and it is not very clear as to wether the first Monday in October is celebtrated as such.

SARA DANE

A few days back as I was going back home I bumped into a huge pile of random books (it happens to me quite often in Sydney). As usual, I took a closer look at some of them to see if I randomly found any interesting surprises.

Surprisingly enough, I opened a green old thick book and voila! It was written in Spanish. Interesting, I thought. A closer look revealed to me it had been printed in Barcelona in 1958, first edition. It even kept the price ticket on the first page, a thin brownish piece of papirus the deeply reminded me those very old books from my grandparents. 100 pts (about $1 aud)

The book's author is Catherine Gaskin, surprisingly enough I discovered through a friend of mine a few days after, it is signed by herself. The novel's title is 'Sara Dane', whose real name was Mary Rieber and it's the real story about a woman who was convicted in England and expatriated to Australia the late years of the 18th Century to one of the many prisons setup during those times around Botany Bay, and the sudden change to which she succeeded in being a socially respected and prosper woman.

Catherine Gaskin was Irish, she moved to Australia when she was 3 months young and was educated in Sydney. Travelled throughout many corners on the glove. She died just last year, May 2006 in the USA. Amongst many other novels, she wrote 'the summer of the spanish woman'.

I enjoy reading the book these days, makes we wonder how fast things have radically changed in this country from the first seen colonies not so far behind in time. The novel has also been moved on a TV series on 1982. My wife's primary school had devoted mentions and studies about Sara Dane as well.