Open source search engines

Nutch (wikipage) is an open source project that offers a set of tools to setup a search engine. There's a bunch of projects based on Nutch.

Deploying advanced text searches

Following along the line on a recent post, I have decided to use tsearch2 for postgres as a text search solution to my database. Installation can get tricky, especially when dealing with uncommon locales, but once it is up and running it is very flexible, fast, and customizable.

For instance, here is a select statement that I use to fetch data from my tables based on text keywords:

• select name,email,description from mytable, to_tsquery('free&as&in&beer') as q where description_idx @@ q;

First thing to note: tsearch2 will discard 'as' and 'in' from the query, as instructed by a file called locale.stop. This is interesting to obtain more accurate results out of the search request. Also the dictionary handles the way similar words are grouped in indexes, thus one can say that both 'beers' and beering' are indexed as 'beer' hence refining the expected search results. This is all part of the dictionary setup, which one should setup properly if intends to have a complete tsearch2 installation.

Tsearch2 comes with C and russian locales after a fresh install, so you'll likely run into problems if your system's locale is set to something different, or your database was created using a different encoding (something that cannot be changed afterwards, unless you dump your database, recreate it using a new encoding, and dump data back in). A typical example of such a problem is this message returned by a tsearch2 query:

• ERROR: could not find tsearch config by locale

As I discussed recently on another post, the prefered way of encoding your data in the backend is Unicode. So in my case I decided to create a utf-8 dictionary, tell postgres that's the default encoding/decoding from the database, then force tsearch2 to use this one instead of its default 'english'. There are prebuilt dictionaries here.

Download the Wikipedia

I randomly came across a page on the wikipedia which tells me that it is possible to download it on your local computer. All at once in one single file. Yes! It is available in a variety of digital formats. SQL dumps, html zipped packages (in 7zip format), and soon to be at the reach of everyone, in dvd format.

How cool would it be to release the wikipedia in DVD, updated every month, at the reach of everybody through a specially crafted dvd edition "wiki-pe-dvd", every Sunday of each month, at your local news shop!

Auditing Postgres activity from HTTP users

The goal of auditing who is updating what data in a postgres database is coming now to a need.

The idea is having an audit of all changes into any of the tables in a postgres database. The solution I need should be independent of wether the frontend is a web application or the command-line or a remote ssh connection. The extra trick is where to grab the user identification when updates come through the web application.

There is a gret Lorenzo has come with a very nice approach I still have to try. It is based on triggers applied to each of the tables you want to audit, and a global function which gets called on every update, insert or delete.

When the frontend is the web application, the design must allow for a user authentication, then the database connection needs to be done on behalf of that user. That way, the postgres procedures will already know who the user is.

Python dates and times

There are currently three ways of dealing with dates and times from within python. time, datetime and mxDateTime (a third party library).

time and datetime are to me a bit confusing and poor intuitive. You can find a good tutorial here.

Contrary to that I find mxDateTime much richer and straightforward. The project has a comprehensive documentation on pdf as well.

SSH Keys Authentication

I have come accross it a thousand times, and I still forget some key details when I try to set it up again, so I will post it here for everyone. It is very convenient to use ssh with key authentication. It saves lots! of typing. The following example is for version 2 DSA keys.

Step 1: From the client system generate your dsa keys

• ssh-keygen -t dsa (no passphrase)
  

Step 2: Let the server give you authorization for your key authentication

• copy your local public key into the server's "authorized_keys" file, in you homedir's .ssh directory
  

Step 3: Make sure your client is using the newly generated key (if you accepted the default key filename (id_dsa) this shouldn't be necessary)

• ssh -i my_private_key_file me@myhost, or...
• add this to your ~/.ssh/config file:
• "Host mysshservername" and "IdentityFile my_key_filename"
  

That's it!

Who am I?

The linux kernel has a wonderful piece of magic called the virtual proc filesystem. It is basically a directory on which the kernel dumps a bunch of low level system details.

It is also an interface for the applications to request special actions be taken by the kernel (these are called sysctl calls). For instance, enabling a running kernel to activate network routing between two network interfaces can be controlled by the file /proc/sys/net/ipv4/ip_forward.

I am currently being faced by a challenge. The organization I am working at the moment has a bunch of linux servers in their network. There are lots of variants of the linux kernel, customizations, distributions, let alone the speed at which they get updated by many different sysops, it is virtually impossible to keep track of what is running on each of those systems.

I am in the process of writing a tool which based on the /proc filesystem will collect as much data as possible from the system to come up with a report.

The linux kernel has an excellent document about the proc filesystem (Documents/filesystems/proc.txt). I can't seem to find the latest version online, it seems it is not being maintained any longer.

The lspci is also another valuable tool which combined with the -vv flag emits just everything that the PCI devices have to say from their eproms.

use cases

Alistair cockburn's excellent tutorial on use cases.

Web pages that suck

http://www.webpagesthatsuck.com

Internet Trends

I read three interesting threads on current state of Internet trends:

• http://it.slashdot.org/article.pl?sid=07/08/01/1938215
• http://slashdot.org/article.pl?sid=07/08/01/1822241
• http://books.slashdot.org/article.pl?sid=07/08/01/149236