Data preparation
Worldwide Botanical Knowledge Base project

J. M. Vanel - See also my My diary about computer science - Last update   - back to home page

The data 

The data provided by the Flora of China project were in 2 large MDB files : flora.mdb and general.mdb. Here are the SQL data description : general.sql ,  flora.sql . The "content" field in flora_text table in file flora.mdb contains the taxon descriptions. It has many other fields for each taxon, depending on the value of the category_id  field. The category_id  field in many different tables is described by the fields name and table in the category table in file general.mdb . Here is an XML version of the category table : category.xml .

Data in flora_text table in file flora.mdb :
Field name

key for the taxon
/table/tr/td[2] publication_id key for a publication (article, book, ...)
/table/tr/td[3] category_id specifies the type of content, 12021 for botanical description
/table/tr/td[4] content the botanical description that gets XML-ized


For a given taxon, not all the categories get populated. In fact, only the following categories are filled somewhere :

I got this by the shell command :
grep "<category_id>" flora_text2.xml | sort | uniq

Dowload the latest XML file : .

Getting XML from various sources

I first used a small Java JDBC program, , to get clean XHTML out of the Flora of China Access database.

Now I use MDB Tools to get the database in CSV format:
mdb-export general.mdb category > category.csv
followed by the Perl module XML::CSV .

Here is an example of a script using Perl module XML::CSV:
use XML::CSV;
$csv_obj = XML::CSV->new();

# Configure  CSV parser :
$default_obj_xs = Text::CSV_XS->new({
     'quote_char'  => '"',
     'escape_char' => '"',
     'sep_char'    => ',',
     'binary'      => TRUE
$csv_obj = XML::CSV->new({csv_xs => $default_obj_xs});

$basename = "flora_text2";
$csv_obj->parse_doc( $basename.".csv", {headings => 1});
$csv_obj->print_xml( $basename.".xml");

Natural language processing

FloraParse is our Natural Language parser. It is a parser for classical Floras generating XML markup. It integrates the Wordnet C  library.  WordNet is a semantic dictionary from Princeton University, accessible through Web interface, GUI interface, and a C  library. FloraParse is writen in C++ with Lex and Yacc. FloraParse transforms Natural Language descriptions into an XML format where informations are marked as organe, sub-organ,  geography, etc.
Sample of the current output of FloraParseHere is a small XML sample with its stylesheet displaying it as colored text.

FloraParse records specific WordNet hypernyms: "plant organ", like petal and most plant organs,  "natural object", and another category including corona, pith, stele, lobe, vein that are considered by WordNet as "body part" .
FloraParse could be generalized into a quite generic XMLizer for punctuation-delimited texts.

General information  about the WWBKB Natural Language processing (see also WWBKB diary of 2003-02-11)
FloraParse Source code in on CVS at : CVS web interface at sourceforge
Link to the project page on Logo
The latest version of the XML markup of the Flora of China is in this directory .

Generate organ list

I do this tag list processing apart from the XML database, it is easier. If I had several XML files, I would concatenate all files (or just some typical XML instances) in one file, put opening and closing tag around it . It could be done more elegantly with XML External Parsed Entities. Here is a way of doing the concatenation in Unix shell (use Cygwin if on Windows : ) :
echo '<root>' > ../data.xml
list=`find . ! -type d -name '*.xml'`
cat $list >> ../data.xml
echo '</root>' >> ../data.xml
# obtaining data.xml in parent directory, and then do:
cd ..
saxon7 data.xml example2Schema.xslt > schema.xml
saxon7 schema.xml XMLSchema2query.xslt > XPath-helper.html
after having downloaded example2Schema.xslt, XMLSchema2query.xslt, unique.xslt, XPath-helper.js from:

saxon7 is the script I use for XSLT transformation :
java -Xmx110m -classpath /home/jmv/install/saxon7/saxon7.jar net.sf.saxon.Transform $*

SAXON XSLT processor is here:
SAXON is 100% Java, efficient, allways synchronized with new W3C specs.

XPath-helper.html now contains an alphabetical list of all tags in all the original XML files, whatever the XML nesting level.
Then I manually edit XPath-helper.html to eliminate some non-relevant or rare tags, and the duplicated root tag. Finally I manually insert the relevant part into the squery.xsp page in the Cocoon search engine application.

Adding rubrics from other tables

scientific names, ???, and upper taxon links

In table hu_card we have this:
        <card_taxon>Cycas rumphii</card_taxon>

In table taxon we have this:
        <name>Cycas rumphii</name>

??? to be continued