{"id":277,"date":"2010-10-31T18:56:05","date_gmt":"2010-10-31T17:56:05","guid":{"rendered":"http:\/\/joernhees.de\/blog\/?p=277"},"modified":"2016-09-28T23:56:08","modified_gmt":"2016-09-28T21:56:08","slug":"setting-up-a-local-dbpedia-mirror-with-virtuoso","status":"publish","type":"post","link":"https:\/\/joernhees.de\/blog\/2010\/10\/31\/setting-up-a-local-dbpedia-mirror-with-virtuoso\/","title":{"rendered":"Setting up a local DBpedia mirror with Virtuoso"},"content":{"rendered":"

Newer version available: Setting up a Linked Data mirror from RDF dumps (DBpedia 2015-04, Freebase, Wikidata, LinkedGeoData, …) with Virtuso 7.2.1 and Docker (optional)<\/a><\/p>\n

So you’re the guy who is allowed to setup a local DBpedia mirror for your work group? OK, today is your lucky day and you’re in the right place. I hope you’ll be able to benefit from my hours of trials and errors \ud83d\ude09 If anything goes wrong, feel free to leave me a comment below.<\/p>\n

Versions<\/h2>\n

There is an updated version of this post for DBpedia 3.7: Setting up a local DBpedia 3.7 mirror with Virtuoso 6.1.5+<\/a>.
\nOriginally this guide was written for DBpedia 3.5.1 and Virtuoso 6.1.2. Later it was updated to DBpedia 3.6 and Virtuoso 6.1.3. The overall process and the main pitfalls are the same, so if you for any reason want to use DBpedia 3.5.1 just make sure to use the correct dumps.<\/p>\n

Prerequisite<\/h2>\n

Before starting you basically need one thing: a powerful server<\/strong> \/ vm with a lot of ram (we used 8 or 4 cores<\/strong> with a total of 32 GB<\/strong> ram, but as you can imagine this strongly depends on what you want to do with your mirror). I’d suggest setting it up with Ubuntu 10.04 LTS<\/strong> but that’s a matter of choice. Anyhow, you’ll need root access.<\/p>\n

A few more words:
\nThe following will largely be bash syntax<\/strong>, sometimes we switch to sql<\/strong>, so comments start with #<\/code> or --<\/code>.
\nShould a command take longer than 5 minutes<\/strong> the approx. times are given.
\nOverall time for this setup (en and de DBpedia 3.5.1) was 7 hours<\/strong>, (en DBpedia 3.6 only took 5 hours) once I had figured out what to do and what not to do. This was tested on Ubuntu 9.10 and 10.04 (both 64-bit) with Virtuoso 06.01.3127.<\/p>\n

Let’s go<\/h2>\n

Download and install virtuoso<\/h3>\n

Go and download virtuoso opensource: http:\/\/sourceforge.net\/projects\/virtuoso\/<\/a>
\nPut the file in your home dir on the server, then do the following.<\/p>\n

sudo aptitude install libxml2-dev libssl-dev autoconf libgraphviz-dev libmagickcore-dev libmagickwand-dev dnsutils\ncd ~\ntar -xvzf virtuoso-*\ncd virtuoso-opensource-6.1.2\nCFLAGS=\"-O2 -m64\"\nexport CFLAGS\n.\/configure --with-layout=debian # NOTICE: this will _not_ install into \/usr\/local but into \/usr (so might clash with packages by your distribution if you install \"the\" virtuoso package)\n# You'll find the db in \/var\/lib\/virtuoso\/db !\n# check output for errors and FIX THEM! (e.g., install missing packages)\nmake -j5\n<\/code><\/pre>\n
This will take about 1 hour<\/strong>. In parallel, you might want to start with downloading the DBpedia files (next section) and come back.<\/p>\n
sudo make install\nsudo mkdir -p \/usr\/local\/data\/dbpedia\/3.6\/en\n<\/code><\/pre>\n
Now change the following values in \/var\/lib\/virtuoso\/db\/virtuoso.ini<\/code>, the performance tuning stuff is according to http:\/\/virtuoso.openlinksw.com\/dataspace\/dav\/wiki\/Main\/VirtRDFPerformanceTuning<\/a>:<\/p>\n
[Parameters]\nDirsAllowed              = ., \/usr\/share\/virtuoso\/vad, \/usr\/local\/data\/dbpedia\n# IMPORTANT: for performance also do this\n[Parameters]\nNumberOfBuffers          = 2400000\n# each buffer caches a 8K page of data and occupies approx. 8700 bytes of memory\n# it's suggested to set this value to 65 % of ram for a db only server\n# so if you have 32 GB of ram: 32*1000^3*0.65\/8700 = 2390804\n# default is 2000 which will use 16 MB ram ;)\n[Database]\nMaxCheckpointRemap = 625000\n# set this to 1\/4th of NumberOfBuffers\n<\/code><\/pre>\n
The next step installs an init-script (autostarts) and starts the virtuoso server. (If you’ve changed directories to edit \/var\/lib\/virtuoso\/db\/virtuoso.ini<\/code>, go back to the virtuoso source dir!):<\/p>\n
sudo cp debian\/init.d \/etc\/init.d\/virtuoso-opensource &&\nsudo chmod a+x \/etc\/init.d\/virtuoso-opensource &&\nsudo bash debian\/virtuoso-opensource.postinst.debhelper\n<\/code><\/pre>\n
Downloading the DBpedia dump files<\/h3>\n
We have decided that we only<\/strong> needed the German<\/strong> and English<\/strong> files (this was for DBpedia 3.5.1, with DBpedia 3.6 we only needed and installed English) in NT format, if you need something different do so. For example if you decide to download the all-languages tar then make sure to exclude the NQ files from the later importing steps. One simple way to do this is to move everything you don’t want to import out of the directory. Also don’t forget to import the dbpedia_3.*.owl file (last step in the script below)!
\nAnother hint: Virtuoso can only import plain (uncompressed) or gzipped files, the DBpedia dumps are bzipped, so you either repack them into gzip format or extract them. On our server the importing procedure was reasonably slower from extracted files than from gzipped ones (ignoring the vast amount of wasted disk space for the extracted files). File access becomes a bottleneck<\/strong> if you have 8 cores idling. This is why I decided on repacking<\/strong> all the files from bz2 to gz. As you can see I do the en and de repacking in parallel, if that’s not suitable for you, feel free to change it. You might also want to change this if you want to do it in parallel to downloading. The repackaging process below took about 1 hour<\/strong> but was worth it in the end. The more CPUs you have, the more you can parallelize this process.<\/p>\n
sudo -i # get root\n# see comment above, you could also get the all_language.tar or another DBpedia version...\nmkdir -p \/usr\/local\/data\/dbpedia\/3.6\/en\ncd \/usr\/local\/data\/dbpedia\/3.6\/en\nwget -r -np -nd -nc -A'*.nt.bz2' http:\/\/downloads.dbpedia.org\/3.6\/en\/\n\n# if you want to save space do this:\nfor i in *.bz2 ; do bzcat $i | gzip --fast > ${i%.bz2}.gz && rm $i ; done &\n# else do:\n#bunzip2 *&\n\nmkdir ..\/de && cd ..\/de\nwget -r -np -nd -nc -A'*.nt.bz2' http:\/\/downloads.dbpedia.org\/3.6\/de\/\n# if you want to save space do this:\nfor i in *.bz2 ; do bzcat $i | gzip --fast > ${i%.bz2}.gz && rm $i ; done &\n# else do:\n#bunzip2 *\n\ncd ..\nwget http:\/\/downloads.dbpedia.org\/3.6\/dbpedia_3.6.owl\n\n# notice that the extraction (and repacking) of *.bz2 takes quite a while (about 1 hour)\n# gzipped data is reasonably packed, but still very fast to access (in contrast to bz2), so maybe this is the best choice.\n<\/code><\/pre>\n
In the mean time<\/strong> let’s start with the DBpedia bulk loading tutorial. We need to setup a few things before we can start, so go on while still downloading & repacking, if virtuoso is running (else go back to the previous section):<\/p>\n
Installing the bulk loader scripts<\/h3>\n
As a reference only: VirtBulkRDFLoaderExampleDbpedia<\/a>
\nA few words about that tutorial: I found it to lack much information, so for this tutorial: USE WHAT’S WRITTEN HERE!
\nWe’ve already set the DirsAllowed param<\/code> in the .ini file and downloaded the files (or go back and continue with the installation of Virtuoso now). Now let’s execute the bulk loading script taken from: 
\nhttp:\/\/virtuoso.openlinksw.com\/dataspace\/dav\/wiki\/Main\/VirtBulkRDFLoaderScript<\/a>.
\nCopy the contents into a plain text file called VirtBulkRDFLoaderScript.vsql<\/code> in your home dir.<\/p>\n
Note: in the following i will assume that your virtuoso isql command is called isql<\/code>. If you’re in lack of such a command it might be called isql-vt<\/code>.<\/em><\/p>\n
cd\n# this will only install all the needed db procedures doing the work later on.\nisql localhost dba dba VirtBulkRDFLoaderScript.vsql\n<\/code><\/pre>\n
Cleaning problematic parts out of the DBpedia dump<\/h3>\n
OK, now a word about the DBpedia dumps:
\nI had problems importing a couple of files in the later steps, in all DBpedia versions i tried, due to URI-lengths of much larger than 1024:<\/p>\n
For DBpedia 3.5.1:<\/p>\n