BaseX-Talk February 2014

basex-talk@mailman.uni-konstanz.de

43 participants
51 discussions

database corruption
by Michel Banguerski 05 Feb '14

05 Feb '14

Greetings, I am currently using BaseX 7.7.2 on windows 7. Periodically I get a corrupted database which topically manifests itself by the following message: <basex-talk(a)mailman.uni-konstanz.de>Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.7.2 Java: Oracle Corporation, 1.7.0_45 OS: Windows 7, x86 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 33 at org.basex.io.random.TableDiskAccess.flush(TableDiskAccess.java:124) at org.basex.data.DiskData.finishUpdate(DiskData.java:212) at org.basex.core.cmd.Add.run(Add.java:109) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.exec(Command.java:321) at org.basex.core.Command.execute(Command.java:78) at org.basex.server.LocalSession.execute(LocalSession.java:122) at org.basex.server.Session.execute(Session.java:37) at org.basex.core.Main.execute(Main.java:146) at org.basex.core.Main.execute(Main.java:130) at org.basex.core.Main.console(Main.java:90) at org.basex.BaseX.<init>(BaseX.java:168) at org.basex.BaseX.main(BaseX.java:38) Once the database is corrupted there is no means to get the information back. In order to restore a backup I have to manually shutdown running java processes to remove the locks. I was able to narrow the circumstances of the corruption event to the following. - The corruption occurs when adding a resource to the database - The added resource needs to be big enough (one additional characher or xml node makes the difference) - The database has to be in a certain state at the moment of the resource addition for the corruption to happen: the same resource added at a different time will note cause corruption - The database could leave the 'pre-corruption' state if some activity happens I was able to capture a database in 'pre-corruption' state in a backup, the database has no resources (all removed previously) and the backup is 44Kb I also have an couple of XML files that cause corruption of this database when added both are 2kb. I mostly experience the corruption when using db:add() or db:replace() functions in xqm files but in the example I managed to capture the corruption happens when issuing the add command in CLI as well. I can provide the backup and XML files but I suppose that sending them to the list as attachments would not be welcomed. I performed the same test with the same result on Linux: - restaure the 'pre-corrupted' backup - open the database - add the XML file Finally I performed the same tests with BaseX78-20140203.150949 and no corruption happened. I'm seeking the most stable configuration for my application and therefore I would like to have Your opinion: Should I seek a workaround in BaseX 7.7.2 to avoid corruption or should I go with 7.8 in spite it being in beta stage? TIA Best regards Michel

2 1

database corruption
by Michel Banguerski 05 Feb '14

05 Feb '14

Greetings, I am currently using BaseX 7.7.2 on windows 7. Periodically I get a corrupted database which topically manifests itself by the following message: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.7.2 Java: Oracle Corporation, 1.7.0_45 OS: Windows 7, x86 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 33 at org.basex.io.random.TableDiskAccess.flush(TableDiskAccess.java:124) at org.basex.data.DiskData.finishUpdate(DiskData.java:212) at org.basex.core.cmd.Add.run(Add.java:109) at org.basex.core.Command.run(Command.java:345) at org.basex.core.Command.exec(Command.java:321) at org.basex.core.Command.execute(Command.java:78) at org.basex.server.LocalSession.execute(LocalSession.java:122) at org.basex.server.Session.execute(Session.java:37) at org.basex.core.Main.execute(Main.java:146) at org.basex.core.Main.execute(Main.java:130) at org.basex.core.Main.console(Main.java:90) at org.basex.BaseX.<init>(BaseX.java:168) at org.basex.BaseX.main(BaseX.java:38) Once the database is corrupted there is no means to get the information back. In order to restore a backup I have to manually shutdown running java processes to remove the locks. I was able to narrow the circumstances of the corruption event to the following. - The corruption occurs when adding a resource to the database - The added resource needs to be big enough (one additional characher or xml node makes the difference) - The database has to be in a certain state at the moment of the resource addition for the corruption to happen: the same resource added at a different time will note cause corruption - The database could leave the 'pre-corruption' state if some activity happens I was able to capture a database in 'pre-corruption' state in a backup, the database has no resources (all removed previously) and the backup is 44Kb I also have an couple of XML files that cause corruption of this database when added both are 2kb. I mostly experience the corruption when using db:add() or db:replace() functions in xqm files but in the example I managed to capture the corruption happens when issuing the add command in CLI as well. I can provide the backup and XML files but I suppose that sending them to the list as attachments would not be welcomed. I performed the same test with the same result on Linux: - restaure the 'pre-corrupted' backup - open the database - add the XML file Finally I performed the same tests with BaseX78-20140203.150949 and no corruption happened. I'm seeking the most stable configuration for my application and therefore I would like to have Your opinion: Should I seek a workaround in BaseX 7.7.2 to avoid corruption or should I go with 7.8 in spite it being in beta stage? TIA Best regards

1 0

Re: [basex-talk] "Out of Main Memory" error
by Christian Grün 04 Feb '14

04 Feb '14

Hi Geoff, have you assigned enough memory to the JVM on the 48GB machine (via -Xmx)? You may need to edit the BaseX start scripts [1] for that (but it depends on how you starting BaseX). Hope this helps, Christian [1] http://docs.basex.org/wiki/Start_Scripts ____________________________________ > I get an "out of main memory" error when I try an xquery in Basex GUI on a > ~5GB xml database file on an HP 48GB RAM 12 core Z600 workstation. I could > run the query on my 8GB RAM laptop without a problem. Is there some way I > can increase my available memory on my workstation to accommodate large xml > files? > > Thanks. > > Geoff > > -- > > ************************************* > Geoffrey Hougland > 7902 Kara Ct > Greenbelt MD 20770 > 240-600-3730 cell > 301-776-3212 home > ************************************* >

1 0

Opening BaseX in read only mode
by iClays 03 Feb '14

03 Feb '14

Is there a way to open basex in readonly mode? Let's say you have two server pointing same basex, could one open in read mode and the other read/write mode? How could that be achieved if any? Thankx

2 1

function idref() and one-to-many
by Tsvetanka Georgieva-Trifonova 03 Feb '14

03 Feb '14

Thank you for your answer! But this solution does not work with one-to-many relationships. When I run the following script: let $i := <root> <a id = "a1" idref = "b1"> aaa1</a> <a id = "a2" idef = "b1">aaa2</a> <a id = "a3" idef= "b2"> aaa3</a> <a id = "a11" idref = "b2"> aaa11</a> bb1 bb2 </root> return $i/*[matches(@idref, 'a1')] BaseX produces the result: bb1 bb2 that is different from: bb1 Besides, my question is about the function idref()? Is it supported? I find some examples here: http://www.xqueryfunctions.com/xq/fn_idref.html I have defined DTD, schema, but the result is the same. BaseX requires the name of the attribute-identifier to contain “id” and the name of the attribute with references to contain ‘idref’. But the function idref() returns result only if the relationship is one-to-one. I would be grateful if you tell me how to apply the function idref() when the relationship is one-to-many. Best regards, Tsvetanka

2 3

Xquery returns 'Out of Main Memory'
by DAIANE ROBERTA CANDIDA 03 Feb '14

03 Feb '14

Hello, The query below returns 'Out of Main Memory': XQUERY collection("/home/arwen/daiane_oracle/loading_inex2009/xml_pages/")//article[.//text() contains text "Nobel" ftand "prize"] I am using basexclient. I have a collection of xml files that have 14GB size. -------------------------------------------------------------------- > LIST name Resources Size Input Path ---------------------------------------------------------------------- bd_inex2009 730283 16263937902 --------------------------------------------------------------------- I dumped those files by the commands below: ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles1/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles2/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles3/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles4/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles5/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles6/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles7/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles8/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles9/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles10/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles11/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles12/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles13/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles14/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles15/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles16/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles17/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles18/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles19/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles20/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles21/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles22/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles23/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles24/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles25/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles26/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles27/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles28/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles29/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles30/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles31/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles32/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles33/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles34/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles35/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles36/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles37/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles38/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles39/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles40/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles41/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles42/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles43/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles44/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles45/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles46/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles47/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles48/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles49/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles50/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles51/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles52/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles53/; ADD /home/arwen/daiane_oracle/loading_inex2009/xml_pages/xmlfiles54/; ---------------------------------------------------------------------------------------- The command INFO returns: INFO General Information Version: 7.7.2 Used Memory: 79 MB Main Options DBPATH: /home/arwen/BaseXData DEBUG: false EVENTPORT: 1985 GLOBALLOCK: false HOST: localhost HTTPLOCAL: false KEEPALIVE: 0 LANG: English LANGKEYS: false LOG: true LOGMSGMAXLEN: 1000 NONPROXYHOSTS: PARALLEL: 8 PASSWORD: PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/arwen/BaseXRepo RESTXQPATH: SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 0 USER: WEBPATH: /home/arwen/BaseXWeb Options ADDARCHIVES: true ADDCACHE: false ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHECKSTRINGS: true CHOP: true COMPPLAN: true CREATEFILTER: *.xml CREATEONLY: false DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false FTINDEXSPLITSIZE: 0 HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 INDEXSPLITSIZE: 0 INLINELIMIT: 0 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,header=false,jsonml=false,lines=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: STRIPNS: false TAILCALLS: 256 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true ----------------------------------------------------------- How can I run XQuery in that collection? Regards, Daiane

2 1

XML DOCTYPE Problem
by Martin Reckziegel 03 Feb '14

03 Feb '14

Hello everybody, i'm using basex 7.7.2 in a university based project. I'm trying to store TEI XML files in the database but there is an error storing certain valid files. Using a rest PUT request to store a file starting like this: <?xml version="1.0"?> <!DOCTYPE TEI.2 PUBLIC "-//TEI P4//DTD Main DTD Driver File//EN" "http://www.tei-c.org/Guidelines/DTD/tei2.dtd" [ <!ENTITY % TEI.XML "INCLUDE"> <!ENTITY % PersProse PUBLIC "-//Perseus P4//DTD Perseus Prose//EN" "http://www.perseus.tufts.edu/DTD/1.0/PersProse.dtd" > %PersProse; ]> <TEI.2> <teiHeader type="text" status="new"> .... results in this error: > "tlg0003.xml.xml" (Line 5): ']' expected, '<' found. (Line 5 is %PersProse;) I have no clue how to interpret the error since non of the mention characters are in that line. Maybe this is resulting in some internal replacement? Anyway deleting line 5 resolves the error (but of course does not solve my problem since i don't want to alter the files) The problematic files are all valid, at least according to http://www.validome.org/xml/validate/ and http://validator.w3.org/check so i wonder why they are rejected by basex? kind regards, Martin Reckziegel

2 10

Comparision of basex with other xml databases.
by Shalin Makhija 03 Feb '14

03 Feb '14

Hello, I am a newbie to using XML- databases. I am working on "SEMANTIC SEARCH ENGINE" on java that uses a parser to parse the input and generates output in xml. I need to use a xml database for storing all those xml files and fire xpath/xqueries to retrieve my result from java clients. My dilemma is which one to use between BASE-X and EXIST-DB. Both have their own selling points. A W3C test summary reads using base-x is efficient. I will be highly obliged to you if you can give some more concrete points that insist on using base-x so that I can make a fair decision. Thanx in advance. -- *-Regards* *Shalin Makhija*

2 1

function idref() and one-to-many
by Tsvetanka Georgieva-Trifonova 02 Feb '14

02 Feb '14

Hi BaseX, My question is about function idref(). How it should be applied in case of one-to-many relationships? For example: <root> <a id = "a1" idref = "b1"> aaa1</a> <a id = "a2" idef = "b1">aaa2</a> <a id = "a3" idef= "b2"> aaa3</a> bb1 bb2 </root> The expression: idref("a3")/.. gives the following result: bb2 But the expression: idref("a1")/.. does not give any result. Thanks & regards, Tsvetanka

2 1

Introspection bug in BaseX7.8 beta 4cfa54c ?
by jean-marc Mercier 01 Feb '14

01 Feb '14

Hello, As bug https://github.com/BaseXdb/basex/issues/839 has been closed, I tried to upgrade from BaseX7.8 6aeaebf to BaseX beta_4cfa54c. Unfortunately, this bug still appears in 4cfa54c, and I rolled back again to beta 6aeaebf . Did I misunderstoog something ? Cheers

2 8

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk February 2014