BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5250 discussions

Out of Main Memory - Inconsistent DB
by Martín Ferrari 14 Sep '15

14 Sep '15

Hi all, I think one of our DB seems to have gotten corrupt. I wonder if there's a command that will recover it. Issuing REPLACE or OPTIMIZE on it will result in "Out of Main Memory" errors. We have DBs that occupy more space and have more documents, so I assume the problem must be that this DB broke somehow. I've tried raising the -Xmx parameter to 1024m, but still get the same "Out of Main Memory" error. Thanks, Martín. INSPECT Checking main table (11292441 nodes):- 0 invalid node kinds- 0 invalid parent references- 87 wrong parent/descendant relationships (pre: 10932445,..)Warning: Database is inconsistent. INFO DB Database Properties Name: OrderInquiryV2_1 Size: 2138 MB Nodes: 11292441 Documents: 251980 Binaries: 0 Timestamp: 2015-09-10T16:40:29.000Z Resource Properties Timestamp: 2015-09-08T23:11:28.497Z Encoding: UTF-8 CHOP: false Indexes Up-to-date: false TEXTINDEX: false ATTRINDEX: false FTINDEX: false LANGUAGE: English STEMMING: false CASESENS: false DIACRITICS: false STOPWORDS: UPDINDEX: false AUTOOPTIMIZE: false MAXCATS: 100 MAXLEN: 96

2 1

Re: [basex-talk] size on collection in the time (Marc)
by cmarchand＠oxiane.com 14 Sep '15

14 Sep '15

I confirm the problem : on another database (same data, same script.xbs, different server), we have exactly the same problem : the database grows each time the script is run. I'm going to prepare a test-case with small public data tonight. Best regards, Christophe Hello Dirk, We use a BaseX script (.xbs) with a list of REPLACE. We try to investigate this point if this is reproducable or not. If I find this problem again, I take a copy of the collection before to delete her. Thank you for answer. Marc Le 03/09/2015 09:52, Dirk Kirsten a ?crit : > Hello Marc, > > sure, a replace should lead to a true replace, i.e. the old data will be > dropped. So this sounds like a bug, but it is difficult to replicate > from what you describe. > > Do you think it would be possible to make the issue reproducable, maybe > also with a smaller data set? Also, how exactly do you add documents or > do you also do any queries and maybe other updating queries? > > Cheers > Dirk > > On 09/02/2015 05:36 PM, Marc wrote: > >> Hi, >> I have a remark about the size of the files of a collection. >> We do a lot of updates (REPLACE) with an attributes index but without >> a text index. >> I see that the size of the files grows up in the time we arrive until >> 30Go until the limit of the filessytems quota. >> When I drop the collection and recreate it from the last version of >> the files the size was only 6Go. >> >> We will redo the test with the time, but I just want to know if the >> program clears the disk use when he does a REPLACE? >> >> Marc ------------------------------ Message: 4 Date: Thu, 3 Sep 2015 10:01:05 +0200 From: Marc <Marc.Liste(a)free.fr> To: Dirk Kirsten <dk(a)basex.org> Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> Subject: Re: [basex-talk] (no subject) Message-ID: <55E7FE41.5080205(a)free.fr> Content-Type: text/plain; charset=iso-8859-15; format=flowed I forgot to give this precision, we use the 7.9 version on this server. Marc Le 01/01/1970 01:00, a ?crit : > Hello Dirk, > We use a BaseX script (.xbs) with a list of REPLACE. > We try to investigate this point if this is reproducable or not. > If I find this problem again, I take a copy of the collection before to > delete her. > > Thank you for answer. > > Marc > Le 03/09/2015 09:52, Dirk Kirsten a ?crit : Hello Marc, > > sure, a replace should lead to a true replace, i.e. the old data will be > dropped. So this sounds like a bug, but it is difficult to replicate > from what you describe. > > Do you think it would be possible to make the issue reproducable, maybe > also with a smaller data set? Also, how exactly do you add documents or > do you also do any queries and maybe other updating queries? > > Cheers > Dirk > > On 09/02/2015 05:36 PM, Marc wrote: Hi, > I have a remark about the size of the files of a collection. > We do a lot of updates (REPLACE) with an attributes index but without > a text index. > I see that the size of the files grows up in the time we arrive until > 30Go until the limit of the filessytems quota. > When I drop the collection and recreate it from the last version of > the files the size was only 6Go. > > We will redo the test with the time, but I just want to know if the > program clears the disk use when he does a REPLACE? > > Marc End of BaseX-Talk Digest, Vol 69, Issue 5 *****************************************

2 1

size on collection in the time
by Marc 14 Sep '15

14 Sep '15

Hi, I have a remark about the size of the files of a collection. We do a lot of updates (REPLACE) with an attributes index but without a text index. I see that the size of the files grows up in the time we arrive until 30Go until the limit of the filessytems quota. When I drop the collection and recreate it from the last version of the files the size was only 6Go. We will redo the test with the time, but I just want to know if the program clears the disk use when he does a REPLACE? Marc

3 4

Schematron validation
by Lizzi, Vincent 14 Sep '15

14 Sep '15

I have just released a module that provides an easy way to use Schematron in BaseX. It's is available on GitHub at https://github.com/vincentml/schematron-basex. This module is a convenience wrapper around the standard ISO Schematron XSLTs. In addition, a version of this module for eXist-db is available at https://github.com/vincentml/schematron-exist Vincent "Everything should be made as simple as possible, but not simpler."

2 1

Fastest way to get resources starting with path
by Martín Ferrari 14 Sep '15

14 Sep '15

Hello: I wonder if there's a fast way to get the contents or names of the resources that start with a certain path. We have something like this: string query = @"for $doc in collection() where substring-after(document-uri($doc), '/') = '" + documentName + @"' return $doc/Root"; But our db has millions of resources so this doesn't scale too well. I'd like a way to get the contents (or names) of resources in which the names start with a certain pattern without having to check all names. Is it possible? The alternative would be to add a field inside the xml resources, index on it, and perform a query. Thanks, Martín.

3 3

Erstellen eines XLSX Sheet mittels Apache POI in einer XQuery Rest Funktion
by Arthur Stocker 14 Sep '15

14 Sep '15

Hallo Zusammen Ich möchte in einer REST Funktion Daten als XLSX zurück geben. Via Java Bindings wollte ich in der Xquery Funktion die Apache POI nutzen und habe mal versucht das einfache Beispiel der JSP Seite zu übersetzen, scheitere nun aber an der Meldung "[XPST0017] Several implementations found for …. „ beim Aufruf der Funktion createSheet(). Einfacher Versuch - XQ Funktion. import module namespace xlsx = "org.apache.poi.xssf.usermodel.XSSFWorkbook"; declare %rest:path('/test') function root:test( ) { let $sheet := xlsx:createSheet() let $row := $sheet:createRow(0) let $cell := $row:createCell(0) return $cell:setCellValue("Some text“) }; JSP Source: <%@page import="org.apache.poi.xssf.usermodel.*" %><%@page import="java.io.*" %><% // create a small spreadsheet if xls then HSSF.... instead of XSSF.... for xlsx XSSFWorkbook wb = new XSSFWorkbook(); XSSFSheet sheet = wb.createSheet(); XSSFRow row = sheet.createRow(0); XSSFCell cell = row.createCell(0); cell.setCellValue("Some text"); // write it as an excel attachment ByteArrayOutputStream outByteStream = new ByteArrayOutputStream(); wb.write(outByteStream); byte [] outArray = outByteStream.toByteArray(); //response.setContentType("application/ms-excel"); response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"); response.setContentLength(outArray.length); response.setHeader("Expires:", "0"); // eliminates browser caching //response.setHeader("Content-Disposition", "attachment; filename=testxls.xls"); response.setHeader("Content-Disposition", "attachment; filename=testxls.xlsx"); OutputStream outStream = response.getOutputStream(); outStream.write(outArray); outStream.flush(); %> Die Dokumentation konnte mir das weitere Vorgehen nicht so ganz aufzeigen. Hat jemand das Apache POI schonmal in XQuery genutzt? Beim Lesen der Doku und den Beispielen bekomme ich den Eindruck, nicht um das Schreiben einer eigenen JAVA Klasse herum zu kommen. VG Arthur

2 1

DELETE Performance
by Martín Ferrari 14 Sep '15

14 Sep '15

Hi: I have a DB with around a million records. Adding resources to it takes some milliseconds. However, executing DELETE on a resource takes more than 60 seconds. Am I doing something wrong? Is this by design? Thanks! Martín.

3 3

http:request timeout
by Marco Lettere 13 Sep '15

13 Sep '15

Hello all, I think there is a mismatch in the semantics of timeout for the http:request datastructure. expath says it's in seconds where as the basex impl treats it as milliseconds. Could you confirm this? The expath states: [...] |timeout|is the maximum number of seconds to wait for the server to respond. If this time duration is reached, an error is thrown[err:HC006] <http://expath.org/spec/http-client#errHC006>. [...] whereas in the HttpClient (lines 168 and 169) the code for setting the timeout is the following: [...] final String timeout = request.attribute(TIMEOUT); if(timeout != null) conn.setConnectTimeout(Strings.toInt(timeout)); [...] conn being an instance of HttpUrlConnection and the documentation of setConnectTimeout states it to be in milliseconds. If this is not my bad interpretation, maybe a note in the docs should be provided ... Regards, Marco.

3 2

FW: duplicate declaration error-message when using dynamic module-imports ( inspect:functions) ( in the rebound)
by Rob Stapper 11 Sep '15

11 Sep '15

Hi, Probably my previous mail got lost somewhere, so here it is again. Regards, Rob Stapper Van: Rob Stapper [mailto:r.stapper@lijbrandt.nl] Verzonden: dinsdag 8 september 2015 16:16 Aan: BaseX Onderwerp: duplicate declaration error-message when using dynamic module-imports ( inspect:functions) Hi, I get a "duplicate declaration" error-message when I using the inspect:functions-function. Consider program: testRun1-2a.xq. The program imports module: module1.xqm and module2a.xqm. Module: module1.xqm, declares a variable: $mod1:x, and a function: mod1:show( $x). Module: module2a.xqm, declare a function: mod2a:run, which calls a function by a given function-name: $name in a dynamically imported module by a given module-uri: $uri. Program: testRun1-2a.xq, calls the run-function in module2a and tells it to use variable: $mod1:x, on function: mod1:show in module: module1.xqm. This triggers an error: Duplicate declaration on static variable $mod1:x. On the other hand, program: testRun1-2b.xq, does exactly the same except that module:module2b.xqm, is statically linked to module: module1.xqm, instead of dynamically as happens in module:module2a.xqm. Shouldn't the dynamically import of modules behave in the same manner as its static counterpart? Is it possible to achieve this? Cheers, Rob PS the examples are a bit awkward but in my real-life program this construction makes sense (to my anyway). --- Dit e-mailbericht is gecontroleerd op virussen met Avast antivirussoftware. https://www.avast.com/antivirus

1 0

Re: [basex-talk] problem with prepared
by michele.greco2＠email.it 10 Sep '15

10 Sep '15

Sorry and thanks for the answer. Michele --------- Original Message -------- Da: "Dirk Kirsten" <dk(a)basex.org> To: Cc: "Maximilian GÃ&fnof;Â¤rber" <mgaerber(a)arcor.de>de>, basex-talk(a)mailman.uni-konstanz.de Oggetto: Re: [basex-talk] problem with prepared Data: 10/09/15 10:28 Hello Michele, please always make sure to send us a SSCCEE (see http://sscce.org/). It is very time-consuming to read such a large XQuery and I highly doubt all this is relevant for your actual problem. Also, I highly doubt that you need 8 (!) for-loops, which are all nested. This takes an execution time of n^8, which is **a lot**, even for small values of n. Also, all your for loops seem to do basically the same and work on the same data, so I doubt that it is really neccessary. Based on this and your previous questions I would recommend you to get a better understanding of XQuery before tackling your actual problem. Otherwise, you will always stumble across new problems, which you could self yourself better and faster if you know the core conecpts. I can recommend the XQuery book by Priscilla Walmsley, I also find the wikibook about XQuery (https://en.wikibooks.org/wiki/XQuery) quite useful. As part of our commercial offering we also offer seminars or on-site training (see http://basexgmbh.de/en/). If you are interested, please contact us in private. Regards, Dirk On 09/10/2015 10:10 AM, michele.greco2(a)email.it wrote: Thanks for the help ,Max. I modified my xquery this way: declare namespace pkg="http://schemas.microsoft.com/office/2006/xmlPackage"; declare namespace w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"; declare function local:indice() as xs:integer* {   for $i in(1 to count(for $document in collection("curriculum") let $c:= document-uri($document) for $a in doc($c) order by $c return  for $e in doc($c)//w:tc[.//text()="Nome"]  return $e/./following::text()except   (for $x in doc($c)//w:tc[.//text()="Indirizzo"] return $x//following::text()))) return $i }; declare function local:nome() as xs:string* { for $n in (for $document in collection("curriculum") let $c:= document-uri($document) for $a in doc($c) order by $c return  for $e in doc($c)//w:tc[.//text()="Nome"]  return $e/./following::text()except   (for $x in doc($c)//w:tc[.//text()="Indirizzo"] return $x//following::text())) return $n }; declare function local:indirizzo() as xs:string* { for $in in (for $document in collection("curriculum") let $c:= document-uri($document) for $a in doc($c) order by $c return  for $e in doc($c)//w:tc[.//text()="Indirizzo"] return $e/./following::text() except   (for $x in doc($c)//w:tc[.//text()="Telefono"] return $x//following::text())) return $in  }; declare function local:telefono() as xs:string* { for $t in (for $document in collection("curriculum") let $c:= document-uri($document) for $a in doc($c) order by $c return  for $e in doc($c)//w:tc[.//text()="Telefono"] return $e/./following::text() except   (for $x in doc($c)//w:tc[.//text()="Fax"] return $x//following::text())) return $t };  declare function local:fax() as xs:string* { for $f in (for $document in collection("curriculum") let $c:= document-uri($document) for $a in doc($c) order by $c return  for $e in doc($c)//w:tc[.//text()="Fax"] return $e/./following::text() except   (for $x in doc($c)//w:tc[.//text()="E-mail"] return $x//following::text())) return $f }; declare function local:email() as xs:string* {  for $em in  (for $document in collection("curriculum") let $c:= document-uri($document) for $a in doc($c) order by $c return  for $e in doc($c)//w:tc[.//text()="E-mail"] return $e/./following::text() except   (for $x in doc($c)//w:tc[.//text()="Nazionalità"] return $x//following::text())) return $em   }; declare function local:nazionalita() as xs:string* { for $na in (for $document in collection("curriculum") let $c:= document-uri($document) for $a in doc($c) order by $c return  for $e in doc($c)//w:tc[.//text()="Nazionalità"] return $e/./following::text() except   (for $x in doc($c)//w:tc[.//text() ="D" ] return $x//following::text())) return $na }; declare function local:dataNascita() as xs:string* { for $d in (for $document in collection("curriculum") let $c:= document-uri($document) for $a in doc($c) order by $c return  for $e in doc($c)//w:tc[.//text()="Data di nascita"] return $e/./following::text() except   (for $x in doc($c)//w:tc[.//text()="Esperienza lavorativa"] return $x//following::text())) return $d }; for $indice at $pos in local:indice() for $nome in local:nome() [$pos] for $indirizzo in local:indirizzo()[$pos] for $telefono in local:telefono()[$pos] for $fax  in local:fax()[$pos] for $email  in local:email()[$pos] for $nazionalita  in local:nazionalita()[$pos] for $datanascita  in local:dataNascita()[$pos] let $init := sql:init("com.mysql.jdbc.Driver"),     $conn := sql:connect("jdbc:mysql://localhost:3306/DbName","user","password''),     $stmt := sql:prepare($conn, "INSERT INTO informazionipersonali(idInformazioniPersonali,nome,indirizzo,telefono,fax,email,nazionalità,dataNascita) VALUES(?,?,?,?,?,?,?,?) ")         let $params:=  <sql:parameters>         <sql:parameter type="int">{$indice}</sql:parameter>        <sql:parameter type="string">{$nome}</sql:parameter>       <sql:parameter type="string">{$indirizzo}</sql:parameter>       <sql:parameter type="string">{$telefono}</sql:parameter>        <sql:parameter type="string">{$fax}</sql:parameter>       <sql:parameter type="string">{$email}</sql:parameter>       <sql:parameter type="string">{$nazionalita}</sql:parameter>        <sql:parameter type="string">{$datanascita}</sql:parameter>      </sql:parameters>  return sql:execute-prepared($stmt,$params)   It works,but it returns only the first row in my database, and i want a row for each value (indice,nome,indirizzo,........) How can i do? --------- Original Message -------- Da: "Maximilian GÃ¤rber" <mgaerber(a)arcor.de> To: "basex-talk(a)mailman.uni-konstanz.de" <basex-talk(a)mailman.uni-konstanz.de> Oggetto: Re: [basex-talk] problem with prepared Data: 09/09/15 09:12 or using a where statement for $indice at $pos in local:indice() for $nome at $pos2 in local:nome() let $params:= <sql:parameters> <sql:parameter type="int">{$indice}</sql:parameter> <sql:parameter type="string">{$nome}</sql:parameter> </sql:parameters> where $pos2 = $pos return $params 2015-09-09 9:05 GMT+02:00 Maximilian Gärber <mgaerber(a)arcor.de>de>: > Hi, > > when you join (by nesting for statements) you will always get the > combinations unless you provide more constraints. > > You might try something like: > > declare function local:indice() as xs:integer* { > > for $i in (1 to 10) > return $i > }; > > declare function local:nome() as xs:string* { > > for $n in ('a', 'b', 'c', 'd') > return $n > }; > > for $indice at $pos in local:indice() > for $nome in local:nome()[$pos] > > let $params:= <sql:parameters> > <sql:parameter type="int">{$indice}</sql:parameter> > <sql:parameter type="string">{$nome}</sql:parameter> > </sql:parameters> > return $params > > > Regards, Max > > 2015-09-08 16:20 GMT+02:00 <michele.greco2(a)email.it>it>: >> Hi i have following xquery: >> declare function local:indice() as xs:integer* { >> count(for $document in collection("curriculum") >> let $c:= document-uri($document) >> for $a in doc($c) >> order by $c >> return >> for $e in doc($c)//w:tc[.//text()="Nome"] >> return $e/./following::text()except >> (for $x in doc($c)//w:tc[.//text()="Indirizzo"] >> return $x//following::text())) >> }; >> >> declare function local:nome() as xs:string* { >> for $document in collection("curriculum") >> let $c:= document-uri($document) >> for $a in doc($c) >> order by $c >> return >> for $e in doc($c)//w:tc[.//text()="Nome"] >> return $e/./following::text()except >> (for $x in doc($c)//w:tc[.//text()="Indirizzo"] >> return $x//following::text()) >> }; >> for $ indice in local:indice() >> let $init := sql:init("com.mysql.jdbc.Driver"), >> $conn := >> sql:connect("jdbc:mysql://localhost:3306/DBName","user","password''), >> $stmt := sql:prepare($conn, "INSERT INTO >> informazionipersonali(idInformazioniPersonali,nome,indirizzo,telefono,fax,email,nazionalità,dataNascita) >> VALUES(?,?,'null','null','null','null','null','null') ") >> for $nome in local:nome() >> let $params:= <sql:parameters> >> <sql:parameter type="int">{$indice}</sql:parameter> >> <sql:parameter type="string">{$nome}</sql:parameter> >> </sql:parameters> >> return sql:execute-prepared($stmt,$params) >> >> >> which return a combination of index and names, but i want only one row for >> each index and name. >> How can i do? >> >> ---- >> ZE-Light e ZE-Pro: servizi zimbra per caselle con dominio email.it, per >> tutti i dettagli clicca qui >> >> Sponsor: >> Registra i domini che desideri ed inizia a creare il tuo sito web >> Clicca qui ---- ZE-Light e ZE-Pro: servizi zimbra per caselle con dominio email.it, per tutti i dettagli clicca qui Sponsor: Soluzioni di email hosting per tutte le esigenze: dalle caselle gratuite a quelle professionali su piattaforma Zimbra, da quelle su proprio dominio a quelle certificate PEC. Confronta le soluzioni Clicca qui -- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22 -- ZE-Light e ZE-Pro: servizi zimbra per caselle con dominio email.it, per tutti i dettagli Clicca qui http://posta.email.it/caselle-di-posta-z-email-it/?utm_campaign=email_Zimbr… Sponsor: Caselle con tuo dominio su piattaforma Zimbra, fino a 30 GB di spazio, sincronizzazione dati e backup Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=13324&d=10-9

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk