BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5269 discussions

Performance issues and query optimizer
by France Baril 20 Oct '17

20 Oct '17

Hi, We are working hard right now on performance issues. I just read this about an upcoming release: Enforce Rewritings In various cases, existing index structures will not be utilized by the query optimizer. This is usually the case if the name of the database is not a static string (e.g., because it is bound to a variable or passed on as argument of a function call). Furthermore, several candidates for index rewritings may exist, and the query optimizer may decide for a rewriting that turns out to be suboptimal. I'm not sure I read it properly so I would like to confirm my understanding: In our code, all our db accesses are in a format like this: db:open($lang)/*[@id='...']. Does the fact that the db name is $lang, a variable, mean that we have been working without the optimizer all this time and therefore that this new feature is there for people like us? Thanks! -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

3 5

ZIP module and ISO-8859-1 filenames
by Andre Bovy 20 Oct '17

20 Oct '17

<DEAR>BaseX</DEAR> I am using the ZIP module for different purpose and until now I didn’t have any problem. Yesterday I received a ZIP file generated by an application and it was impossible to open the file in BaseX. I tried to write a little class opening it in Java. Same behaviour. After different tries, I noticed that the filenames were containing special characters encoded in ISO-8859-1 In my class, I used the java.util.zip.ZipFile method with a charset parameter. It works perfectly. Is there a way to do that with the present ZIP module ? If not, could you please modify the ZIP module to allow the process of ZIP file or ZIP entry with encoded filenames different from UTF-8 Thank you in advance André

2 1

TR: Options for creating database...
by Fabrice ETANCHAUD 20 Oct '17

20 Oct '17

De : Fabrice ETANCHAUD Envoyé : vendredi 20 octobre 2017 11:46 À : 'France Baril' Objet : RE: [basex-talk] Options for creating database... The XQuery will succeed if you remove the INDENT option (serialization), and lower case all your option names. You are right, you may not have a memory overflow issue on creation. But I had the problem with db:add with several huge files, before I switched to BXS. Best regards, Fabrice De : France Baril [mailto:france.baril@architextus.com] Envoyé : vendredi 20 octobre 2017 11:36 À : Fabrice ETANCHAUD Cc : BaseX Objet : Re: [basex-talk] Options for creating database... In db:create I don't think the issue is memory. I get: [bxerr:BASX0002] Unknown database option 'UPDINDEX'. My function is: let $options := map:merge(( map:entry("CHOP", false()), map:entry('INDENT', false()), map:entry('STRIPNS', false()), map:entry('INTPARSE', true()), map:entry('DTD', false()), map:entry('XINCLUDE', false()), map:entry('UPDINDEX', true()), map:entry('AUTOOPTIMIZE', true()) )) return ( db:create($db-name, $src-folder, (), $options), db:output(<div>done</div>) ) On Fri, Oct 20, 2017 at 11:29 AM, Fabrice ETANCHAUD <fetanchaud(a)pch.cerfrance.fr<mailto:fetanchaud@pch.cerfrance.fr>> wrote: Bonjour France, Didn’t you find Updindex and autooptimize options in the ‘Options’ panel of the db creation window ? Db:create last parameter is the place to put all your options : http://docs.basex.org/wiki/Database_Module#db:create But the Pending update list may overflow memory during db creation/update. You should definitively have a look at the BaseX Scripts (BXS). You can declare a batch of commands in XML, and ask BaseX to run it : http://docs.basex.org/wiki/Commands#Command_Scripts This is the way to set options before invoking the CREATE-DB command : http://docs.basex.org/wiki/Commands#SET I usually write XQuery to generate a BXS that will do the job. Cordialement, Fabrice CERFrance Poitou-Charentes De : basex-talk-bounces(a)mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de> [mailto:basex-talk-bounces@mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de>] De la part de France Baril Envoyé : vendredi 20 octobre 2017 10:44 À : BaseX Objet : [basex-talk] Options for creating database... Hi, I usually create dbs using the gui. I'm now trying to create asb without it to be able to set the updindex and autooptimize options, which I can't find in the GUI. I couldn't figure how to set them out using db:create either, so I switch to command line. Now I'm feeling dumb, I can't find how to set up parsing/indexing options for the command line. I'm looking at: http://docs.basex.org/wiki/Command-Line_Options. I found -sindent and -wchop only. What am I missing? Here are all the options that I'd want to set: let $options := map:merge(( map:entry("CHOP", false()), map:entry('INDENT', false()), map:entry('STRIPNS', false()), map:entry('INTPARSE', true()), map:entry('DTD', false()), map:entry('XINCLUDE', false()), map:entry('UPDINDEX', true()), map:entry('AUTOOPTIMIZE', true()) )) I don't want to set UPDINDEX and AUTOOPTIMIZE in .basex because I only want them to be true() on one of my DBs. Other DBs should remain as is. -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com<mailto:france.baril@architextus.com> -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com<mailto:france.baril@architextus.com>

2 3

Options for creating database...
by France Baril 20 Oct '17

20 Oct '17

Hi, I usually create dbs using the gui. I'm now trying to create asb without it to be able to set the updindex and autooptimize options, which I can't find in the GUI. I couldn't figure how to set them out using db:create either, so I switch to command line. Now I'm feeling dumb, I can't find how to set up parsing/indexing options for the command line. I'm looking at: http://docs.basex.org/wiki/Command-Line_Options. I found -sindent and -wchop only. What am I missing? Here are all the options that I'd want to set: *let **$options* := *map:merge*(( *map:entry*("CHOP", *false*()), *map:entry*('INDENT', *false*()), *map:entry*('STRIPNS', *false*()), *map:entry*('INTPARSE', *true*()), *map:entry*('DTD', *false*()), *map:entry*('XINCLUDE', *false*()), *map:entry*('UPDINDEX', *true*()), *map:entry*('AUTOOPTIMIZE', *true*()) )) I don't want to set UPDINDEX and AUTOOPTIMIZE in .basex because I only want them to be true() on one of my DBs. Other DBs should remain as is. -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

Jobs, locking and doing large updates
by Omar Siam 18 Oct '17

18 Oct '17

Hi list! I was experimenting with the jobs module for the last weeks to speed up updates and to make them fit into less than 6 GB of memory. It does not work the way I expected. * Updating jobs don't seem to run in parallel even if they don't work with and lock the same database. Or do they start much later than I expected? * It seems to me there is an upper limit of jobs that can be queued (about 100?). I automatically started all of the update jobs but some updates did not run. The upside is: I can run my updates with just about 1 GB of memory which is much better for me. I work with dictionary like XML documents. They look like <root> <entry>Contents with further tags</entry> <entry>Contents with further tags</entry> <entry>Contents with further tags</entry> ... a few thousand more ... <entry>Contents with further tags</entry> <entry>Contents with further tags</entry> </root> I add or change larger parts of them and I also need to keep track of changes. So in a separate database there are old versions of entries with time stamps (@dt) like <hist> <entry dt="">Contents with further tags</entry> <entrydt="">Contents with further tags</entry> <entrydt="">Contents with further tags</entry> ... a few thousand more ... <entrydt="">Contents with further tags</entry> <entrydt="">Contents with further tags</entry> </hist> I tried to do everything at once and when I need to update most of my entries (about 30000) I exhaust my memory. So I use the jobs module to do this 100 at a time so the updates list does not grow beyond any reasonable size. Having those 100 as one transaction is good enough for my needs. Then I thought maybe I should use jobs to seprate the two tasks. One async job saves a memory (update) copy of the old entry, the other writes the new one. After some transformations to my XQuery BaseX told me they don't lock the same database or global. I expected the jobs running on different databases to work in parallel. They don't. I don't quite understand why. Also I have a much larger dataset split into a number of databases where it would be quite useful to execute updates in parallel. Am I missing something? Is this perhaps the wrong way to tackle this scaling problem? Best regards Omar Siam

2 2

AUTHMETHOD = Custom
by Dave Day 18 Oct '17

18 Oct '17

Greetings list, In the Basex configuration file, I added AUTHMETHOD = Custom After starting the server, the 1st response I get from the server when I connect to it is the realm:nonce values. In the Basex manual, it states for AUTHMETHOD _Specifies the default authentication method, which will be used by the HTTP server for negotiating__ __credentials. Allowed values are Basic, Digest, and Custom:_ _ With custom authentication, the server will not do any authentication._ Is it possible to connect to Basex server without going thru any authentication? Regards, -- Dave Day

3 7

Every query executed in dba Queries causes a global read lock
by Omar Siam 17 Oct '17

17 Oct '17

Hi list! Just discovered sth. unexpected: Every query executed in dba Queries causes a global read lock. Also other (very simple) queries using RESTXQ which use a DB explicit in the query are executed with a global read lock. Is this intentional? Is this just a side effect? I noticed that internally the code I run is wrapped in an xquery:eval. Maybe this causes this lock? Or does every RESTXQ cause a global read lock? I ran into this because I wanted to execute async code using this interface using the jobs module and then wait for the result. This of course locks up as soon as any access to data in dbs is attempted. Best regards Omar Siam

2 1

Verlosung+Umfrage: Essstörungen-eine Belastung für Freundschaften?
by Freunda＠students.uni-marburg.de 17 Oct '17

17 Oct '17

Liebe Kommilitonin, lieber Kommilitone, Du hast eine Freundin oder einen Freund, die/der (vermutlich) an einer Essstörung erkrankt ist? Im Rahmen unserer Abschlussarbeiten im Master-Studiengang Psychologie in der AG Kinder- und Jugendpsychologie (Betreuung: Prof. Dr. Hanna Christiansen) beschäftigen wir uns mit einem möglichen Unterstützungsangebot speziell für Freunde essgestörter Menschen. Um herauszufinden, an welchen Stellen welche Art von Unterstützung hilfreich wäre, führen wir eine anonyme Online-Befragung durch. Es dauert nur ca. 10 Minuten! Als Dankeschön für die Teilnahme verlosen wir fünf Amazon-Gutscheine im Gesamtwert von 100 Euro. Hier der Link zur Umfrage: www.soscisurvey.de/Hilfsangebot_Freunde/?r=UkoTK Mit Deiner Teilnahme würdest du einen wichtigen Beitrag zu diesem Projekt leisten und uns sehr helfen! Anika Knorn und Annika Freund

1 0

Incorrect result with negative 7-digit number from fn:format-number()
by Andrew Savikas 17 Oct '17

17 Oct '17

Hello, I'm struggling with some odd behavior while trying to format currency amounts. When the number is negative and 7-digits long, the picture string is not working properly. You should (I hope!) be able to replicate the issue in the GUI: format-number(-999999, "$#,##0") *Result:* -$999,999 <-- Expected Result format-number(-9999999, "$#,##0") *Result:* -$9999,999 <-- *Comma is in the wrong place* format-number(-99999999, "$#,##0") *Result:* -$99,999,999 <-- Expected Result Am I missing something here? Thank you!

2 2

404
by Xavier-Laurent SALVADOR 14 Oct '17

14 Oct '17

Hi List, a question I have in mind since a long time: I have a Basex serving multiple project sites behind Nginx Redirections. I wrote a servlet for 404 errors in web.xml: <error-page> <error-code>404</error-code> <location>/error404</location> </error-page> and a function: declare %rest:path("/error{$code}") %output:method("html") But if i try to get "http://www.bla.fr/xzxzxzxzxz", basex answers: "No function found that matches the request." How could I inhibit this behavior ? thanks ! -- Ce message peut contenir des informations réservées exclusivement à son destinataire. Toute diffusion sans autorisation est interdite. Si vous n'en êtes pas le destinataire, merci de prendre contact avec l'expéditeur et de détruire ce message. *This email may contain material for the sole use of the intended recipient. Any forwarding without express permission is prohibited. If you are not the intended recipient, please contact the sender and delete all copies*.

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk