- BaseX-Talk - mailman.uni-konstanz.de

cross-platform file name parameters
by Graydon Saunders 26 Jan '21

26 Jan '21

Hello -- So I'm on a Linux box and my colleague is on a Windows system. We're both using a query that takes before and after versions of a file, converts each to a single string, and writes that back out as two additional files. (This exists to check if we still have all the words in the same order between before and after.) There must be a sensible way to parameterize the file names so they can be relative to the query on both platforms/easily loaded from a file/something that isn't hand-editing a variable (external or not), but I can't think of it. What's the sensible way to do this? Thanks! Graydon

3 2

[Configuration improvements] Separate client and server config, stop overwriting config files
by Volodymyr Prokopyuk 26 Jan '21

26 Jan '21

Hello the BaseX team, First of all I'm really impressed with the quality of the BaseX XML database. I'd particularly highlight the high compliance with the standards XQuery, XPath and the outstanding performance! Thank you very much for the great product! I'm new to the system and I've faced several configuration deficiencies while trying to configure the BaseX database to fit my needs. What follows is my use case (I hope it is a basic and classical one required by much wider audience), the configuration problems I've faced (I hope these can be easily solved by extending your current implementation) and my proposal to solve the above mentioned configuration problems (I hope this will further improve the quality of your product). *My use case (access the basexserver run as a systemd service via the basexclient)* - I've installed the basex package on Manjaro (Arch linux) into /usr/share/java/basex/ - I want to run the basexserver as a systemd service under the already created basex system user (see the attached basex.server unit file) - I want to store the XML databases and the BaseX logs under /var/lib/basex/data /var/lib/basex/log respectively - I want to manage the basexserver configuration (namely DBPATH, LOGPATH, SERVERPORT) separately from the basexclient configuration (namely HOST, PORT, USER, PASSWORD) in a separate files under /etc/basex and ~/ respectively. I do not want to get my configuration overwritten with unnecessary default options that are already in the system and so clutter my clean just-what-you-need configuration files - I want to manage the XML databases and execute queries against the server using the basexclient from my console by providing XQuery command and queries in a file *Configuration problems (mixed config for client and server in a single .basex file under hardly linked home directory)* - My .basex file mixes server and client configuration in one single file so I cannot separate the configuration into server-related at /etc/basex and client-retated at ~/ - My .basex file gets overwritten with unnecessary default options that clutter my minimal and clean configuration - When I start the basexclient a new .basex configuration file is created in the current working directory. I start the basexclient from many different direcotries, but I want to have the basexclient configuration only in my home ~/ directory - It seems that the location of the .basex file is hardly linked to the home directory forcing me to place data, log and config directories under a single home directory. I want to manage my data in /var/lib/basex/data, my logs in /var/lib/basex/log and my server and client configuration in /etc/basex and ~/ respectively - I cannot find any way (via environment variables or command line options) to instruct both the basexserver and the basexclient to look for the configuration file .basex *Proposal to improve configuration* - Do not overwrite the user-provided configuration files with unnecessary default options that are already in the system or at least use a different file (e. g. .basex-default) - Separate the server-related configuration from the client-related configuration into different files (e. g. .basexserver, .basexclient) - Do not mandate the location of configuration files to be under the home directory because there are different aspects of database management that are usually managed in different directories with different policies and quotas (e. g. data, log, client and server config) - Provide separate options for basexserver and basexclient to locate the corresponding configuration file (e. g. via environment variables BASEX_SERVER_CONFIG, BASEX_CLIENT_CONFIG or command line options -server-config, -client-config) - In order to introduce the proposed improvement you can keep your current implementation unchanged (following the open-closed principle) and just extend the implementation with the alternative locations for the separate configuration files via environment variables or command line options. The only change that is required is to stop overwriting the user-provided configuration files with the default options Can the above described use case be configured with the current implementation of the configuration management? Thank you in advance! I'm open to further discussion and collaboration mostly on the design side to get these improvements to work for all other use cases not covered here. Thank you, Vlad

2 1

BaseX python replace bug
by Daniel Kaplan 22 Jan '21

22 Jan '21

Hello, I'm experiencing some issue(s) when using BaseX... wanted to report it here, but not sure what information you need. I'm accessing baseX via python. Attached is some code that reproduces it for me. It doesn't seem to occur when I am not manually flushing, and only seems to occur with replace, but not add. How should I proceed? Cheers, Daniel PS: I am using a folder containing all the items in question.

3 5

Securing All Endpoints
by Adam Law 18 Jan '21

18 Jan '21

Golang has great middleware functionality for endpoints that allows a person to readily secure all of them. I was wondering there would be anyway doing similar in BaseX for restxq & websockets. For example by requiring all requests to have a particular query parameter (very basic example). ... to protect say against browser / javascript port scanning. Eg https://stackoverflow.com/questions/26204485/gorilla-mux-custom-middleware and check that request contains a particular parameter (very basic security) It seems that I have to secure each endpoint individually. Adam

2 6

error "lock file does not exist" after "add" command
by Thomas Spitaler 13 Jan '21

13 Jan '21

Hi, in one of our projects we are using BaseX. So far we are very happy with our choice, but recently we got an error that we could not fix. The only solution that worked was to create a new, empty database. When trying to add a document to the database, we get an exception and a message „lock file does not exist“: $ open cap10_cap_store_prod Database 'cap10_cap_store_prod' was opened in 219.99 ms. $ add /home/thomas/tmp/test.xml Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 9.0.1 Java: Debian, 11.0.9 OS: Linux, amd64 Stack Trace: java.lang.RuntimeException: cap10_cap_store_prod: lock file does not exist. at org.basex.util.Util.notExpected(Util.java:61) at org.basex.data.DiskData.finishUpdate(DiskData.java:246) at org.basex.core.cmd.ACreate.update(ACreate.java:97) at org.basex.core.cmd.Add.run(Add.java:56) at org.basex.core.Command.run(Command.java:257) at org.basex.core.Command.execute(Command.java:93) at org.basex.api.client.LocalSession.execute(LocalSession.java:132) at org.basex.api.client.Session.execute(Session.java:36) at org.basex.core.CLI.execute(CLI.java:92) at org.basex.core.CLI.execute(CLI.java:76) at org.basex.BaseX.console(BaseX.java:196) at org.basex.BaseX.<init>(BaseX.java:171) at org.basex.BaseX.main(BaseX.java:42) Which lock file is the error referring to? Can we create it manually to make it work again? The error only happens in the database that we have been using so far (cap10_cap_store_prod). After creating and using a new one (cap10_cap_store_prod2) the problem does not occur. Here is the output of running list on the database: $ list Name Resources Size Input Path cap10_cap_store_prod 233387 1583797639 cap10_cap_store_prod2 241 1912414 testdb 1 4611 3 database(s). Did we hit a size limit of BaseX? I checked the server and it has enough free disk space and free inodes available. Some more background info: The system runs on a cluster of servers. BaseX runs on one of them, bound to port 1984. The other nodes in the cluster connect to the BaseX-Server via the network. In my understanding, this type of concurrency is supported by BaseX. The software on the nodes is implemented in Elixir. We wrote our own Elixir client library that is a thin wrapper around https://github.com/zadean/basexerl, an existing open source Erlang client library for BaseX. Our Elixir client library keeps open a pool of connections to the BaseX server. So far and until we encountered the above error everything worked perfectly. Also, after creating a new database everything works fine again. This error also occurs after shutting down all the nodes that connect to our BaseX instance. We checked using the lsof command and there are no connections open to the database other then the client we use to reproduce the error. Any help would be greatly appreciated! Kind regards, Thomas Kind regards, Thomas

2 4

file:read-text
by Giuseppe G. A. Celano 12 Jan '21

12 Jan '21

Hi, I am trying to open a bunch of files with file:read-text, but I get the error [file:io-error] invalid XML character: #0, even I am not dealing with XML. Any idea why this happens? Thanks! Best, Giuseppe

2 3

Brew upgrade for 9.4.6 fails with SHA256 mismatch
by Johan Mörén 08 Jan '21

08 Jan '21

Just wanted to let you know that the latest BaseX version can't be installed or upgraded via homebrew. I get the following error message: brew upgrade basex ==> Upgrading 1 outdated package: basex 9.4.5 -> 9.4.6 ==> Upgrading basex 9.4.5 -> 9.4.6 ==> Downloading https://files.basex.org/releases/9.4.6/BaseX946.zip ######################################################################## 100.0% Error: SHA256 mismatch Expected: c6dd9ac56e72de19c88152c76a6e4052deccd4f0db7a4b072aaaa560b92318a5 Actual: 96012025d749062540f9b16f830d7b99876696b66534da2fe27c833d1b1fc43b File: /Users/johmor/Library/Caches/Homebrew/downloads/befef7ec63e74dbf695bd24e06e16773055a9fb0c7f69d9d7e9fd1787766a377--BaseX946.zip To retry an incomplete download, remove the file above. Regards, Johan

2 4

Improper use/potential bug error
by Eliúd Santiago Meza y Rivera 08 Jan '21

08 Jan '21

Hi, I just update to the latest version of BaseX 9.4.6, and the web server now do not refer correctly the link, checking every individually broken link, the server response this message: I attach the code. In advance I appreciate your help. HTTP ERROR 500 java.lang.NoSuchMethodError: org.basex.query.value.type.SeqType.cast(Lorg/basex/query/value/Value;Lorg/basex/query/QueryContext;Lorg/basex/query/StaticContext;Lorg/basex/util/InputInfo;)Lorg/basex/query/value/Value; URI: /app/static/pngkit_realistic-spider-web-png_1049388.png STATUS: 500 MESSAGE: java.lang.NoSuchMethodError: org.basex.query.value.type.SeqType.cast(Lorg/basex/query/value/Value;Lorg/basex/query/QueryContext;Lorg/basex/query/StaticContext;Lorg/basex/util/InputInfo;)Lorg/basex/query/value/Value; SERVLET: RESTXQ CAUSED BY: java.lang.NoSuchMethodError: org.basex.query.value.type.SeqType.cast(Lorg/basex/query/value/Value;Lorg/basex/query/QueryContext;Lorg/basex/query/StaticContext;Lorg/basex/util/InputInfo;)Lorg/basex/query/value/Value;Caused by: java.lang.NoSuchMethodError: org.basex.query.value.type.SeqType.cast(Lorg/basex/query/value/Value;Lorg/basex/query/QueryContext;Lorg/basex/query/StaticContext;Lorg/basex/util/InputInfo;)Lorg/basex/query/value/Value; at org.basex.http.web.WebFunction.bind(WebFunction.java:137) at org.basex.http.restxq.RestXqFunction.bind(RestXqFunction.java:210) at org.basex.http.restxq.RestXqResponse.bind(RestXqResponse.java:60) at org.basex.http.web.WebResponse.create(WebResponse.java:61) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:73) at org.basex.http.BaseXServlet.service(BaseXServlet.java:65) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:738) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905) at java.lang.Thread.run(Unknown Source) ------------------------------ Powered by Jetty:// 9.4.34.v20201102 <https://eclipse.org/jetty> -- LSC Eliud Santiago Meza y Rivera

2 1

BaseX 9.4.6
by Christian Grün 07 Jan '21

07 Jan '21

Dear all, Our new year patch release features various new performance tweaks and minor bug fixes. Thanks all for your support! Have a pleasant 2021, Christian BaseX Team

2 1

FOP module
by Jerome Chauveau 07 Jan '21

07 Jan '21

Hi, I am trying to use the (too old ?) FOP module, but unsuccessfully. I copied that example : import module namespace fop = 'org.basex.modules.fop.FOP'; let $xsl := xslt:transform('fop.xml', 'fop.xsl') let $pdf := fop:transform($xsl) return file:write-binary('fop.pdf', $pdf) from this thead : [ https://mailman.uni-konstanz.de/pipermail/basex-talk/2014-February/006326.h… | https://mailman.uni-konstanz.de/pipermail/basex-talk/2014-February/006326.h… ] which generates that error (in a BaseX 9.3) : [XPTY0004] class org.basex.modules.fop.FOP tried to access private method 'void org.basex.query.value.item.B64.<init>(byte[])' (org.basex.modules.fop.FOP and org.basex.query.value.item.B64 are in unnamed module of loader 'app'). Caused by: FOP:transform(document-node()). Any suggestions on how to fix this ? Thanks Regards Jérôme

3 9

error "lock file does not exist" after "add" command
by Zachary N. Dean 04 Jan '21

04 Jan '21

Hi Thomas, Author of the Erlang client here. :-) I haven't seen this before, but I haven't used pooled connections either. Is there a minimal example for the commands that you run that I could use to reproduce this and ensure it's not a problem in the client? Thanks, Zack

2 1

LDAP Module
by Maciej Nestorowicz 28 Dec '20

28 Dec '20

Hi, I wanted to know, is there any module in BaseX which is responsible for LDAP authentication and authorization? Kind Regards, Maciej Nestorowicz

1 0

Database access in a function
by Giuseppe G. A. Celano 18 Dec '20

18 Dec '20

Hi, I have a script where I join two databases. It works. However, when I try to put the content of this script in a function, and then call it (with the databases being the arguments), the execution is slowed down (I guess the indexes are not properly accessed). Is there a way to overcome this? Thanks.

2 1

write result to file?
by Nicholas 16 Dec '20

16 Dec '20

how is the return result from a query written to a file? thanks, Nick

3 5

set parser csv header
by Nicholas 15 Dec '20

15 Dec '20

How do I set the CSV parser to use "header" true? so that when the record is imported it has the header information as below? nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ basex BaseX 9.0.1 [Standalone] Try 'help' to get more information. > > create database addressbook Database 'addressbook' created in 240.02 ms. > > open addressbook Database 'addressbook' was opened in 0.06 ms. > > set parser csv PARSER: csv > > add addressbook.csv Resource(s) added in 23.74 ms. > > xquery . <csv> <record> <entry>Name</entry> <entry>First Name</entry> <entry>Address</entry> <entry>City</entry> </record> <record> <entry>Huber</entry> <entry>Sepp</entry> <entry>Hauptstraße 13</entry> <entry>93547 Hintertupfing</entry> </record> </csv> Query executed in 219.65 ms. > > exit Enjoy life. nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ basex addressbook.xq <csv> <record> <Name>Huber</Name> <First_Name>Sepp</First_Name> <Address>Hauptstraße 13</Address> <City>93547 Hintertupfing</City> </record> </csv>nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ cat addressbook.csv Name,First Name,Address,City Huber,Sepp,Hauptstraße 13,93547 Hintertupfing nicholas@mordor:~/flwor/csv$ nicholas@mordor:~/flwor/csv$ cat addressbook.xq xquery version "3.1"; let $text := file:read-text('addressbook.csv') return csv:parse($text, map { 'header': true() }) nicholas@mordor:~/flwor/csv$ see also: https://stackoverflow.com/q/65275618/4531180 thanks, Nick

2 1

java bindings - Data types
by Andre Bovy 15 Dec '20

15 Dec '20

Dear BasexTalk, for different (good or bad) reasons, I would like to use the java HTTPConnection instead of the http:client module. But, I am not able to find the correct syntax for these statements. I have tried the middle dot character to force the correct type, without success I receive always the message : [XPTY0004] java.net.URL instance expected as first argument, xs:anyURI found. Any idea? Thank you in advance declare namespace HttpURLConnection ="java.net.HttpURLConnection"; declare namespace URL ="java.net.URL"; let $urlString := "https:www.google.com" let $url := URL:new($urlString) let $httpConn := URL:openConnection($url) return $urlString

2 1

Slow BaseX cold performance
by Peter Villadsen 14 Dec '20

14 Dec '20

All, I am facing some issues with BaseX (9.3.1) performance when running cold, i.e. immediately after the server is started. The problem I am seeing is that the performance of moderately complex queries is terrible for a some time after the database is opened, but then improves dramatically after a few query evaluations have taken place. For instance, a certain query takes around 7 minutes to when the server has just been started, but then performance improves to consistently to take around 18 seconds. This happens for most queries, it is not a characteristic of any particular query, as far as I can tell. This is annoying, of course, and users are complaining that the system is not responsive. Here are some observations: 1. The first time the query is submitted the java process does not consume any CPU cycles and no footprint is added to the 10GB that is allocated when the database is opened (it takes around 1 minute to open the database. You can see the database info below). I would have expected a surge of activity as soon as the query was submitted, as all the 16 cores contribute to evaluating the result. 2. If I use the BaseX GUI (with an allowed 26GB but consuming around 10GB in this case) I never get any delays - The query performance is consistently around 18 secs, immediately after opening the database. 3. Both the HTTP server or TCP server exhibit this anomaly. 4. After the initial sluggishness, when the database is running warm, the performance stays satisfactory, at around 18 seconds. I have never seen it return to the bad situation. 5. The first time the query is run, the footprint rises about 1GB to around 11GB. It does not increase with subsequent query evaluations. 6. I think that the system remains warm even if I close down the java server process and start it again. Is this an accepted behavior or an anomaly? Why is the behavior different if the database is queried in the stand-alone situation (the BaseX UI client)? Is there some setting that I need to provide to the Java process to ameliorate the cold performance? Any indexes that I have to put in place? Any suggestions are welcome. Here are some stats describing the hardware setup: The machine hosting both the server and the client is a powerful 16 core box running a AMD EPYC 7452 Processor with 64GB RAM running in the cloud; since they both the client and the server are running on the same box, there should be negligible network latency in this scenario. The machine (running windows server 2016 datacenter) is doing nothing else at the time of these tests: The Idle process consumes 99% of the CPU time and almost no disk or network access takes place. The machine does not have hardware disks - They are hosted on another box, I believe. The database information: <database> <databaseproperties> <name>generated</name> <size>16 GB</size> <nodes>705422480</nodes> <documents>136673</documents> <binaries>0</binaries> <timestamp>2020-09-17T22:24:15.037Z</timestamp> <uptodate>false</uptodate> </databaseproperties> <resourceproperties> <inputpath>C:/Users/pvillads/Desktop/generated/</inputpath> <inputsize>16 GB</inputsize> <inputdate>2020-09-17T22:24:14.943Z</inputdate> </resourceproperties> <indexes> <textindex>false</textindex> <attrindex>false</attrindex> <tokenindex>false</tokenindex> <ftindex>false</ftindex> <textinclude/> <attrinclude/> <tokeninclude/> <ftinclude/> <language>English</language> <stemming>false</stemming> <casesens>false</casesens> <diacritics>false</diacritics> <stopwords/> <updindex>false</updindex> <autooptimize>false</autooptimize> <maxcats>100</maxcats> <maxlen>96</maxlen> <splitsize>0</splitsize> </indexes> </database> $ java -version java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode) Thanks, Best Regards Peter Villadsen Principal Architect Microsoft Business Applications Group

2 1

Copying databases between servers: pth.basex?
by James Ball 12 Dec '20

12 Dec '20

Hello, I’ve been debugging a strange issue on a system and I can’t quite understand what’s causing it. I create some databases on my local laptop (macOS). They provide master data for a larger system so I then copy them to a cloud server (Linode/Amazon). I recently switched updated some queries to use db:open with a path restriction to make them run faster. Everything worked on the laptop but nothing was returned on the server for the same paths. If I optimise(all) the database on the server it then works. After the optimisation, on the server there is a file called pth.basex and a file called idp.basex. On the Mac there is NO file in the database folder called pth.basex or idp.basex. I’m not sure what pth.basex or idp.basex are as they’re not referenced on Storage Layout[1] This is copying the database files directly. I have had mixed results using backup zip files - sometimes it works and sometimes it fails. Has anyone else observed this behaviour? What are the steps I should take to ensure reliable movement of databases? What are these mysterious files? I’m happy to update the Wiki with any new information . Many thanks, James [1] https://docs.basex.org/wiki/Storage_Layout <https://docs.basex.org/wiki/Storage_Layout>

2 3

Performance: first and second xQuery execution / local PC and server
by BIRKNER Michael 30 Nov '20

30 Nov '20

Hello BaseX-Community, I have a question concerning performance ... once again ... sorry ;-) First thing I noticed is that when I run a query that involves quite a lot of data (more information below), the first execution of the query takes much more time as the second execution. In my example, the first execution on a certain machine (more information below) took 4,61 hours, the second execution of the very same query took 23,8 minutes. I suppose that there is some caching mechanism in place? Is that true? Second thing is that the same xQuery mentioned above has an extremly different performance on two different machines. I am testing the xQuery on my local PC where it runs quite fast (4,7 minutes on first execution; 4,5 minutes on second execution). On a (virtual) server (where the xQuery should run in the end) it takes much longer (as mentioned above: 4,61 hours on first execution; 23,8 minutes on second execution). Here are some more details: Local PC (fast execution): - OS: Ubuntu Desktop 18.04.5 LTS - RAM: 16GB LPDDR3, 2133 MT/s - Processor: Intel Core i7-8565U 1.80GHz, 8 CPUs - Java version: OpenJDK 11.0.9.1 - BaseX version: 9.4.4 (from the zip archive), using the command line client with Xmx6g Virtual Server, VMware (slow execution): - OS: Ubuntu Server 20.04.1 LTS - RAM: 16GB (no other details available) - Processor: Intel Xeon Gold 6126 2.60GHz, 4 CPUs - Java version: OpenJDK 11.0.9.1 - BaseX version: 9.4.4 (from the zip archive), using the command line client with Xmx6g The data for testing and the xQuery that is executed are exactly the same, also the BaseX version (9.4.4) and the assigned memory (6 GB). The data that I use are freely available library data. They can be downloaded here if that is for interest: 1. https://data.dnb.de/GND/authorities-geografikum_dnbmarc_20201013.mrc.xml.gz 2. https://data.dnb.de/GND/authorities-koerperschaft_dnbmarc_20201013.mrc.xml.… 3. https://data.dnb.de/GND/authorities-kongress_dnbmarc_20201013.mrc.xml.gz 4. https://data.dnb.de/GND/authorities-person_dnbmarc_20201013.mrc.xml.gz 5. https://data.dnb.de/GND/authorities-sachbegriff_dnbmarc_20201013.mrc.xml.gz 6. https://data.dnb.de/GND/authorities-werk_dnbmarc_20201013.mrc.xml.gz So I have 6 data sets from the downloaded data. For each of these data sets I create a separate BaseX database (with options INTPARSE=true and STRIPNS=true ... other options are not touched, so that they have their default values). The databases are named "geografikum", "koerperschaft", "kongress", "person", "sachbegriff" and "werk". I have to mention that the data set "person" is quite big (~20 GB). The others are between 1 GB and 6 GB in size. The databases are not altered before the execution, so all indexes (ATTRINDEX, TEXTINDEX) should be up to date. The xQuery (see below) that I execute does this: 1. Get ID numbers from two other databases ("db1" and "db2" in the xQuery below). I get 144560 ID numbers in total. 2. Iterate over the 6 databases created from the data sets mentioned above and get all records that have one of the 144560 ID numbers that were retrieved before. 3. Count the retreived records from the 6 databases. The result is 134388 records. As mentioned: The execution (especially the first run) on the virtual server is very slow compared to the execution on the local PC. The final goal is not to count the retrieved records, but to create a new database from them. I just encountered the performance issues when I did some testing with "count()". But creating the new database has the same performance issues. So now I basically have 3 questions: 1. Why is the first execution of the xQuery so slow and the second one faster, especially on the server? 2. Why is the execution on the server much slower than on the local PC? 3. I there any way to speed that up? Would it e. g. make sense to split up the very big dataset "person" into multiple databases? Could that big dataset be the bottleneck? Finally, here is the xQuery that is executed: (: Get IDs from 2 other databases :) let $otherDBs := ('db1', 'db2') let $idsInOtherDBs := (# db:enforceindex #) { for $otherDB in $otherDBs return distinct-values(db:open($otherDB)/collection/record /datafield[@tag=('689', '100', '110', '111', '700', '710', '711')] /subfield[@code='0']) } (: Get records from 6 datasets by ID from other databases :) let $searchDBs := ('geografikum', 'koerperschaft', 'kongress', 'person', 'sachbegriff', 'werk') let $result := (# db:enforceindex #) { for $searchDB in $searchDBs return (db:text($searchDB, $idsInOtherDBs) /parent::element(subfield))[(@attribute(code)='a')] /parent::element(datafield)[(@attribute(tag)='035')] /parent::element(record) union (db:text($searchDB, $idsInOtherDBs) /parent::element(controlfield))[(@attribute(tag)='001')] /parent::element(record) } (: Count the records :) return count($result) As always: Thank you very much for any suggestions! Michael

2 2

Future of the BaseX crypto library
by Joe Wicentowski 28 Nov '20

28 Nov '20

Dear fellow BaseX users, For anyone interested in the future of the EXPath Crypto spec and lib, I posted an important update to the EXPath community group with questions for users of the spec and implementations thereof. Please feel free to chime in: https://github.com/expath/expath-cg/issues/132. Thanks very much to Christian, who already has. I'd welcome any other users of the crypto library to register their ideas. Thank you, Joe

1 0

Re: [basex-talk] Re: Sequence comparison
by Martin Honnen 27 Nov '20

27 Nov '20

4 3

Sequence comparison
by Giuseppe G. A. Celano 26 Nov '20

26 Nov '20

Hi, I have to compare two sequences and find common items, irrespective of their positions: for $a in (1,2,3,5,3) for $u in (1,3,2,3) where $a = $u group by $o := $a return <c>{$a}</c> This returns <c>1</c> <c>2</c> <c>3 3 3 3</c> I would like 3 to be repeated only twice (i.e., each item in the first sequence should pair only with one in the second sentence): is there an XQuery "trick" for that in the FLOWR expression?

3 4

Improper use/potential bug error
by Giuseppe G. A. Celano 25 Nov '20

25 Nov '20

Hi, I got an "Improper use? Potential bug?” error (see below) with a (complex) query which actually works when applied to many files (but not to all). This is the query (temporary links): https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/scripts… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/scripts…> applied to all files here: https://git.informatik.uni-leipzig.de/celano/latinnlp/-/tree/master/texts/p… Essentially, I have some words associated to an XPath expression + offsets (see, for example, [1]), which (words) I can retrieve by using substring($xpath, $start, $long) applied to [2]. My goal is to try to identify the text nodes the words are children of, because I want to check whether they are contained in some specific elements. Ciao, Giuseppe [1] https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p…> [2] https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p… <https://git.informatik.uni-leipzig.de/celano/latinnlp/-/blob/master/texts/p…> --- Error: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de <mailto:basex-talk@mailman.uni-konstanz.de> Version: BaseX 9.3.2 Java: Ubuntu, 11.0.9.1 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: Maximum array size reached. at org.basex.util.Array.newSize(Array.java:299) at org.basex.util.Array.newSize(Array.java:288) at org.basex.util.TokenBuilder.add(TokenBuilder.java:267) at org.basex.util.TokenBuilder.add(TokenBuilder.java:252) at org.basex.query.QueryInfo.toString(QueryInfo.java:142) at org.basex.query.QueryContext.info <http://org.basex.query.querycontext.info/>(QueryContext.java:474) at org.basex.query.QueryProcessor.info <http://org.basex.query.queryprocessor.info/>(QueryProcessor.java:272) at org.basex.core.cmd.AQuery.extError(AQuery.java:212) at org.basex.core.cmd.AQuery.query(AQuery.java:130) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:257) at org.basex.core.Command.execute(Command.java:93) at org.basex.gui.GUI.exec(GUI.java:416) at org.basex.gui.GUI.lambda$execute$4(GUI.java:359) at java.base/java.lang.Thread.run(Thread.java:834)

2 3

RESTXQ and posting files
by celano＠informatik.uni-leipzig.de 25 Nov '20

25 Nov '20

Hi, I have a script which takes a path to an xml or txt file as an input and outputs its content (as xml or txt) after a few modifications have been applied. I would like to make the script available as a RESTXQ application, but it is not clear to me how I can specify, in a RESTXQ function, that an xml or txt file should be accepted as an input. I have been able to post by sending the content of an entire file as a string, but I guess there is a specific way to handle files. Ciao, Giuseppe

3 2

where is the "lib" folder?
by Nicholas 23 Nov '20

23 Nov '20

Is there even a lib directory? https://unix.stackexchange.com/a/621089/101935 My understanding is that, for example, a JDBC driver can be "added" to BaseX itself. Perhaps this is incorrect. thanks, Nick

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk