- BaseX-Talk - mailman.uni-konstanz.de

Support for format “XQuery” by CSV parser configuration?
by Markus Elfring 08 Apr '22

08 Apr '22

Hello, I would like to create another database from a CSV file with the software “BaseX 9.7”. It seems that I can not choose the format option “XQuery” on the tab “Parsing” of the dialogue “Create Database”. https://docs.basex.org/wiki/CSV_Module#Conversion Will any further configuration parameters become relevant for the CSV data import? Regards, Markus

2 1

Reversal of Hebrew?
by Patrick Durusau 08 Apr '22

08 Apr '22

I have attached two images, one that shows an unpointed Hebrew word entered in find, and the result as presented in the result window. Notice the "b" and "r" read right to left in Find, but read left to right in the result. If you switch the character positions in the result it would be correct. I'm not sure that searching without pointing but returning pointed results is a feature, but it would be nice to turn that on or off. Including cantillation marks (which are a separate category of mark from vowels). That is: 1) match exact text as entered 2) match text if only consonants match (for both vowels and cantillation) 3) match text if consonants and vowels match but not cantillation 4) match text if some consonants, some vowels, or some cantillation matches (yes, I know about regexes but I can either ask scholars to type in their "find" conditions or I can try to teach them regexes. Which do you think I'm going to pick? If that's a problem we can always create a XQuery library/pod whatever to keep users from having to type the filters in manually. Thanks! Hope everyone is having a great week! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

2 1

Enclosing strings by double quotes for CSV data output
by Markus Elfring 08 Apr '22

08 Apr '22

Hello, I constructed a small XQuery script which contains the following specifications. declare option output:method "csv"; declare option output:csv "header=yes, separator=|, quotes=yes"; … The query execution succeeded (in principle) with the software “BaseX 9.7”. I observed data processing results where displayed columns contained space characters. I would expect that such data should be handled as text strings then. Thus I would expect also that these data should be enclosed by double quotes (according to the safe handling of CSV files). But this did not happen for my software test. Will any additional settings become relevant for this use case? Regards, Markus

5 23

An automated workflow for creating tested and sustainable REST API/RestXQ containers that use BaseX
by Omar Siam 07 Apr '22

07 Apr '22

For the last few years, I tried to integrate BaseX into a CI/CD workflow (the one used by gitlab [1]). My understanding of CI/CD explicitly includes automated tests so I can be as sure as possible I don’t break anything when doing further development. I pondered with using BaseX’ built in unit test framework [2], which I think is very good in many situations, but I was not really satisfied when it comes to checking RestXQ endpoints. The way it needs to be launched [3] is a bit awkward to me as a test for RestXQ and I could not come up with an easy way to automate this. When serving HTML based apps that interact with BaseX as a data store there is also no well-established way to do end-to-end testing, that is to for example programmatically click a button and capture the result, but they do exist for Node.js [4]. Unfortunately my projects almost never allow for more than a few tests due to budget and time constraints. So, I want to make them count. My experience is that such end-to-end tests are what can test the most code in JS and RestXQ with the least amount of test code. Admittedly, if you get an error, you go hunting for the real problem. Similarly, I use Node.js to do kind of an outside integration test for RESTful APIs. I have to admit I have a personal aversion against using many programming languages throughout my day. I try to use two or maximum three on any given day. But there are also other reasons: * For one JS and TypeScript are very likely consumers of a RESTful API. It is hard to escape JS or TypeScript today. * Other people are probably much better at their preferred programming languages but I hope almost anyone has a passive knowledge of JS. If this is true, then tests written in JS are good examples and candidates for translation into another languages test framework if necessary. The (now old) gitlab CI/CD workflow uses herokuish [5] and Heroku buildpacks [6] (but not the actual commercial Heroku service) to achieve two things: * Bundle an application written in some programming language in a container without the author of the program needing to have any knowledge of containers, Dockerfiles etc. The author just needs to know the dependency tools of their programming language well * Having a well-defined way to start up tests and proceed with deployment if the tests succeed or stopping right there. The second point is something that the currently used process for building containers in gitlab, Cloud Native Buildpacks [7], still cannot do because they are only in the planning phase of a container interface for launching tests. As I wanted to bundle everything needed to run and test an application running in the RestXQ environment provided by BaseX for the GitLab CI/CD process, I created a Heroku buildpack for BaseX based on the one for Node.js [8]. As I extended the Node.js buildpack everything is still controlled from package.json in principle. The “engines” definitions are amended to also include a version of BaseX and a version of Saxon to use [9]. Due to inheriting from the Node.js buildpack, JS dependencies for a HTML page that acts as the user interface for some RestXQ endpoints can be easily specified, a “build” script can build some sources written to use the Vue.js, React or angular frameworks. Or maybe only the dependencies of a more elaborate test suite need to be fetched like mocha, chai etc. [10] Also, there is a shell script “deployment/initial.sh” [11] is run so XML data can be fetched and loaded into BaseX at the time the container is built. For example, I use it to pull data from another git repository and execute a BaseX bxs script [12] to load the data, generate indices etc. A “test” script then needs to be defined to run the RESTful API endpoint tests [13] or the end-to-end test [14] automatically. Using this approach, it is possible to also launch any external process for testing so for example BaseX Unit module based tests could be launched additionally or instead of something truly Node.js based. We have decided a while ago that we want to have private git repositories with all their CI/CD at gitlab.com. We provide the compute resources to run all builds and checks using a Kubernetes cluster we own. Back then github just didn’t provide any private repositories for free but we wanted to be present there and so we have all the public projects at github.com. The good thing about public repositories on github.com is the Actions workflows there which run in a VM that provides just about any conceivable programming languages build environment plus container building tools and more. The bad thing is: there is no immediately obvious recipe of how to make good use of that VM on github.com. But if there are container building tools we can run pretty much a copy of the gitlab CI/CD workflow that people over there came up with and that is documented in their AutoDevOps workflow definitions. So I cannot really show you the gitlab CI/CD workflow I made the buildpack for in action, you have to take my word for it or try it yourself on some gitlab instance. Reenabling the old herokuish workflow for build (AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED=false) and referencing my BaseX buildpack (BUILDPACK_URL=https://github.com/simar0at/heroku-buildpack-basex) will integrate BaseX into the AutoDevOps workflow. But you can also see it running in some of our public projects on github.com. There just a pretty stereotypical .github/workflows YAML file is needed and you can use the github.com Actions just fine [15]. That also has the advantage that one can utilize the infrastructure on github.com instead of providing some compute resources. The package might be still a little rough around the edges but I use it now for a few projects of mine (and my institution). I hope someone finds this useful. There are probably a lot of things that need to be fine tuned and I would be happy to work together to get this package better with time. [1] https://docs.gitlab.com/ee/topics/autodevops/ [2] https://docs.basex.org/wiki/Unit_Module [3] https://docs.basex.org/wiki/Unit_Module#Example [4] for one of the solutions existing for Node.js check out https://www.cypress.io/ . That is what I used recently. [5] https://github.com/gliderlabs/herokuish [6] https://devcenter.heroku.com/articles/buildpacks [7] https://buildpacks.io/ [8] https://github.com/simar0at/heroku-buildpack-basex [9] https://github.com/acdh-oeaw/zuludict-app/blob/master/package.json#L6-L12 [10] https://github.com/acdh-oeaw/vleserver_basex/blob/main/package.json#L19-L25 [11] https://github.com/acdh-oeaw/zuludict-app/tree/master/deployment [12] https://github.com/acdh-oeaw/zuludict-app/blob/master/deployment/deploy-zul… [13] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/package.json#L6… [14] https://github.com/acdh-oeaw/vicav-app/tree/master/cypress/integration [15] https://github.com/acdh-oeaw/vleserver_basex/blob/main/.github/workflows/bu… Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

1 0

Incompatibility with Tomcat 10
by Marc Balston 05 Apr '22

05 Apr '22

Hello, I've been successfully running BaseX on Amazon AWS using Tomcat 8.5, but I've been looking to upgrade my server to Tomcat 10.0.2.However, I've run into an issue: BaseX does not startup and from the logs the cause appears to be a missing class: java.lang.NoClassDefFoundError: javax/servlet/http/HttpSessionListener I believe this is because there was a change between Tomcat 9.x and Tomcat 10 which led to a renaming of the javax.* package to jakata.* (related to a change in the servlet API version) see here for a discussion: https://stackoverflow.com/questions/66711660/tomcat-10-x-throws-java-lang-n… Is my understanding correct and is this a known issue?If it is a genuine incompatibility, it would be useful to mention this in the documentation (Web Application - BaseX Documentation) ThanksMarc Marc Balston

2 5

Support for performing queries on query results
by Markus Elfring 05 Apr '22

05 Apr '22

Hello, XQuery scripts can be used also to compute values from selected XML data structures. Query results can be exported then into other file formats. But I would like to reuse query results for further queries directly. The SQL standard is providing named views for this purpose. How will the support evolve for such data processing requirements with XML tools? Regards, Markus

6 18

copy-paste, can't copy in BaseX and then paste into Emacs (Debian)
by Patrick Durusau 04 Apr '22

04 Apr '22

Hello, I tried to copy a string out of results with the copy command on the drop down but don't seem to be able to paste into Emacs. Trying to be good and document my steps but if I can't copy-paste, that's going to be tedious. Suggestions? Thanks! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

3 8

Typo in basexgui.bat? since 9.6.4? Also in 9.7
by Patrick Durusau 04 Apr '22

04 Apr '22

Greetings! Friday afternoon is a bad time to open a Windows VM for anything but while pursuing a question about BaseX 9.7 and Windows 10,... basexgui fails with: "Windows cannot find 'javaw.' Make sure you typed the name correctly, then try again." basexgui.bat reports: "start javaw -cp "%CP%" ..." All the other .bat file report "start java ...." BTW, Windows is very cranky about running "unknown publisher" software. Hope everyone has already started a great weekend and will tend to this next Monday. Patrick PS: I'm still running 9.4.6 on Debian and report the same from the files there. 9.7 was on Windows 10. -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

2 3

keeping relative paths in the document-uri?
by Graydon Saunders 31 Mar '22

31 Mar '22

Hello -- An approach that uses db:create and file:descendant doesn't seem to work; the file paths have all been truncated to the name part of the path in the document-uri property of the individual documents in the DB. I'm using 9.6.4 with Oracle Java 1.8.0_211 on a Windows 10 machine. (I have no control over any of the environment.) Is there a way to keep the full path (relative or absolute) in the document uri property? I'm trying to inspect files with identical names in different directories, so I need the directory name to tell the files apart. Thanks! Graydon

3 2

Simple delete bug
by Jonathan Robie 30 Mar '22

30 Mar '22

Expressions like this fail silently without raising an error in BaseXGUI: delete //m/@n Of course, it should be: delete nodes //m/@n But there should be an error to remind me ;-> Jonathan

2 2

Serialization ... one attribute per line ...
by Jonathan Robie 29 Mar '22

29 Mar '22

I have a set of syntax trees that I would like to serialize with one attribute per line, like this: https://github.com/biblicalhumanities/greek-new-testament/blob/master/synta… The current serialization looks more like this: https://github.com/biblicalhumanities/greek-new-testament/blob/master/synta… Is there a way to get there using just serialization parameters, or do I need to do something fancier? Jonathan

1 0

fn:count() performance
by Josselin Morvan 29 Mar '22

29 Mar '22

Hi everyone, We are experimenting a small issue with the count() function, and I wanted to know if you have any idea to reduce the response time of the server : We have a database containing almost 5.000 descriptions of expert reports from the 18th century. As a report can take place over several years, we want to list all the years mentioned in the reports and then count for each year how many reports we have. Our goal with this query is to produce a filter by year for our web application. But, if the first part of the query is quite fast, it is not the case for the second part… here is a simplified sample of our code : xquery version "3.1"; declare default element namespace "xpr" ; (: to create the xprDB ;) db:create('xpr', 'https://raw.githubusercontent.com/anrExperts/data/master/db/xpr.xml') :) let $years := fn:distinct-values(db:open('xpr')/xpr/expertises/expertise/description/sessions/date[@when castable as xs:date]/fn:year-from-date(@when)) for $year in $years return $year || ' : ' (:this first part of the query is quite fast 0.09sec on my old computer:) || fn:count(db:open('xpr')/xpr/expertises/expertise[description/sessions/date[fn:matches(@when, xs:string($year))]]) (:it takes 5sec to execute the second part of the query:) Do you see anything we are doing wrong or we can improve to reduce the server response time ? We thank you in advance! Best, Josselin

2 2

(no subject)
by Charles Bearden 29 Mar '22

29 Mar '22

I have loaded all the Pubmed baseline XML records into a series of 20 BaseX databases, 55 or 56 of the baseline files per database, each of which is about 12GB in size and has between 520 and 530 million nodes in 55 or 56 documents. Text, Token, and Attribute indices are enabled, but with 6GB RAM allocated to the Java VM it would not create a full text index. Each of the 55 or 56 documents has 30000 article records in it under a root PubmedArticleSet element. I typically use basexgui for interactive work and basex for scripted loads & queries, and I allocate 6G to the Java VM in each case: BASEX_JVM="-Xmx6g $BASEX_JVM" I'm exploring ways to search the data in a moderately performant way, starting with the realtively simple lookup by PubMed ID: /PubmedArticleSet/PubmedArticle[MedlineCitation/PMID[text()=$pmid]] I generated a seriers of five index XML files that pair PubMed IDs with database names, like so: <index> <entry> <dbname>pmed_baseline_a</dbname> <pmid>579614</pmid> </entry> … </index> Each file contains entries for four of the 20 BaseX databases. I loaded the five files into a single database. My hope was that I could quickly lookup the name of the database that contained a record by that record's PMID, and that I could then open that collection and quickly obtain that record, but it isn't working the way I had hoped. If I query the index database by PMID, I get the answer in 156ms: let $pmid := '22345065' let $icoll := collection('idx_pmed_baseline') let $pmid_lookup := $icoll/index/entry/pmid let $entry := $pmid_lookup[text()=$pmid] let $dbname := $entry/parent::entry/dbname/text() return $dbname (: returns 'pmed_baseline_s' :) If I open and query that collection by XPath for that PMID, also get the answer quickly, in about 420ms: let $coll := collection('pmed_baseline_s') let $pmid := '22345065' let $wanted := $coll/PubmedArticleSet/PubmedArticle[MedlineCitation/PMID/text()=$pmid] return $wanted (: returns desired XML record :) But if I combine the code to run in a single execution, it takes about 40s: let $pmid := '22345065' let $icoll := collection('idx_pmed_baseline') let $pmid_lookup := $icoll/index/entry/pmid let $entry := $pmid_lookup[text()=$pmid] let $dbname := $entry/parent::entry/dbname/text() let $coll := collection($dbname) let $wanted := $coll/PubmedArticleSet/PubmedArticle[MedlineCitation/PMID/text()=$pmid] return $wanted I feel like I must be doing some simple thing wrong, but the only difference I see in my code between the two separate steps and the single execution version is that I'm passing the db name in a variable instead of as a string literal to the collection() function, and I'm running the whole thing in a single execution. Note that before each execution of an XQuery, I exited basexgui and restarted it to avoid any caching effect in memory at least. The VM I'm running on is modest (spinning drives in RAID 1, four modest AMD CPU cores, dynamic memory growth up to 32GB). But these factors would not explain the difference in speed between the two steps in separate executions and both steps in a single execution. Can anyone point out what I'm doing wrong? And is there a better way to go about this? Many thanks & all the best, Chuck

2 5

attribute-range()
by Johannes Bauer 29 Mar '22

29 Mar '22

Hello, when I use the db:attribute-range function with the attribute name filter I always get all attributes: db:attribute-range("db", "1", "1000", "id") I would expect that it only returns the "id" attributes. It seems the attribute name does not matter at all. The results are always the same even for a bogus attribute name. Regards Johannes

2 1

BaseX for Newbies Tutorial
by Tamara Marnell 28 Mar '22

28 Mar '22

Hello everyone, When I began developing our finding aids website last summer, I was a complete newbie to XML databases. I lacked the basic knowledge needed to understand the documentation, or even how to navigate it. Even Paul Swennenhuis's "BaseX for Dummies" (2014) was too advanced for me to follow. "FLWOR?" "Indexes"? I just want to do a keyword search of some files; why is this so hard?! After extensive Googling, Stack Overflowing, and reading every conversation in this list, I finally got a handle on the basics and completed our website. I've written what I learned in a foundational primer, "BaseX for Newbies," which contains basic definitions of concepts like "client/server architecture" and "modules," XQuery examples using U.S. Library of Congress finding aids, and solutions for problems that tripped me up: https://drive.google.com/file/d/19A5twBhA6OT3YdlcII7fazlwXewUSNkX/view -Tamara -- Tamara Marnell Program Manager, Systems Orbis Cascade Alliance (orbiscascade.org <https://www.orbiscascade.org/>) Pronouns: she/her/hers

3 2

RBaseX-client
by Ben Engbers 25 Mar '22

25 Mar '22

Hi, Version 1.1.1 is finally stable and available at CRAN (https://cran.r-project.org/package=RBaseX). Hopefully this blog "https://r-posts.com/rbasex-a-basex-client-written-in-r/" will contribute to more support for BaseX from the R community. Ben Engbers

1 0

Storing maps in BaseX
by Jonathan Robie 24 Mar '22

24 Mar '22

Is there a way to store a map in a database and use it in other queries without recomputing it each time? Jonathan

3 5

BaseX 9.7 • Ukraine Edition
by Christian Grün 24 Mar '22

24 Mar '22

Dear all, Just next to us, millions of people are currently being torn out of their home country, and we desperately hope that the situation in Ukraine will calm down soon. For those of you who are planning to donate money to us in the upcoming weeks, we kindly ask you this time to redirect it to Médecins Sans Frontières or any NGO of your choice. – We are back to provide you with a new and fresh version of BaseX. It comes with a big bunch of advanced performance tweaks and optimizations and some new XQuery features: • Support for xsl:messages via xslt:transform-report: • Support for formatting integers and dates in French • XSD validation: Improved support for XML catalogs • Remove namespaces from elements via util:strip-namespaces For a more comprehensive list of added and updated features, look into our documentation (docs.basex.org) and check out the GitHub issues (github.com/BaseXdb/basex/issues). And one more pleasant announcement: Today, our company BaseX GmbH Our celebrates its 10th anniversary! Thanks to all of you who have keeping us busy with challenging projects. Rest assured we’ll be happy to continue to give you advice and support for the upcoming decades. All the best, Your BaseX Team

6 6

Local deployment of a database
by Jonathan Robie 24 Mar '22

24 Mar '22

We are creating server instances, but we also have users who want to work locally. Ideally, I would like to be able to create an installer that would add BaseX and a set of databases to a user's system. How hard would it be to do that? Jonathan

3 5

GUI syntax highlighting
by marc＠crosseyed.nl 23 Mar '22

23 Mar '22

The syntax highlighting doesn't change when changing the look and feel in the BaseX GUI (9.6.2). I noticed this also happens in earlier versions. After changing to a dark background this results in blue on black, for instance. This happens with the JTattoo lib, but also GTK under a dark Linux theme. This occurs in Linux, all kinds of distro's, both Debian and Fedora based. Open JDK 11.0.6. In BaseX GUI 8.6.x all is fine, running under the same conditions. Regards, Marc

3 2

Windows, WSL, and BaseX
by Jonathan Robie 22 Mar '22

22 Mar '22

I installed Windows System for Linux and installed BaseX on it, and also installed BaseX on Windows. Unfortunately, the two do not share databases. Is there a way to make them do so? Or a way to run basexgui from WSL? Or a better question that I should ask if I want to work productively on BaseX under Windows and am more accustomed to developing under Linux? Jonathan

2 1

Anticipating more complex data to follow
by Patrick Durusau 22 Mar '22

22 Mar '22

Greetings! While working on some large but fairly simple data, I encountered BaseX using the underscore "_" as an element name when importing JSON. Not leaving well enough alone for present purposes, I encountered this post about anonymous objects in MarkLogic and how they resolve the problem. https://developer.marklogic.com/recipe/query-array-anonymous-objects/ Is there already a recommended solution for this problem? Or should I wait until I actually encounter it and ask again? Hope everyone is at the start of a great week! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

3 5

OpenJDK vs. Oracle JDK versus ...
by Jonathan Robie 22 Mar '22

22 Mar '22

I am setting up a Windows 11 box. Which Java is best for BaseX? Jonathan

5 6

XML Parsing Error: syntax error, Location: http://localhost:8984/rest/oshb?query=count(.), Line Number 1, Column 1:
by Patrick Durusau 16 Mar '22

16 Mar '22

Greetings! What puzzles is the correct count is returned, despite the error message, with the cursor under the first digit. Looking in the response, it appears that "favicon.ico" is missing and I suspect is the cause of the error. But I wanted to confirm that assumption. Thanks! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

2 2

Dumb question: Adding files to BaseX database
by Patrick Durusau 14 Mar '22

14 Mar '22

Greetings! I'm using db:add to add files to a BaseX database, but am I just missing how to do that in the GUI? Thanks! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

3 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk