- BaseX-Talk - mailman.uni-konstanz.de

Way to Dynamically import a module?
by Buddy Kresge 14 Mar '22

14 Mar '22

I was wondering if in BaseX there is a way at run time to import a .xqm module, outside of the traditional "import module" statement. Thanks in advance for any guidance you can provide. Buddy

2 3

Fuzzy matching in ft:search for terms with spaces
by Tamara Marnell 10 Mar '22

10 Mar '22

Hi everyone, The results for ft:search() with the "fuzzy" option set to "true" are unexpected for phrases with spaces in them. Usually "fuzzy" set to "false" will return fewer search results than when it's set to true, but I'm finding the opposite for phrases. Sometimes ft:search() doesn't return known exact matches for a phrase unless "fuzzy" is "false." For example, in my development environment I have a document titled Test Finding Aid. I'm tokenizing all text() nodes in an index database and verified that "test finding aid" is a phrase in the node included for full-text. <tokens> test xml test finding aid 1964 2002 test finding aid finding aid prepared central oregon community college 2008 [etc.] </tokens> When I ft:search() for the complete phrase "test finding aid" as one term with "fuzzy" set to "true," I get 4 results, none of which are the document titled Test Finding Aid with the index entry above. They're all results with "West" in the title that happen to have "west finding aid" in them when tokenized. But when I set "fuzzy" to "false," I get 1 result, and it is for the document titled Test Finding Aid. I expect all exact results to be included in fuzzy results, but other searches show that the exact results are often more numerous than the fuzzy ones for phrases. league women voters (as 3 distinct terms) Fuzzy true: 430 results Fuzzy false: 373 results (expected) league women voters (as 1 term) Fuzzy true: 281 results Fuzzy false: 299 results (not expected) pacific coast (as 2 distinct terms) Fuzzy true: 2,551 results Fuzzy false: 1,866 results (expected) pacific coast (as 1 term) Fuzzy true: 893 results Fuzzy false: 935 results (not expected) Does anyone know what's causing this, and what can be done about it? -Tamara -- Tamara Marnell Program Manager, Systems Orbis Cascade Alliance (orbiscascade.org <https://www.orbiscascade.org/>) Pronouns: she/her/hers

2 3

How to return/use the value of a nested counter?
by Ben Engbers 10 Mar '22

10 Mar '22

Hi, I have a collection of 740 documents with the following structure: <?xml version="1.0" encoding="utf-8"?> <officiele-publicatie xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://technische-documentatie.oep.overheid.nl/schema/op-xsd-2012-1"> <metadata> <meta name="OVERHEIDop.externMetadataRecord" scheme="" content="https://zoek.officielebekendmakingen.nl/h-tk-20202021-102-2/metadata.xml" /> </metadata> <handelingen> <agendapunt> <spreekbeurt nieuw="nee"> <tekst status="goed"> <al-groep> <al>Allereerst hebben we het traditionele mondelinge vragenuur. </al> </al-groep> </tekst> </spreekbeurt> <spreekbeurt nieuw="ja"> <tekst status="goed"> <al-groep> <al>Voorzitter. Het was altijd al een eer om hier te staan.</al> </al-groep> <al-groep> <al>De vragen die ik ga stellen, gaan over stikstof.</al> </al-groep> <al-groep> <al>We zijn allemaal 100 kilometer per uur gaan rijden, maar er is nog geen gram ammoniak uit de veehouderij minder uitgestoten.</al> </al-groep> </tekst> </spreekbeurt> </spreekbeurt> <spreekbeurt nieuw="nee"> <tekst status="goed"> <al-groep> <al>U heeft helaas maar één vraag, meneer Ephraim, als Groep Van Haga.</al> </al-groep> <al-groep> <al>Ik wil de minister bedanken voor haar beantwoording.</al> </al-groep> </tekst> </spreekbeurt> </agendapunt> </handelingen> </officiele-publicatie> I am trying to concatenate all the //<al> childs from <spreekbeurt> elements. Together with an ID that I construct from //meta/@content and a counter for <spreekbeurt>, I want this output: 20202021-102-2, 1, Allereerst ... 20202021-102-2, 2, Voorzitter... + De vragen ... + We zijn ... 20202021-102-2, 3, U heeft ... + Ik wil.. I expected that the following XQuery-statemnt would do it. import module namespace functx = "http://www.functx.com"; for $Debat at $CountOuter in collection("Parliament") (: where $CountOuter <= 3:) let $debate-id := fn:analyze-string( $Debat/officiele-publicatie/metadata/meta/@content, "(\d{8}-\d*-\d*)")//fn:match/*:group[@nr="1"]/text() order by $debate-id for $Beurt at $CountInner in $Debat let $tekst := $Beurt//spreekbeurt//al/text() return($debate-id, $CountInner, $tekst) Instead it returns: 20202021-102-2, 1, Allereerst ...+ Voorzitter... + De vragen ... + We zijn ... + U heeft ... + Ik wil.. How can I use the value of $CountInner? Ben

2 3

XQuery versus XSL
by Ben Engbers 09 Mar '22

09 Mar '22

Hi, I have a collection of 740 documents with the following structure: <?xml version="1.0" encoding="utf-8"?> <officiele-publicatie xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://technische-documentatie.oep.overheid.nl/schema/op-xsd-2012-1"> <metadata> <meta name="OVERHEIDop.externMetadataRecord" scheme="" content="https://zoek.officielebekendmakingen.nl/h-tk-20202021-102-2/metadata.xml" /> </metadata> <handelingen> <agendapunt> <spreekbeurt nieuw="nee"> <tekst status="goed"> <al-groep> <al>Allereerst hebben we het traditionele mondelinge vragenuur. </al> </al-groep> </tekst> </spreekbeurt> <spreekbeurt nieuw="ja"> <tekst status="goed"> <al-groep> <al>Voorzitter. Het was altijd al een eer om hier te staan.</al> </al-groep> <al-groep> <al>De vragen die ik ga stellen, gaan over stikstof.</al> </al-groep> <al-groep> <al>We zijn allemaal 100 kilometer per uur gaan rijden, maar er is nog geen gram ammoniak uit de veehouderij minder uitgestoten.</al> </al-groep> </tekst> </spreekbeurt> </spreekbeurt> <spreekbeurt nieuw="nee"> <tekst status="goed"> <al-groep> <al>U heeft helaas maar één vraag, meneer Ephraim, als Groep Van Haga.</al> </al-groep> <al-groep> <al>Ik wil de minister bedanken voor haar beantwoording.</al> </al-groep> </tekst> </spreekbeurt> </agendapunt> </handelingen> </officiele-publicatie> I want to experiment with textmining and for these experiments, it would be usefull if for every <spreekbeurt>, all <al>/text() elements were concated.The first option is to use XQuery for concatenating. Another option is to use XSL to transform the original documents to the following structure: <?xml version="1.0" encoding="utf-8"?> <officiele-publicatie xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://technische-documentatie.oep.overheid.nl/schema/op-xsd-2012-1"> <metadata> <meta name="OVERHEIDop.externMetadataRecord" scheme="" content="https://zoek.officielebekendmakingen.nl/h-tk-20202021-102-2/metadata.xml" /> </metadata> <handelingen> <agendapunt> <spreekbeurt nieuw="nee"> <tekst status="goed"> Allereerst hebben we het traditionele mondelinge vragenuur. </tekst> </spreekbeurt> <spreekbeurt nieuw="ja"> <tekst status="goed"> Voorzitter. Het was altijd al een eer om hier te staan. De vragen die ik ga stellen, gaan over stikstof. We zijn allemaal 100 kilometer per uur gaan rijden, maar er is nog geen gram ammoniak uit de veehouderij minder uitgestoten. </tekst> </spreekbeurt> </spreekbeurt> <spreekbeurt nieuw="nee"> <tekst status="goed"> U heeft helaas maar één vraag, meneer Ephraim, als Groep Van Haga. Ik wil de minister bedanken voor haar beantwoording. </tekst> </spreekbeurt> </agendapunt> </handelingen> </officiele-publicatie> Question: What are the pros and cons of both methods? Is it difficult to do this in XSL (I have only used very simple transformations)? Ben

2 1

Edit to https://docs.basex.org/wiki/Database_Server and question
by Patrick Durusau 08 Mar '22

08 Mar '22

Greetings! I edited https://docs.basex.org/wiki/Database_Server to change: Add the stop keyword to gracefully shut down the server. to: Use |basexserverstop| or |basexserverstop.bat| to gracefully shut down the server. Question, in the first paragraph following this list, the text reads: "Pressing |Ctrl+c| will close all connections and databases and gracefully shut down the server process." True? False? Somewhere in between? If true it should be the third alternative after basexserverstop.bat, but then why the scripts if we can always Ctrl+c? What am I missing? Thanks! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

2 3

Re: [basex-talk] Keyboard issue on MacBook (Apple M1 Pro)
by Alexander Holupirek 07 Mar '22

07 Mar '22

> Hello, I'm having a small but annoying issue with keyboard input in the BaseX GUI on a new MacBook (M1 Pro). I tend to use the U.S. International keyboard layout for ease of typing accented characters: e.g., using the apostrophe/quotation key as a dead key when typing é or ü. However, with this keyboard layout, the apos/quot key doesn't register at all in the BaseX GUI. It works as expected everywhere else. I've tried switching fonts, but that doesn't seem to help. Any ideas? Thanks in advance! Tim Hi Tim, I just tried what you described on a M1. Using 'International - PC' as keyboard layout and pressing three keys, namely 'SHIFT' + 'QUOT' + 'A' gives me the following behaviour: Terminal-App (and other apps): App | Keys | Description -------------|-----------|----------------------------------------------- Terminal-App | SHIFT | (nothing happens) | " | a double quot sign (") in a (yellow) highlighted square | a | the german umlaut a (ä) on default background appears In BaseXGUI: App | Keys | Description -------------|-----------|----------------------------------------------- BaseXGUI | SHIFT | (nothing happens) | " | (nothing happens) | a | the german umlaut a (ä) on default background appears So it seems that the indication / user feedback is missing in BaseXGUI, but the umlaut character can nevertheless be entered. Do you experience the same? Alex

2 3

Results of some experiments for improving full-text search speeds
by Tamara Marnell 05 Mar '22

05 Mar '22

Good morning--I hope everyone in Europe is safe and well. This week I've been working on improving the speed of full-text searches in our website, and I wanted to share what I've found to be helpful and not so helpful. Our project: We run a website of finding aids from 47 archival repositories in the western US. Archivists submit their finding aids as EADs through a CMS that indexes each document for full-text searching in BaseX, with other custom indexes for facets like subject, material type, etc. https://archiveswest.orbiscascade.org/ Our issue: We need to perform full-text searches that match documents with all of the terms in the query. The BaseX full-text index and ft:search() function work at the text node level. So we need to use the mode "any word," then group by a unique identifier in the root and run an ft:contains() with the mode "all words," like this: for $db_id in tokenize($d, '\|') for $result in ft:search($db_id, $terms, map{'mode':'any word','fuzzy':$f}) let $ead := $result/ancestor::ead let $ark := normalize-space($ead/eadheader/eadid/@identifier) (: Other node values fetched here :) group by $ark where ft:contains(string-join($result, ' '), $terms, map{'mode':'all words','fuzzy':$f}) (: Custom ranking calculations based on other node values performed here :) return <ark>{$ark}</ark> Most users of the site are interested in their repositories only, and they use only a few words to find known collections, so most of the time results are delivered in less than 5 seconds. If users want to search for multiple terms across all 47 repositories, though, the search was taking 30-40 seconds. What I DIDN'T find helpful: Modifying the XQuery 1. Performing a separate search for each term and mapping to fetch with db:open-id() later, instead of searching for all terms at once and grouping. This was slightly slower: for $db_id in tokenize($d, '\|') let $results := fold-left( (: Get the ft:search() results as a map of node IDs and results :) ) for $node_id in map:keys($results) let $ead := db:open-id($db_id, $node_id) (: Ranking etc. :) 2. For multi-term queries, creating a temporary table to search instead of the complete full-text indexes of all databases. a. Create a database like temp123456 with a full-text index. b. For the first term, loop through all databases add the documents to temp123456 with db:add($temp_id, db:path($ead)) and optimize. c. For subsequent terms, map the results of a full-text search temp123456 instead of looping through all 47 databases, and then loop through all EADs in the temp database and remove the ones that don't match with... if (not(exists($temp_results(db:node-id($ead))))) then db:delete($temp_id, db:path($ead)) d. For the final term, return the usual ranked <ark /> values, then drop the database. This didn't work because the optimization of the temporary databases took a long time--25-30 seconds for a term with 1K document results, according to the BaseX logs. It would work in theory only if the existing full-text index could be parceled out for subsequent terms, instead of being freshly rebuilt. What I DID find helpful: Changing the server We use Amazon Web Services, so I experimented with putting the project on different instance types: General, Memory-Optimized, and Compute-Optimized. The contenders: General t2.medium: 2 vCPU, 4 GB memory General t3.large: 2 vCPU, 8 GB memory Compute-optimized c6g.xlarge: 4 vCPU, 8 GB memory Memory-optimized r6g.large: 2 vCPU, 16 GB memory At the time of testing we were on t2.medium, which is the most affordable and the smallest I thought would be sufficient. For one-term queries, the difference in speed was not that much across the instance types. For longer queries, the number of virtual CPUs and memory made a big difference. Query: native american tribes in oregon t2.medium: 55 seconds t3.large: 30 seconds r6g.large: 22 seconds c6g.xlarge: 21 seconds Memory: - Increasing from 4 GB on the t2.medium to 8 GB on the t3.large and c6g.xlarge nearly doubled the speed of my queries, even for queries of 2 words like "women's march" (20 seconds on t2.medium, 8-10 seconds on the others). - Increasing from 8 GB to 16 GB on a memory-optimized instance didn't affect the speed. The java process used only 10% of the available memory, versus 50-60% on other instance types, but the speed wasn't any faster, even increasing the maximum heap size in the basexhttp startup script. This could be because heap size isn't the right property to adjust. CPU: - Increasing from 2 vCPU to 4 vCPU on a compute-optimized instance improved speeds only for extreme queries of 4 words or more. papers t3.large: 6 seconds c6g.xlarge: 7 seconds family papers t3.large: 18 seconds c6g.xlarge: 18 seconds adams family papers t3.large: 15 seconds c6g.xlarge: 14 seconds adams family papers utah t3.large: 25 seconds c6g.xlarge: 21 seconds For most queries our users are performing, which are short and restricted to one repository, it's a negligible difference that didn't justify the increase in cost for a compute-optimized instance ($40 per month vs. $60 per month). -Tamara -- Tamara Marnell Program Manager, Systems Orbis Cascade Alliance (orbiscascade.org <https://www.orbiscascade.org/>) Pronouns: she/her/hers

3 7

Keyboard issue on MacBook (Apple M1 Pro)
by Tim Thompson 04 Mar '22

04 Mar '22

Hello, I'm having a small but annoying issue with keyboard input in the BaseX GUI on a new MacBook (M1 Pro). I tend to use the U.S. International keyboard layout for ease of typing accented characters: e.g., using the apostrophe/quotation key as a dead key when typing é or ü. However, with this keyboard layout, the apos/quot key doesn't register at all in the BaseX GUI. It works as expected everywhere else. I've tried switching fonts, but that doesn't seem to help. Any ideas? Thanks in advance! Tim -- -- Tim A. Thompson Discovery Metadata Librarian Yale University Library 344 Winchester Street New Haven, Connecticut 06511 (203) 432-4189 (office) (201) 423-9972 (mobile) www.linkedin.com/in/timathompson timothy.thompson(a)yale.edu

1 0

have a query report a warning?
by Graydon Saunders 04 Mar '22

04 Mar '22

Hello all -- XPath has an error function, https://www.w3.org/TR/xpath-functions-31/#func-error which stops processing, which isn't wanted when it's just a warning. ("this thing has problems; on to the next thing" shouldn't stop processing.) XPath has a trace function, https://www.w3.org/TR/xpath-functions-31/#func-trace which emits text, but which I think is like xsl:message and will write to STDOUT rather than STDERR. This leaves me thinking I've missed something -- how should I have a query give a warning? I'm not finding something BaseX specific; I'm fairly sure neither error() nor trace() are correct. Any suggestions? Thanks! Graydon

2 2

validate:xsd-info agains more than one schema
by Arantza Etxebarria Pablo 04 Mar '22

04 Mar '22

Dear Community, We have been trying to validate a file using the function *validate:xsd-info*($input as item(), $schema as item()?) as xs:string* against several different schemas and, we don't know how to send the parameter “schema” with more than one schema for this function. We have tried to declare the variable *schema *like this : let $schema:= " http://inspire.ec.europa.eu/schemas/bu-base/4.0/BuildingsBase.xsd http://schemas.opengis.net/gml/3.2.1/gml.xsd" But it returns this error message: XML document 'example.gml': -1:-1: schema_reference.4: Failed to read schema document ' http://inspire.ec.europa.eu/schemas/bu-base/4.0/BuildingsBase.xsd http://schemas.opengis.net/gml/3.2.1/gml.xsd' , because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>. I have tried this function when no schema is given and it takes the contain of schemaLocation attribute, and in this case, is able to validate more than one schema. But I need to pass the schemas through a variable Could any of you point us in the right direction? Thanks -- *Arantza Etxebarria *

3 2

Re: [basex-talk] feature request: opening database at arbitrary file path
by Eric Levy 03 Mar '22

03 Mar '22

On Tue, 2022-02-22 at 23:10 +0100, Christian Grün wrote: > > The other contributor's comments were deeply rooted in particular > > assumptions that stood in stark contrast to the scenario I had > > described. > > It always depends on the way you interpret others’ comments. But > we're getting circular again (there must be a reason for it ;). > > > To me, "embedded" and "standalone" carry the same > > implication in the context of a database > > I see. With standalone, we want to indicate that the application > stands “on its own” and needs no external accessors. In contrast, > something that's embedded is fully dependant on other applications. > > In other words, if we use BaseX embedded, we import it as Java > library. If we use it as standalone processor, we access it > externally, similar to other installed applications and along with > all their idiosyncrasies. > > Would Java be an option for you? I understand the distinction. It seems the issues I originally raised would be the same in both cases. The difference is whether the application would be built as a collection of scripts versus a full Java application. I had planned to begin with Bash scripts calling the command-line database application, and then migrate to a more integrated application model (e.g. Java application using a library) only if required by the evolving needs. Frankly, I would hope to avoid the latter for the foreseeable future, as both the overall application complexity and maintenance resources are rather limited. However, as I explained, the overriding concern is whether the databases may be maintained inside of projects' folders without any dependency on resources elsewhere on the file system.

3 11

How Best to Determine What BaseX Is Doing When It Pegs the Processor?
by Eliot Kimber 03 Mar '22

03 Mar '22

I have a RESTXQ page that makes a query call to construct a report and then displays it. When I run this page something is triggering BaseX to peg the processor and it basically never returns from whatever it is doing. Using proc:dump() I can see that the call to construct the report runs and returns the expected value—this takes about 3 seconds, and then the page never returns and top shows that the java process for the BaseX HTTP server is pegged at 100%. The behavior is consistent so I know it must be something I’m doing or the effect of some configuration setting I have but no idea what might be going on. I get the behavior both on my development machine and on my production server, so it must be something in my code or some reproducible issue with the databases I’m managing. I’ve tried connecting the BaseX HTTP server to the Eclipse debugger, which works, but I’m not sure where I should be focusing my attention in the code. Turning on the -d flag isn’t that helpful either as the log is flooded with exceptions from BaseX trying to load Java classes for my XQuery modules, which swamps any more detailed debugging log messages and what messages I see appear to be entirely about the HTTP interaction, not what’s happening in the core BaseX server. Is there a particular place in the code I can focus on in the interactive debugger that would be a likely source of the behavior or another debugging technique I haven’t considered? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 4

Re: [basex-talk] Declaring a variable in bxs that carries over to called xquery scripts
by ETANCHAUD Fabrice 02 Mar '22

02 Mar '22

Hi Patrick, I don't think bxs scripts have something like variables (but I may have missed the point with all the work BaseX team achieves). Bindings are way to parametrize XQuery scripts, and you can indeed set external variables' value in bxs, or on command line. Best regards, Fabrice Here are a few examples : https://docs.basex.org/wiki/Options#BINDINGS Options - BaseX Documentation<https://docs.basex.org/wiki/Options#BINDINGS> This page is linked from the Getting Started Section.. The options listed on this page influence the way how database commands are executed and XQuery expressions are evaluated. Two kinds of options exist: Global Options are valid for all BaseX instances in the same JVM. This is particularly relevant if you are working with the client/server architecture. docs.basex.org https://docs.basex.org/wiki/Command-Line_Options Command-Line Options - BaseX Documentation<https://docs.basex.org/wiki/Command-Line_Options> Standalone []. The following options are available for the standalone Command-Line Client: $ basex -h BaseX [Standalone] Usage: basex [-bcdiIoqrRstuvVwxz] [input] [input] XQuery or command file, or query string -b<args> Bind external query variables -c<input> Execute commands from file or string -d Toggle debugging output -i<input> Bind file or database to context -I<input> Bind input string ... docs.basex.org ________________________________ De: BaseX-Talk de la part de Patrick Durusau Envoyé: Mercredi 02 mars 2022 01:38 À: basex-talk(a)mailman.uni-konstanz.de Objet: [basex-talk] Declaring a variable in bxs that carries over to called xquery scripts Greetings! What I thought would be a simple task has turned out to not be so simple. I wrote: SET CHOP false SET EXPORTER indent=no,omit-xml-declaration=no SET BINDINGS $ndb=Ruth CREATE DB $ndb ./sources/Ruth.xml In BaseX 9.6.4, and when testing it I got this result: CHOP: false EXPORTER: indent=no,omit-xml-declaration=no BINDINGS: $ndb=Ruth Database '$ndb' created in 544.5 ms. What? Why isn't the mapping of Ruth to the variable $ndb effective in the bxs file? Moreover, there are query files that follow in the bxs file that I want to have resolve $ndb to Ruth (the desired name of the database). 1) How should I declare a variable in a bxs file? 2) In such a way that it carries over to following xquery files? BTW, I've seen this posting by Christian: https://stackoverflow.com/questions/28018354/how-to-run-a-xquery-with-xml-f… but it doesn't seem to be working for me. What am I missing? Thanks! Patrick <https://twitter.com/servicenow>-- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

1 0

CREATEFILTER Setting Not Being Applied
by Eliot Kimber 01 Mar '22

01 Mar '22

In my basex/.basex file I have this local option: # Local Options CREATEFILTER=*.xml,*.dita,*.ditamap I have three BaseX instances running. And in the DBA panel for an instance not running on the default port (i.e., on port 10894) I see this in the Local Options column: CREATEFILTER *.xml,*.dita,*.ditamap SKIPCORRUPT ✓ However, if I do this from the Query console of that same server: db:create(‘mydatabase’, ‘/path/to/docs’) The resulting database only reflects .xml files, not .dita or .ditamap files. I’m wondering if this is a bug in how CREATEFILTER is being applied or my misunderstanding of the implications for CREATEFILTER? This is running 9.6.4 on Centos. If I specify createfilter directly on the db:create() call then of course it works as expected. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 3

Identify Unparseable XML Files in File System
by Eliot Kimber 26 Feb '22

26 Feb '22

My validation server loads all the XML documents it can from the file system. I have “skipcorrupt” set to true so that non-well-formed documents don’t fail the database creation or update attempt. However, as part of my validation services, I need to be able to report those documents that are not well formed and therefore didn’t make it into the database. I’m wondering what the easiest/most efficient way to do that would be within BaseX? I’m working with on the order of 36K files. The files as stored in BaseX have the same path and filename as the files on the file system relative to the directory I import from, so the correlation between files and docs in BaseX is direct and simple. One easy solution would be to simply get the disjoint of the list of files on the file system and the docs in the database and then attempt to load each file to verify that is in fact unparseable and not just not-yet-imported. But maybe there’s a more direct way that I’ve overlooked? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

Integration with LDAP for Authentication?
by Eliot Kimber 26 Feb '22

26 Feb '22

For my validation application (now christened “Project Mirabel” after the Encanto character Mirabel Madrigal), I want to add user validation and authentication via connection to our internal LDAP server I think the solution is to build a custom version of BaseX server that replaces the build-in base User implementation with one that uses JNDI to talk to the LDAP server for user validation and authentication. Looking at the Java code I’m not quite sure what the best way to go about this is but I think it’s to provide a library that provides my custom LDAP-using User classes and then replace the built-in user module implementation with one that uses my User classes rather than the base User classes. Am I on the right track? I didn’t see anything that looked like a way to do a run-time configuration of the classes used for authentication. I kind of expected something like a UserValidator interface that is then used by the UserCheck implementation, which would allow for different UserValidator implementations to be set at runtime but the code is much simpler than that, which is obviously sufficient for the majority of BaseX uses. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 0

Multithreading
by Philip Oliver 26 Feb '22

26 Feb '22

There are some very old messages regarding the subject of multithreading in the BaseX server but nothing recent that I can find. With the proliferation of CPUs with large core counts, it seems logical to revisit the topic so that a BaseX server can make efficient use of that resource. Has this been considered lately? BaseX is a fantastic system but I'm afraid that it receives less attention than it should because competitors, particularly Elasticsearch/OpenSearch, are architected to take advantage of multithreading and clustering. In my case, the vast majority of the time, the engine is used on a static database, for full text searching. Even a constrained case of permitted multithreading on a (temporarily) read-only database would be very useful.

1 0

Develop a module, HOWTO
by Ben Engbers 25 Feb '22

25 Feb '22

Hi, I know that it is possible to create a module with functions (I have even done that once), but I can't find the documentation anymore on how to do that. Could please someone provide the URL to this information? Thanks, Ben Engbers

3 3

.
by Christian Grün 24 Feb '22

24 Feb '22

It’s a dire day. We wish safety and the rapid establishment of peace to our fellows and friends in Ukraine. BaseX 9.7 was scheduled for today, it will be postponed.

3 4

string-join with a newline separator?
by Ben Engbers 24 Feb '22

24 Feb '22

Hi, My xml has the structure <base> <element1> <al-group> <al>bla</al> </al-group> </element1> <element1> <al-group> <al>bla</al> </al-group> <al-group> <al>bla</al> </al-group> </element1> </base> The <al-group> element contains 1 to many <al> elements. let $tekst := fn:string-join(fn:data($par//al/text()), ".") concatenates this to: bla.bla.bla But I want it to return: bla bla bla Is it possible to add a newline item-separator to fn:string-join? Ben Engbers

4 4

How to call XQuery function from Java?
by Paul L. Merchant Jr. 23 Feb '22

23 Feb '22

Is it possible for a Java function called from XQuery in BaseX to call back to an XQuery function that's passed to it in BaseX? I haven't been able to find any documentation or examples of this. I got the following example to work, but it doesn't feel right. Is there a better way that might take advantage of the automatic type conversions BaseX performs and doesn't require calling something named "...Internal"? Thanks! ------------------- xquery ------------------- import module namspace java = 'java:my.Module; declare function local:hello($p as xs:string) as xs:string { concat("Hello ", $p) }; java:invoke(local:hello#1) ------------------- Java ------------------- package my; import java.io.UnsupportedEncodingException; import org.basex.query.QueryException; import org.basex.query.QueryModule; import org.basex.query.value.type.AtomType; import org.basex.query.value.item.FuncItem; import org.basex.query.value.item.Str; import org.basex.query.value.Value; import org.basex.util.InputInfo; public class Module extends QueryModule { public String invoke(final FuncItem func) throws UnsupportedEncodingException, QueryException { InputInfo ii = new InputInfo("my.Module.invoke", 0, 0); String s = "Java"; Str p1 = new Str(s.getBytes("UTF-8"), AtomType.STRING); Value[] args = new Value[] {p1}; String r = (String) func.invokeInternal(queryContext, ii, args).toJava(); return r; } }

2 1

Content is not allowed in prolog
by Ben Engbers 23 Feb '22

23 Feb '22

Hi, I have a directory with 12 testfiles. In the BaseX-GUI, the command: CREATE DB Parl_Test /home/bengbers/R/x86_64-redhat-linux-gnu-library/4.1/RBaseX/extdata/xml_files/ Creates database "Parl_Test" and loads the xml-files. In my R-client, Session$Create("Parl_Test") creates database "Parl_test"=> OK I want to create the same database with my client. I initialize the variable "XML_Files" with "/home/bengbers/R/x86_64-redhat-linux-gnu-library/4.1/RBaseX/extdata/xml_files". The client translates the command: Session$Create("Parl_Test", XML_Files) into a raw vector: '\bParl_Test\0/home/bengbers/R/x86_64-redhat-linux-gnu-library/4.1/RBaseX/extdata/xml_files' which is sent to the server. But the server responds with: "\"Parl_Test.xml\" (Regel 1): Content is not allowed in prolog." I didn't touch the xml-files. Where is the content inserted? Ben Engbers

3 13

feature request: opening database at arbitrary file path
by Eric Levy 22 Feb '22

22 Feb '22

I recently learned that BaseX offers no true support for access to a database given directly by its path. In contrast, opening a database by file path is a well-established pattern in embedded databases, seen for example in SQLite, which permits use of any database given by an arbitrary path, without any dependence on external configuration or other data elsewhere on the system. I wish to submit as a feature request the suggestion that BaseX include similar support.

6 23

Faster in the cloud?
by Jonathan Robie 22 Feb '22

22 Feb '22

If I am running my queries and updates on a typical laptop, would they run much faster if I ran them on a suitably configured instance in the cloud? I know this is a very general question, but I'm wondering what experiences y'all have had with this. Jonathan

6 8

How to produce multipart/mixed results with RESTXQ?
by Imsieke, Gerrit, le-tex 22 Feb '22

22 Feb '22

I was faintly hoping that a sequence will be magically output as multipart/mixed with appropriate part content types when I annotate it with %rest:produces("multipart/mixed"): declare %rest:GET %rest:path("/test") %rest:produces("multipart/mixed") function my:test-multipart() { (<ok/>, json:serialize(map{'some':'json'})) }; But the response header says "Content-Type: application/xml", and the result is <ok/> { "some": "json" } Are there some other functions or annotations or rest:response elements that I can use in order to get multipart/mixed correctly assembled? Bonus: With the possibility to add file names to each chunk? Gerrit

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk