- BaseX-Talk - mailman.uni-konstanz.de

CSV and automatic encoding detection
by Kristian Kankainen 24 May '21

24 May '21

Hi folks, I am aware that with the HTML module you can let it guess a file's encoding by itself by providing it in binary format: > If the input encoding is unknown, the data to be processed can be passed on in its binary representation. The HTML parser will automatically try to detect the correct encoding: > > Query > html:parse(fetch:binary("https://en.wikipedia.org")) But is there a way to guess encoding of CSV files? So far I have tried with the fetch and CSV module with no results. I have a huge bunch of CSV files and they are all in different encodings. Maybe it is possible to pipe the content of the fetch:binary to a system command for guessing the encoding, and use this to read in the csv? Best regards, Kristian Kankainen

3 3

Re: [basex-talk] automatic encoding
by Philippe Pons 11 May '21

11 May '21

Dear Christian, Thank you very much for this solution! It works perfectly. I use it on fairly short texts, so it is very fast. However, I have a subsidiary question: I sometimes have dates (for example 1600) or groups of words. In these two cases only, the automatic encoding does not work? Do you have any tips? Thank you so much. Philippe Le 06/05/2021 à 16:37, Christian Grün a écrit : > Hi Philippe, > > here’s one way to solve your challenge: > > let $csv := csv:doc('exemple.csv') > let $doc := doc('exemple1.xml') > return $doc update { > for $text in .//text() > return replace node $text with ( > for $token in analyze-string($text, '\p{L}+')/* > let $record := $csv//record[entry[1] = $token] > return if ($record) then ( > element { data($record/entry[2]) } { data($token) } > ) else ( > text { $token } > ) > ) > } > > In the update block, all text nodes are processed and replaced by a > sequence of new element and text nodes. The text is tokenized with > fn:analyze-string, such that the non-letter characters will not get > lost. If a CSV entry exists for a single token, it is replaced by a > new element that contains the token. Otherwise, the original token > will be adopted. > > If large texts need to be processed, it may be recommendable to > organize all CSV entries in a map: > > let $words := map:merge( > csv:doc('exemple.csv')/csv/record > ! map:entry(data(entry[1]), data(entry[2])) > ) > return doc('exemple1.xml') update { > .//text() ! (replace node . with ( > for $token in analyze-string(., '\p{L}+')/* > let $name := $words($token) > return if ($name) then ( > element { $name } { data($token) } > ) else ( > text { $token } > ) > )) > } > > Alternative solutions are appreciated. > > Hope this helps, salutations, > Christian > > > > On Thu, May 6, 2021 at 3:40 PM Philippe Pons > <philippe.pons(a)college-de-france.fr> wrote: >> Hi, >> >> This may be more of an xquery question than a BaseX question, but I'm trying to ask it here anyway. I can of course remove it if necessary. >> >> However, I have an XML document with a sequence of paragraphs and a CSV spreadsheet (examples 1 and 2 below). The csv gives a list of terms in the first column (present in the XML), and the name of the index in which we want to integrate this term in the second. >> >> I use BaseX and XQuery a lot, and I'm looking for a way to automatically encode the terms in the TEI file without success so far. >> >> I've tried to rely on the ft:mark function to achieve this, but it doesn't completely fit my need. >> >> Maybe you could give me some advice to get the result I'm looking for (example 3)? >> >> With kind regards, >> Philippe >> >> Exemple 1, XML : >> >> <div> >> <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis nisl ultrices, viverra erat id, volutpat magna. Mauris cursus tellus non nisi commodo faucibus. Pellentesque condimentum feugiat sem quis interdum. Vestibulum tempus lectus a augue viverra molestie. Ut condimentum vehicula nisi, vel tincidunt mauris accumsan malesuada. Aliquam quis facilisis justo. Proin convallis eget enim vel eleifend. Nullam faucibus ultricies diam, iaculis feugiat odio condimentum non. Cras vitae dignissim lectus, in pellentesque est. Vivamus pharetra semper magna, sed sodales dui porttitor in. Pellentesque eget sodales quam, et dignissim velit. Aliquam vulputate pulvinar cursus. Phasellus commodo nibh a diam imperdiet cursus. Maecenas dui orci, aliquet quis porttitor non, auctor fringilla leo.</p> >> <p>Vestibulum venenatis velit in imperdiet iaculis. Vivamus consectetur mollis augue, ac efficitur nisl ultrices eu. Cras commodo eleifend mi a luctus. Vivamus sed odio maximus, laoreet mauris quis, tincidunt lorem. Duis non elementum tortor. Nam hendrerit dolor ac interdum condimentum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Aenean vitae massa commodo, condimentum quam in, malesuada massa. Maecenas finibus convallis erat at aliquet. Aliquam erat volutpat. Praesent ligula nisi, tempus id arcu id, scelerisque condimentum dolor. Sed quis tincidunt sem. Nulla ac ex >> hendrerit, ullamcorper leo et, sodales nulla.</p> >> </div> >> >> Exemple 2, CSV : >> >> Vestibulum, placeName >> Cras, persName >> condimentum, persName >> >> >> Exemple 3, Results : >> >> <div> >> <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis nisl ultrices, viverra erat id, volutpat magna. Mauris cursus tellus non nisi commodo faucibus. Pellentesque <persName>condimentum</persName> feugiat sem quis interdum. <placeName>Vestibulum</placeName> tempus lectus a augue viverra molestie. Ut <persName>condimentum</persName> vehicula nisi, vel tincidunt mauris accumsan malesuada. Aliquam quis facilisis justo. Proin convallis eget enim vel eleifend. Nullam faucibus ultricies diam, iaculis feugiat odio <persName>condimentum</persName> non. <persName>Cras</persName> vitae dignissim lectus, in pellentesque est. Vivamus pharetra semper magna, sed sodales dui porttitor in. Pellentesque eget sodales quam, et dignissim velit. Aliquam vulputate pulvinar cursus. Phasellus commodo nibh a diam imperdiet cursus. Maecenas dui orci, aliquet quis porttitor non, auctor fringilla leo.</p> >> <p><placeName>Vestibulum</placeName> venenatis velit in imperdiet iaculis. Vivamus consectetur mollis augue, ac efficitur nisl ultrices eu. <persName>Cras</persName> commodo eleifend mi a luctus. Vivamus sed odio maximus, laoreet mauris quis, tincidunt lorem. Duis non elementum tortor. Nam hendrerit dolor ac interdum <persName>condimentum</persName>. <placeName>Vestibulum</placeName> ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Aenean vitae massa commodo, <persName>condimentum</persName> quam in, malesuada massa. Maecenas finibus convallis erat at aliquet. Aliquam erat volutpat. Praesent ligula nisi, tempus id arcu id, scelerisque <persName>condimentum</persName> dolor. Sed quis tincidunt sem. Nulla ac ex hendrerit, ullamcorper leo et, sodales nulla.</p> >> </div>

2 1

Getting profile information in server responses.
by Reece Dunn 10 May '21

10 May '21

Hi all, In BaseX, is there a way to get the profile timings (compile, run, print, etc.) in the response of a HTTP request via the Server-Timing header ( https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing)? Kind regards, Reece

2 2

automatic encoding
by Philippe Pons 06 May '21

06 May '21

Hi, This may be more of an xquery question than a BaseX question, but I'm trying to ask it here anyway. I can of course remove it if necessary. However, I have an XML document with a sequence of paragraphs and a CSV spreadsheet (examples 1 and 2 below). The csv gives a list of terms in the first column (present in the XML), and the name of the index in which we want to integrate this term in the second. I use BaseX and XQuery a lot, and I'm looking for a way to automatically encode the terms in the TEI file without success so far. I've tried to rely on the ft:mark function to achieve this, but it doesn't completely fit my need. Maybe you could give me some advice to get the result I'm looking for (example 3)? With kind regards, Philippe Exemple 1, XML : <div> <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis nisl ultrices, viverra erat id, volutpat magna. Mauris cursus tellus non nisi commodo faucibus. Pellentesque condimentum feugiat sem quis interdum. Vestibulum tempus lectus a augue viverra molestie. Ut condimentum vehicula nisi, vel tincidunt mauris accumsan malesuada. Aliquam quis facilisis justo. Proin convallis eget enim vel eleifend. Nullam faucibus ultricies diam, iaculis feugiat odio condimentum non. Cras vitae dignissim lectus, in pellentesque est. Vivamus pharetra semper magna, sed sodales dui porttitor in. Pellentesque eget sodales quam, et dignissim velit. Aliquam vulputate pulvinar cursus. Phasellus commodo nibh a diam imperdiet cursus. Maecenas dui orci, aliquet quis porttitor non, auctor fringilla leo.</p> <p>Vestibulum venenatis velit in imperdiet iaculis. Vivamus consectetur mollis augue, acefficitur nisl ultrices eu. Cras commodo eleifend mi a luctus. Vivamus sed odio maximus, laoreet mauris quis, tincidunt lorem. Duis non elementum tortor. Nam hendrerit dolor ac interdum condimentum. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Aenean vitae massa commodo, condimentum quam in, malesuada massa. Maecenas finibus convallis erat at aliquet. Aliquam erat volutpat. Praesent ligula nisi, tempus id arcu id, scelerisque condimentum dolor. Sed quis tincidunt sem. Nulla ac ex hendrerit, ullamcorper leo et, sodales nulla.</p> </div> Exemple 2, CSV : Vestibulum, placeName Cras, persName condimentum, persName Exemple 3, Results : <div> <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis nisl ultrices, viverra erat id, volutpat magna. Mauris cursus tellus non nisi commodo faucibus. Pellentesque <persName>condimentum</persName>feugiat sem quis interdum. <placeName>Vestibulum</placeName>tempus lectus a augue viverra molestie. Ut <persName>condimentum</persName>vehicula nisi, vel tincidunt mauris accumsan malesuada. Aliquam quis facilisis justo. Proin convallis eget enim vel eleifend. Nullam faucibus ultricies diam, iaculis feugiat odio <persName>condimentum</persName>non. <persName>Cras</persName>vitae dignissim lectus, in pellentesque est. Vivamus pharetra semper magna, sed sodales dui porttitor in. Pellentesque eget sodales quam, et dignissim velit. Aliquam vulputate pulvinar cursus. Phasellus commodo nibh a diam imperdiet cursus. Maecenas dui orci, aliquet quis porttitor non, auctor fringilla leo.</p> <p><placeName>Vestibulum</placeName>venenatis velit in imperdiet iaculis. Vivamus consectetur mollis augue, ac efficitur nisl ultrices eu.<persName>Cras</persName>commodo eleifend mi a luctus. Vivamus sed odio maximus, laoreet mauris quis, tincidunt lorem. Duis non elementum tortor. Nam hendrerit dolor ac interdum <persName>condimentum</persName>. <placeName>Vestibulum</placeName>ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Aenean vitae massa commodo, <persName>condimentum</persName>quam in, malesuada massa. Maecenas finibus convallis erat at aliquet. Aliquam erat volutpat. Praesent ligula nisi, tempus id arcu id, scelerisque <persName>condimentum</persName>dolor. Sed quis tincidunt sem. Nulla ac ex hendrerit, ullamcorper leo et, sodales nulla.</p> </div>

2 1

Clarification on registering a job service
by Zimmel, Daniel 06 May '21

06 May '21

Hi, can someone clarify on the following behaviour? The docs say: "Job services can be updated: If a new job is registered, and if there is already a job with the same id, the old entry will be replaced." https://docs.basex.org/wiki/Jobs_Module#Services So when I try to update an existing job service with the following query, I get an unexpected error "Job id already exists: myquery.": jobs:eval( xs:anyURI('myquery.xq'), (), map { 'id':'myquery', 'start':'19:00:00', 'interval':'P1D', 'service': true() } ) Am I reading the documentation wrong? Thanks Daniel

2 3

Re: [basex-talk] feature request: EXI support
by Christian Grün 06 May '21

06 May '21

Hi Mikhail (cc to the list), If your queries generate hundreds of Megabytes, you could check if the full result is actually required. If yes, you may need to check which step is responsible for letting you wait: Is it the pure transmission over the network, is it the evaluation time or is it the client that processes the query result? If it’s the data traffic, and if you use Jetty as BaseX web server, you could enable GZIP compression [1]. If you use another web server, or if your architecture is more complex, you could install a proxy in between that takes care of compressing the transferred data. Best, Christian [1] https://docs.basex.org/wiki/Options#GZIP On Wed, May 5, 2021 at 7:17 AM Mikhail Kuznetsov <mikekuznetsov(a)mail.ru> wrote: > > Hi, Christian. > > I am trying to build a central server for processing multiple QXuery tasks. In some cases the output will be used by a BI system for visualization. For example: > > the BaseX server will accumulate a lot of "raw" xml files collected from multiple entities, say their tax returns > this is great that I can have the flexibility to write new queries and adapt existing ones without modifying the underlying data (which would be the case if I had chosen an SQL database for storage) > a Power BI server gets connected to BaseX through RESTful API and retrieves two kinds of data: individual by entity and benchmark based on the average. > > Some queries may result in hundreds of Mb of xml data. My BaseX server is hosted in AWS, while the Power BI server is located internally. The communication takes too much time, which is a show stopper for me. I am looking for ways to optimize the data transfer. The BaseX server itself is perfect, many thanks for the great job. > > Regards, > Mikhail > > > Среда, 5 мая 2021, 2:45 +03:00 от Christian Grün <christian.gruen(a)gmail.com>: > > Hi Mikhail, > > > Hi there. I have recently been looking at ways to improve the efficiency of my XQuery server and have come across Efficient XML Interchange (EXI) Format. Is there a way that BaseX could support encoding/ decoding for this format? > > The internal storage of XML documents in BaseX differs quite a lot > from the original and verbose textual representation (for which EXI > provides improvements). > > What kind of performance penalties did you encounter so far? > > Best, > Christian > > > On Tue, May 4, 2021 at 10:34 PM Mikhail Kuznetsov <mikekuznetsov(a)mail.ru> wrote: > > > > > > Thanks. > > Mikhail > > > > -- > Mikhail Kuznetsov >

2 2

feature request: EXI support
by Mikhail Kuznetsov 04 May '21

04 May '21

Hi there. I have recently been looking at ways to improve the efficiency of my XQuery server and have come across Efficient XML Interchange (EXI) Format. Is there a way that BaseX could support encoding/ decoding for this format? Thanks. Mikhail

2 1

Re: [basex-talk] Make use of external Identity Management for BaseX
by Adam Law 04 May '21

04 May '21

If Rob isn't interested, I certain am... Yes please share >>We have also a RestXQ module that in combination with basex:perm annotations protects access to a GUI implementing the Oauth2 Code-grant flow. If Rob is interested we could share. Many thanks for sharing https://code-repo.d4science.org/gCubeSystem/d4science-keycloak-themes/src/b… This is a bit forward of me, but would you consider changing the license away from European Union Public Licence which appears to be 'copyleft' to say BSD 3-Clause, MIT, ... Adam On Fri, Apr 30, 2021 at 6:00 PM <basex-talk-request(a)mailman.uni-konstanz.de> wrote: > Send BaseX-Talk mailing list submissions to > basex-talk(a)mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request(a)mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner(a)mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Make use of external Identity Management for BaseX (Rob) > 2. Re: Reloading jars on a running http server. (Reece Dunn) > 3. Getting profile information in server responses. (Reece Dunn) > 4. Re: Make use of external Identity Management for BaseX > (Christian Gr?n) > 5. Re: Reloading jars on a running http server. (Christian Gr?n) > 6. Re: Getting profile information in server responses. > (Christian Gr?n) > 7. Re: Reloading jars on a running http server. (Reece Dunn) > 8. Re: Reloading jars on a running http server. (Christian Gr?n) > 9. Re: Make use of external Identity Management for BaseX > (Marco Lettere) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 29 Apr 2021 15:54:08 +0200 > From: Rob <private(a)tjalma.com> > To: BaseX <basex-talk(a)mailman.uni-konstanz.de> > Subject: [basex-talk] Make use of external Identity Management for > BaseX > Message-ID: <39036EF3-E67D-470D-B2F4-E6FBFFDF40E6(a)tjalma.com> > Content-Type: text/plain; charset=utf-8 > > Hi, > > I have a question. > Can BaseX make use of an Identity Management application like Keycloak for > Users? > > I can?t find anything about it :) > > With kind regards, > Rob Tjalma > > ------------------------------ > > Message: 2 > Date: Thu, 29 Apr 2021 20:48:09 +0100 > From: Reece Dunn <msclrhd(a)googlemail.com> > To: Christian Gr?n <christian.gruen(a)gmail.com> > Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> > Subject: Re: [basex-talk] Reloading jars on a running http server. > Message-ID: > <CAGdtn24Qdgbe3= > nebYA1je4hCrZVAgi_sercCRxsTxmZcWcAxQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Christian, > > Thanks for the response. Unfortunately, I've not been able to get the > reloading working. > > Kind regards, > Reece > > On Wed, 21 Apr 2021 at 18:49, Christian Gr?n <christian.gruen(a)gmail.com> > wrote: > > > Hi Reece, > > > > If you install your Java code as JAR file in the repository [1], the > > code will be loaded and unloaded every time when your query is > > executed. If you get an error message? > > > > java.lang.reflect.InaccessibleObjectException: Unable to make field > > private final jdk.internal.loader.URLClassPath > > java.net.URLClassLoader.ucp accessible: module java.base does not > > "opens java.net" to unnamed module @79e2c065 > > > > ?unloading fails [2], as you?re probably using a more recent version > > of the JDK, which restricts reflective access to internal variables. > > You can get around this by adding Java flags at startup time: > > > > --add-opens java.base/java.net=ALL-UNNAMED > > --add-opens java.base/jdk.internal.loader=ALL-UNNAMED > > > > Maybe there are better solutions to unload JAR files today. > > Suggestions are welcome! > > > > Hope this helps, > > Christian > > > > [1] https://docs.basex.org/wiki/Repository#Java > > [2] > > > https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b… > > > > > > > > On Tue, Apr 20, 2021 at 6:44 PM Reece Dunn <msclrhd(a)googlemail.com> > wrote: > > > > > > Hi all, > > > > > > I'm working on a Java class that I'm importing into an XQuery, so I can > > do additional processing on the data that isn't easily expressible in > > XQuery (or XSLT). In order to get BaseX to pick up a modified version of > > the jar file I'm building, I'm restarting the BaseX http server. > > > > > > This makes it slower to turn around testing the changes. Is there a > > better way of doing this? > > > > > > Kind regards, > > > Reece > > >

3 8

Re: [basex-talk] BaseX-Talk Digest, Vol 137, Issue 3
by Adam Law 03 May '21

03 May '21

Thanks Marco, I understand. As an aside for others, a GO edge router with BaseX is another option - say https://github.com/thomseddon/traefik-forward-auth. OIDC, ACME and SSL are all built in. BaseX can trust / check the forwarded cookie as part of BaseX permissions checking. https://docs.basex.org/wiki/Permissions Adam On Mon, May 3, 2021 at 6:00 PM <basex-talk-request(a)mailman.uni-konstanz.de> wrote: > Send BaseX-Talk mailing list submissions to > basex-talk(a)mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request(a)mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner(a)mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Re: Make use of external Identity Management for BaseX > (Marco Lettere) > 2. Re: Make use of external Identity Management for BaseX > (Marco Lettere) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 3 May 2021 09:20:45 +0200 > From: Marco Lettere <m.lettere(a)gmail.com> > To: basex-talk(a)mailman.uni-konstanz.de > Subject: Re: [basex-talk] Make use of external Identity Management for > BaseX > Message-ID: <0978b86f-a884-052c-dd2c-0eb18291f7f5(a)gmail.com> > Content-Type: text/plain; charset="utf-8"; Format="flowed" > > Hi Adam, > sorry but licensing terms are out of our choice. > We've developed that code in the framework of EU H2020 funded projects > [1,2] thus we have to adhere to the policies of the Institution we > cooperate with (Italian Council of Research). > > Anyway as I told this code is more a proof of concept for administrative > activities related to batch creation of clients ... Feel free to use and > adapt the code as you prefer. > > Regards, > Marco. > > [1] https://www.blue-cloud.org/ > [2] http://project.sobigdata.eu/ > > On 01/05/21 05:29, Adam Law wrote: > > If Rob isn't interested, I certain am... Yes please share > > > > >>We have also a RestXQ module that in combination with basex:perm > > annotations protects access to a GUI implementing the Oauth2 Code-grant > > flow. If Rob is interested we could share. > > > > Many thanks for sharing > > > https://code-repo.d4science.org/gCubeSystem/d4science-keycloak-themes/src/b… > > < > https://code-repo.d4science.org/gCubeSystem/d4science-keycloak-themes/src/b… > > > > > > This is a bit forward of me, but would you consider changing?the > > license away from European Union Public Licence which appears to be > > 'copyleft' to say BSD 3-Clause, MIT, ... > > > > Adam > > > > On Fri, Apr 30, 2021 at 6:00 PM > > <basex-talk-request(a)mailman.uni-konstanz.de > > <mailto:basex-talk-request@mailman.uni-konstanz.de>> wrote: > > > > Send BaseX-Talk mailing list submissions to > > basex-talk(a)mailman.uni-konstanz.de > > <mailto:basex-talk@mailman.uni-konstanz.de> > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > <https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk> > > or, via email, send a message with subject or body 'help' to > > basex-talk-request(a)mailman.uni-konstanz.de > > <mailto:basex-talk-request@mailman.uni-konstanz.de> > > > > You can reach the person managing the list at > > basex-talk-owner(a)mailman.uni-konstanz.de > > <mailto:basex-talk-owner@mailman.uni-konstanz.de> > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of BaseX-Talk digest..." > > > > > > Today's Topics: > > > > ? ?1. Make use of external Identity Management for BaseX (Rob) > > ? ?2. Re: Reloading jars on a running http server. (Reece Dunn) > > ? ?3. Getting profile information in server responses. (Reece Dunn) > > ? ?4. Re: Make use of external Identity Management for BaseX > > ? ? ? (Christian Gr?n) > > ? ?5. Re: Reloading jars on a running http server. (Christian Gr?n) > > ? ?6. Re: Getting profile information in server responses. > > ? ? ? (Christian Gr?n) > > ? ?7. Re: Reloading jars on a running http server. (Reece Dunn) > > ? ?8. Re: Reloading jars on a running http server. (Christian Gr?n) > > ? ?9. Re: Make use of external Identity Management for BaseX > > ? ? ? (Marco Lettere) > > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Thu, 29 Apr 2021 15:54:08 +0200 > > From: Rob <private(a)tjalma.com <mailto:private@tjalma.com>> > > To: BaseX <basex-talk(a)mailman.uni-konstanz.de > > <mailto:basex-talk@mailman.uni-konstanz.de>> > > Subject: [basex-talk] Make use of external Identity Management for > > ? ? ? ? BaseX > > Message-ID: <39036EF3-E67D-470D-B2F4-E6FBFFDF40E6(a)tjalma.com > > <mailto:39036EF3-E67D-470D-B2F4-E6FBFFDF40E6@tjalma.com>> > > Content-Type: text/plain;? ? ? ?charset=utf-8 > > > > Hi, > > > > I have a question. > > Can BaseX make use of an Identity Management application like > > Keycloak for Users? > > > > I can?t find anything about it :) > > > > With kind regards, > > Rob Tjalma > > > > ------------------------------ > > > > Message: 2 > > Date: Thu, 29 Apr 2021 20:48:09 +0100 > > From: Reece Dunn <msclrhd(a)googlemail.com > > <mailto:msclrhd@googlemail.com>> > > To: Christian Gr?n <christian.gruen(a)gmail.com > > <mailto:christian.gruen@gmail.com>> > > Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de > > <mailto:basex-talk@mailman.uni-konstanz.de>> > > Subject: Re: [basex-talk] Reloading jars on a running http server. > > Message-ID: > > ? ? ? ? > > <CAGdtn24Qdgbe3=nebYA1je4hCrZVAgi_sercCRxsTxmZcWcAxQ(a)mail.gmail.com > > <mailto:nebYA1je4hCrZVAgi_sercCRxsTxmZcWcAxQ@mail.gmail.com>> > > Content-Type: text/plain; charset="utf-8" > > > > Hi Christian, > > > > Thanks for the response. Unfortunately, I've not been able to get the > > reloading working. > > > > Kind regards, > > Reece > > > > On Wed, 21 Apr 2021 at 18:49, Christian Gr?n > > <christian.gruen(a)gmail.com <mailto:christian.gruen@gmail.com>> > > wrote: > > > > > Hi Reece, > > > > > > If you install your Java code as JAR file in the repository [1], > the > > > code will be loaded and unloaded every time when your query is > > > executed. If you get an error message? > > > > > >? ?java.lang.reflect.InaccessibleObjectException: Unable to make > > field > > > private final jdk.internal.loader.URLClassPath > > > java.net.URLClassLoader.ucp accessible: module java.base does not > > > "opens java.net <http://java.net>" to unnamed module @79e2c065 > > > > > > ?unloading fails [2], as you?re probably using a more recent > version > > > of the JDK, which restricts reflective access to internal > variables. > > > You can get around this by adding Java flags at startup time: > > > > > >? --add-opens java.base/java.net <http://java.net>=ALL-UNNAMED > > >? --add-opens java.base/jdk.internal.loader=ALL-UNNAMED > > > > > > Maybe there are better solutions to unload JAR files today. > > > Suggestions are welcome! > > > > > > Hope this helps, > > > Christian > > > > > > [1] https://docs.basex.org/wiki/Repository#Java > > <https://docs.basex.org/wiki/Repository#Java> > > > [2] > > > > > > https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b… > > < > https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b… > > > > > > > > > > > > > > On Tue, Apr 20, 2021 at 6:44 PM Reece Dunn > > <msclrhd(a)googlemail.com <mailto:msclrhd@googlemail.com>> wrote: > > > > > > > > Hi all, > > > > > > > > I'm working on a Java class that I'm importing into an XQuery, > > so I can > > > do additional processing on the data that isn't easily > > expressible in > > > XQuery (or XSLT). In order to get BaseX to pick up a modified > > version of > > > the jar file I'm building, I'm restarting the BaseX http server. > > > > > > > > This makes it slower to turn around testing the changes. Is > > there a > > > better way of doing this? > > > > > > > > Kind regards, > > > > Reece > > > > >

2 1

Make use of external Identity Management for BaseX
by Rob 30 Apr '21

30 Apr '21

Hi, I have a question. Can BaseX make use of an Identity Management application like Keycloak for Users? I can’t find anything about it :) With kind regards, Rob Tjalma

3 2

Reloading jars on a running http server.
by Reece Dunn 30 Apr '21

30 Apr '21

Hi all, I'm working on a Java class that I'm importing into an XQuery, so I can do additional processing on the data that isn't easily expressible in XQuery (or XSLT). In order to get BaseX to pick up a modified version of the jar file I'm building, I'm restarting the BaseX http server. This makes it slower to turn around testing the changes. Is there a better way of doing this? Kind regards, Reece

2 5

7.9 to 9.4 upgrade
by Bondeson, Carl 28 Apr '21

28 Apr '21

We have been using BaseX for about 10 years. We were stuck at 7.9 because the performance of 8 was too restrictive. We tried 9.4 as a last ditch effort to correct some issues and the performance was just as good as 7.9. You made mention that many tcp/ip issues were resolved. We are using BaseX in a multithreaded client server configuration. I have architected a singleton as the server, allowing synchronized clients access to it. Everything is fine until we are under heavy load. We get the following error. IO exception Software caused connection abort: socket write error in mapperProcess query: declare option output:item-separator ""; distinct-values(for $h in //Concept[@NameSpace='Pilotfish.MergeLevel'] for $d in //Concept[@NameSpace=$h/AssociatedConcept/@NameSpace and .=$h/AssociatedConcept/@Concept] where $h/@Value='ByProductCodeDaily' return $d/@Value) retries: 9 We try 8 times, with a 100 msec wait between each attempt, and then generate an exception on the 9th. We have added tcp/ip registry fixes (MaxUserPort, TcpTimedWaitDelay) to the Windows server as a recommended fix for these types of errors. Do you have any other suggestions? Thanks for a great product! Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson(a)ct.gov<mailto:Carl.Bondeson@ct.gov> [cid:image001.jpg@01D73B38.801C1390][cid:image002.png@01D73B38.801C1390]

5 10

BaseX and Oxygen scenario
by Zimmel, Daniel 22 Apr '21

22 Apr '21

Hi, am I right in assuming that in order to use the XQJ interface as described here (https://docs.basex.org/wiki/Integrating_oXygen) , you will always need a running local database instance of BaseX? I am not able to use the BaseX.jar standalone in a transformation scenario as I would for example do in a custom ant scenario where I can use BaseX as a standalone transformer (using it as a commandline process). Thanks, Daniel

2 3

Using other Java runtimes?
by Peter Villadsen 21 Apr '21

21 Apr '21

Guys, I was wondering what the options are for running BaseX on top of a runtime that is different from the Oracle one? I see Oracle is aggressively monetizing the Java runtime, but I know there are alternatives out there. Have any been tested with BaseX? Best Regards Peter Villadsen Principal Architect Microsoft Business Applications Group

7 7

Re: [basex-talk] Using other Java runtimes?
by James Ball 17 Apr '21

17 Apr '21

Hello Peter, I’ve been running a system using Amazon’s Corretto 11 from their Docker image with good results. I did also play with using AWS Lambda to run BaseX but I’ve never put anything into production - I had success there with both the openjdk 1.8 and Corretto options. Regards, James > > From: Peter Villadsen <Peter.Villadsen(a)microsoft.com <mailto:Peter.Villadsen@microsoft.com>> > Subject: [basex-talk] Using other Java runtimes? > Date: 16 April 2021 at 22:39:46 BST > To: BaseX <basex-talk(a)mailman.uni-konstanz.de <mailto:basex-talk@mailman.uni-konstanz.de>> > > > Guys, > > I was wondering what the options are for running BaseX on top of a runtime that is different from the Oracle one? I see Oracle is aggressively monetizing the Java runtime, but I know there are alternatives out there. Have any been tested with BaseX? > > > Best Regards > > Peter Villadsen > Principal Architect > Microsoft Business Applications Group >

1 0

Java type mismatch when reflecting.
by Andreas Mixich 15 Apr '21

15 Apr '21

Hi I can successfully create an object instance of https://www.stringtemplate.org/api/index.html when using the parameter-less default init: declare namespace ST = 'java:org.stringtemplate.v4.ST'; let $template := "Hello <who>" let $stInstance := ST:new($template) let $call := ST:add($stInstance, "who", "World!") return ST:render($stInstance) Template delimiters can be changed by supplying left and right delimiters (type char) upon init (by default ST4 uses < and > as template delimiters). The signature for this would be: ST(java.lang.String template, char delimiterStartChar, char delimiterStopChar) Documentation is at https://www.stringtemplate.org/api/org/stringtemplate/v4/ST.html#ST-java.la… So I defined the types at invocation, but executing following code I get an error. declare namespace ST = 'java:org.stringtemplate.v4.ST'; let $template := "Hello $who$" let $stInstance := ST:new·java.lang.String·char·char($template,"$","$") let $call := ST:add($stInstance, "who", "World!") return ST:render($stInstance) The error is: [XPTY0004] java.lang.IllegalArgumentException: argument type mismatch. Caused by: org.stringtemplate.v4.ST(String, String, String). The query plan of this unsuccessfull invocation is: <QueryPlan compiled="true" updating="false"> <GFLWOR type="item()*"> <Let type="item()*" name="$stInstance" id="1"> <DynJavaConstr name="org.stringtemplate.v4.ST" type="item()*"> <Str type="xs:string" size="1">Hello $who$</Str> <Str type="xs:string" size="1">$</Str> <Str type="xs:string" size="1">$</Str> </DynJavaConstr> </Let> <List type="item()*"> <ProfVoid name="prof:void" type="empty-sequence()" size="0"> <DynJavaFunc name="org.stringtemplate.v4.ST:add" type="item()*"> <VarRef type="item()*" name="$stInstance" id="1"/> <Str type="xs:string" size="1">who</Str> <Str type="xs:string" size="1">World!</Str> </DynJavaFunc> </ProfVoid> <DynJavaFunc name="org.stringtemplate.v4.ST:render" type="item()*"> <VarRef type="item()*" name="$stInstance" id="1"/> </DynJavaFunc> </List> </GFLWOR> </QueryPlan> I can see no deviation from https://docs.basex.org/wiki/Java_Bindings#Functions_and_Variables in my code. What am I doing wrong? Thanks. -- Goody Bye, Minden jót, Mit freundlichen Grüßen, Andreas Mixich

2 1

Re: [basex-talk] Datamining MS Windows eventlogs converted to xml
by Adam Law 15 Apr '21

15 Apr '21

Another approach may be to use WinLogBeat and get BaseX to query ElasticSearch. Did this, worked ok On Wed, Apr 14, 2021 at 6:00 PM <basex-talk-request(a)mailman.uni-konstanz.de> wrote: > Send BaseX-Talk mailing list submissions to > basex-talk(a)mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request(a)mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner(a)mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Re: Datamining MS Windows eventlogs converted to xml > (Christian Gr?n) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 13 Apr 2021 16:51:06 +0200 > From: Christian Gr?n <christian.gruen(a)gmail.com> > To: commandline-be <commandline(a)protonmail.com> > Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> > Subject: Re: [basex-talk] Datamining MS Windows eventlogs converted to > xml > Message-ID: > <CAP94bnPEUXc9biKsC5TNmN6-5bERatSW-uMF6n6cw4G70y= > MQQ(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hi Joris, > > > The kind of queries I'll need to write may be quite complex. At least to > me at this point, I noticed BaseX to have more under the hood i have yet to > explore. > > As XQuery has a much broader scope than SQL, you can write very > sophisticated queries. In our documentation, you can find some simple > use cases; see e.g. [1,2,3]. > > > One other thing I'd like to explore is if I can query and/or index .evt > and. evtx files directly by repurposing source code. > > As far as I know, these formats are proprietary, but you could include > external converters for that [4]. Such a converter could also be > invoked from XQuery [5], and the result could be further processed. > > Hope this helps, > Christian > > [1] https://docs.basex.org/wiki/XQuery_3.0 > [2] https://docs.basex.org/wiki/XQuery_3.1 > [3] https://docs.basex.org/wiki/XQuery > [4] > https://www.vanimpe.eu/2015/07/16/use-evtxparser-convert-windows-event-log-… > [5] https://docs.basex.org/wiki/Process_Module > > > > > Typically I expect to query for unique eventid to create a baseline and > for one or more reoccurring field values which may span any or all eventid. > Here i will need to do pattern matching for example for uuid, ip, fqdn etc. > > > > My preliminary super simple tests have shown this to be feasible. > > > > Based on these results and queries I'd the seek to export to json or > other formats. Maybe also to send data to elasticsearch or neo4j. > > > > One other thing I'd like to explore is if I can query and/or index .evt > and. evtx files directly by repurposing source code. > > > > > > Best Regards, > > > > Joris > > > > > > > > > > > > > > -------- Oorspronkelijk bericht -------- > > Aan 12 apr. 2021 12:51, Christian Gr?n < christian.gruen(a)gmail.com> > schreef: > > > > > > Hi Joris, > > > > Have you already exported the MS windows events to XML, and are you > > now trying to extract specific information from that files? > > > > Best, > > Christian > > > > On Wed, Apr 7, 2021 at 2:13 PM Joris Lambrecht > > <commandline(a)protonmail.com> wrote: > > > > > > Dear, > > > > > > For the longest time a good tool to datamine ms windows eventlogs > > > escaped me. > > > > > > BaseX appears to offer the toolkit which could permit to do so after an > > > affordable conversion to XML. > > > > > > Now i seek to build a set of queries to extract information from > > > multiple converted eventlog files at once. > > > > > > Are there people on this list who have experience or are open to > > > building experience on this topic ? > > > > > > Br, > > > > > > Joris > > > > > > End of BaseX-Talk Digest, Vol 136, Issue 7 > ****************************************** >

2 1

Call for install/setup stories from users
by Bridger Dyson-Smith 14 Apr '21

14 Apr '21

Hi all - Per the recent thread about installing, I was hoping to convince some of you to share your experiences installing and running BaseX. Whether you use Mac OS, Windows, a Linux, or something else: how are you installing and running BaseX? I'm asking as there could be some possible improvements to the wiki that might help new users (or even less-new users!) get up and running. Thanks in advance for your time and trouble! If I can hear about number of approaches, I'm happy to attempt writing up some details for the wiki. Best, Bridger PS Here's mine: Operating systems: I'm typically using BaseX on either a FreeBSD or a Void Linux system (rarely on other Linuxes, and only once or twice on Windows). Install process: my OS' don't have a pre-packaged download for BaseX, so I do something like: 1. cd ~/bin 2. mkdir basex-src basex-data 3. fetch|wget https://files.basex.org/releases/BaseX.zip 4. unzip BaseX.zip 5. rm -rf basex/src basex/data 6. ln -s ~/bin/basex-src ~/bin/basex/src; ln -s ~/bin/basex-data ~/bin/basex/data ... 7. enjoy basex! Upgrade process: something like 1. cd ~/bin 2. fetch|wget https://files.basex.org/releases/BaseX.zip 3. mv basex basex-old 4. unzip BaseX.zip 5. rm -rf basex/src basex/data 6. ln -s ~/bin/basex-src ~/bin/basex/src; ln -s ~/bin/basex-data ~/bin/basex/data ... Usage note: I have a line in my shell RC file (~/.zshrc) that helps my shell/command line know where to find the various basex executables: `export PATH="$PATH:$HOME/bin/basex/bin"`

6 5

Datamining MS Windows eventlogs converted to xml
by Joris Lambrecht 13 Apr '21

13 Apr '21

Dear, For the longest time a good tool to datamine ms windows eventlogs escaped me. BaseX appears to offer the toolkit which could permit to do so after an affordable conversion to XML. Now i seek to build a set of queries to extract information from multiple converted eventlog files at once. Are there people on this list who have experience or are open to building experience on this topic ? Br, Joris

3 3

anti bug
by Hans-Juergen Rennau 13 Apr '21

13 Apr '21

Hello, I would like to let you know that just as there is matter and anti matter, there are bugs and anti bugs. An anti bug is when something works which cannot possibly work. I am not surprised that the first anti bug detected (by me) is a BaseX one. The following invocation basex "doc('result.xml')/*/page/id => distinct-values() => sort()" processes a file the size of which is more than 8 GB. It cannot be processed on my machine, unless processing is streaming. A streaming fn:doc? Amazing. (At least me.) Hans-Jürgen

2 4

Re: [basex-talk] Multi-platform lightweight C/C++ BaseX client API
by Christian Grün 12 Apr '21

12 Apr '21

A new C++ BaseX Client is available, based on the SDL2_net library [1]. Thanks to Karim Salama for the contribution! [1] https://github.com/bsella/BaseX_CppClient

1 0

MAXLEN
by Beck, Julia 07 Apr '21

07 Apr '21

Dear all, The option MAXLEN tricked me today which is why I need some advice. I have the problem that I have a query that normally works smoothly but with one certain data set, it's really slow. I think it's because at some point in the query I am trying to match a value from the index that is cut due to MAXLEN of 96 (I made a test: the query is fast for values below 96, but 97 and above is slow). So I followed the documentation and created the database again with a MAXLEN of 200 instead. What tricked me was that whenever I optimize the database, MAXLEN is back to 96. The documentation says about MAXLEN: "Specifies the maximum length for strings to be stored in index structures<https://docs.basex.org/wiki/Indexes>. The value of this option will be assigned once to a new database, and can *only be changed* by creating a new database or doing a full optimization<https://docs.basex.org/wiki/Commands#OPTIMIZE>." I assumed "full optimization" means db:optimize with $all is true(). But I noticed that the value of MAXLEN also changes when I do db:optimize without true(). For Optimization, the documentation states: * If ALL is specified, all database structures are completely reconstructed. The database size will be reduced, and all orphaned data will be deleted. * Without ALL, only the outdated index structures and database statistics will be updated. If the database is completely up-to-date, nothing will be done. * Database options will be adopted from the original database. Only AUTOOPTIMIZE<https://docs.basex.org/wiki/Options#AUTOOPTIMIZE> and (if ALL is specified) UPDINDEX<https://docs.basex.org/wiki/Options#UPDINDEX> will be adopted from the current options. If I understand correctly from the last bullet point, it says that the MAXLEN option is not considered from the original database as it's an index option. For the second bullet point I would not say though, that "nothing will be done", if the database is completely up-to-date. Because when I create a new database "test" with a MAXLEN of 200 and do a db:optimize("test"), MAXLEN goes back to 96. How to keep MAXLEN at the value I set? Do I have to set it every time, I do an optimization? Like db:optimize("test", map {"maxlen": 200})? That seems to work but it feels a bit over the top, considering that I have many queries with optimizations that I would need to adapt for this one data set then. What is the reason for resetting MAXLEN during optimization? Best wishes, Julia

3 3

BaseX 9.4.2: Tweaks & Fixes
by Christian Grün 02 Apr '21

02 Apr '21

Dear all, We have released another 9.4 late-summer maintenance release with new performance tweaks and minor bug fixes: http://basex.org In addition, we have started a new Wiki article on XQuery optimizations, which we plan to extend step by step: https://docs.basex.org/wiki/XQuery_Optimizations We are looking forward to your feedback; have fun, Your BaseX Team

2 7

A question about the arrow operator and predicates
by Bridger Dyson-Smith 31 Mar '21

31 Mar '21

Hi all - I was wondering if anyone would have an insight for me as to why the following expression is wrong: ``` for $s in ("/a/b/c", "/1/2/3") let $t := $s => tokenize("/")[last()] return $t ``` This is because of the predicate filter, but I'm not clear on *why* it's because of that :) Thanks for your help! Best, Bridger PS I don't always remember to look at the optimized query in the Info window, but when I do I always get a hint about something; "util:last()" in this case. correctly ``` for $s in ("/a/b/c", "/1/2/3") let $t := tokenize($s, "/")[last()] return $t ``` ``` for $s in ("/a/b/c", "/1/2/3") let $t := $s => tokenize("/") => util:last() return $t ```

2 3

Missing namespaces when storing an excerpt?
by Clark, Ash 29 Mar '21

29 Mar '21

Hi! I’m pulling JSON from an API, converting the string to the XML representation, and storing part of the resulting XML into BaseX. The problem is, I can’t get the xmlns namespace declaration to stick to the XML excerpt once it’s saved to BaseX. It seems to work when I display the unstored excerpt. Here’s a short XQuery that illustrates my problem. Am I missing something? Is this a bug? xquery version "3.1"; (: NAMESPACES :) declare default element namespace "http://www.w3.org/2005/xpath-functions"; declare namespace db="http://basex.org/modules/db"; declare namespace output="http://www.w3.org/2010/xslt-xquery-serialization"; declare namespace update="http://basex.org/modules/update"; (: OPTIONS :) declare option db:chop 'false'; declare option db:stripns 'false'; declare option output:indent 'yes'; declare option output:method 'xml'; (: MAIN QUERY :) let $uri := "https://wwp.northeastern.edu/exist/rest/db/exhibits/xquery/people.xquery?fi…" let $full := json-to-xml(unparsed-text($uri)) let $snippet := $full/fn:array/fn:map[1] let $workaround := <map xmlns="http://www.w3.org/2005/xpath-functions">{ $snippet/* }</map> return ( db:create("test-ns", ($full, $snippet, $workaround), ("full.xml", "snippet.xml", "workaround.xml")), update:output(($snippet, $workaround)) ) Thanks for your time! Warmly, Ash Ash Clark (my pronoun.is/e/em/eir<http://pronoun.is/e/em/eir>) XML Applications Developer Digital Scholarship Group Northeastern University Libraries as.clark(a)northeastern.edu (617) 373-5983

2 6

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk