BaseX-Talk April 2021

basex-talk@mailman.uni-konstanz.de

21 participants
18 discussions

Java type mismatch when reflecting.
by Andreas Mixich 15 Apr '21

15 Apr '21

Hi I can successfully create an object instance of https://www.stringtemplate.org/api/index.html when using the parameter-less default init: declare namespace ST = 'java:org.stringtemplate.v4.ST'; let $template := "Hello <who>" let $stInstance := ST:new($template) let $call := ST:add($stInstance, "who", "World!") return ST:render($stInstance) Template delimiters can be changed by supplying left and right delimiters (type char) upon init (by default ST4 uses < and > as template delimiters). The signature for this would be: ST(java.lang.String template, char delimiterStartChar, char delimiterStopChar) Documentation is at https://www.stringtemplate.org/api/org/stringtemplate/v4/ST.html#ST-java.la… So I defined the types at invocation, but executing following code I get an error. declare namespace ST = 'java:org.stringtemplate.v4.ST'; let $template := "Hello $who$" let $stInstance := ST:new·java.lang.String·char·char($template,"$","$") let $call := ST:add($stInstance, "who", "World!") return ST:render($stInstance) The error is: [XPTY0004] java.lang.IllegalArgumentException: argument type mismatch. Caused by: org.stringtemplate.v4.ST(String, String, String). The query plan of this unsuccessfull invocation is: <QueryPlan compiled="true" updating="false"> <GFLWOR type="item()*"> <Let type="item()*" name="$stInstance" id="1"> <DynJavaConstr name="org.stringtemplate.v4.ST" type="item()*"> <Str type="xs:string" size="1">Hello $who$</Str> <Str type="xs:string" size="1">$</Str> <Str type="xs:string" size="1">$</Str> </DynJavaConstr> </Let> <List type="item()*"> <ProfVoid name="prof:void" type="empty-sequence()" size="0"> <DynJavaFunc name="org.stringtemplate.v4.ST:add" type="item()*"> <VarRef type="item()*" name="$stInstance" id="1"/> <Str type="xs:string" size="1">who</Str> <Str type="xs:string" size="1">World!</Str> </DynJavaFunc> </ProfVoid> <DynJavaFunc name="org.stringtemplate.v4.ST:render" type="item()*"> <VarRef type="item()*" name="$stInstance" id="1"/> </DynJavaFunc> </List> </GFLWOR> </QueryPlan> I can see no deviation from https://docs.basex.org/wiki/Java_Bindings#Functions_and_Variables in my code. What am I doing wrong? Thanks. -- Goody Bye, Minden jót, Mit freundlichen Grüßen, Andreas Mixich

2 1

Re: [basex-talk] Datamining MS Windows eventlogs converted to xml
by Adam Law 15 Apr '21

15 Apr '21

Another approach may be to use WinLogBeat and get BaseX to query ElasticSearch. Did this, worked ok On Wed, Apr 14, 2021 at 6:00 PM <basex-talk-request(a)mailman.uni-konstanz.de> wrote: > Send BaseX-Talk mailing list submissions to > basex-talk(a)mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request(a)mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner(a)mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Re: Datamining MS Windows eventlogs converted to xml > (Christian Gr?n) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 13 Apr 2021 16:51:06 +0200 > From: Christian Gr?n <christian.gruen(a)gmail.com> > To: commandline-be <commandline(a)protonmail.com> > Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de> > Subject: Re: [basex-talk] Datamining MS Windows eventlogs converted to > xml > Message-ID: > <CAP94bnPEUXc9biKsC5TNmN6-5bERatSW-uMF6n6cw4G70y= > MQQ(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hi Joris, > > > The kind of queries I'll need to write may be quite complex. At least to > me at this point, I noticed BaseX to have more under the hood i have yet to > explore. > > As XQuery has a much broader scope than SQL, you can write very > sophisticated queries. In our documentation, you can find some simple > use cases; see e.g. [1,2,3]. > > > One other thing I'd like to explore is if I can query and/or index .evt > and. evtx files directly by repurposing source code. > > As far as I know, these formats are proprietary, but you could include > external converters for that [4]. Such a converter could also be > invoked from XQuery [5], and the result could be further processed. > > Hope this helps, > Christian > > [1] https://docs.basex.org/wiki/XQuery_3.0 > [2] https://docs.basex.org/wiki/XQuery_3.1 > [3] https://docs.basex.org/wiki/XQuery > [4] > https://www.vanimpe.eu/2015/07/16/use-evtxparser-convert-windows-event-log-… > [5] https://docs.basex.org/wiki/Process_Module > > > > > Typically I expect to query for unique eventid to create a baseline and > for one or more reoccurring field values which may span any or all eventid. > Here i will need to do pattern matching for example for uuid, ip, fqdn etc. > > > > My preliminary super simple tests have shown this to be feasible. > > > > Based on these results and queries I'd the seek to export to json or > other formats. Maybe also to send data to elasticsearch or neo4j. > > > > One other thing I'd like to explore is if I can query and/or index .evt > and. evtx files directly by repurposing source code. > > > > > > Best Regards, > > > > Joris > > > > > > > > > > > > > > -------- Oorspronkelijk bericht -------- > > Aan 12 apr. 2021 12:51, Christian Gr?n < christian.gruen(a)gmail.com> > schreef: > > > > > > Hi Joris, > > > > Have you already exported the MS windows events to XML, and are you > > now trying to extract specific information from that files? > > > > Best, > > Christian > > > > On Wed, Apr 7, 2021 at 2:13 PM Joris Lambrecht > > <commandline(a)protonmail.com> wrote: > > > > > > Dear, > > > > > > For the longest time a good tool to datamine ms windows eventlogs > > > escaped me. > > > > > > BaseX appears to offer the toolkit which could permit to do so after an > > > affordable conversion to XML. > > > > > > Now i seek to build a set of queries to extract information from > > > multiple converted eventlog files at once. > > > > > > Are there people on this list who have experience or are open to > > > building experience on this topic ? > > > > > > Br, > > > > > > Joris > > > > > > End of BaseX-Talk Digest, Vol 136, Issue 7 > ****************************************** >

2 1

Call for install/setup stories from users
by Bridger Dyson-Smith 14 Apr '21

14 Apr '21

Hi all - Per the recent thread about installing, I was hoping to convince some of you to share your experiences installing and running BaseX. Whether you use Mac OS, Windows, a Linux, or something else: how are you installing and running BaseX? I'm asking as there could be some possible improvements to the wiki that might help new users (or even less-new users!) get up and running. Thanks in advance for your time and trouble! If I can hear about number of approaches, I'm happy to attempt writing up some details for the wiki. Best, Bridger PS Here's mine: Operating systems: I'm typically using BaseX on either a FreeBSD or a Void Linux system (rarely on other Linuxes, and only once or twice on Windows). Install process: my OS' don't have a pre-packaged download for BaseX, so I do something like: 1. cd ~/bin 2. mkdir basex-src basex-data 3. fetch|wget https://files.basex.org/releases/BaseX.zip 4. unzip BaseX.zip 5. rm -rf basex/src basex/data 6. ln -s ~/bin/basex-src ~/bin/basex/src; ln -s ~/bin/basex-data ~/bin/basex/data ... 7. enjoy basex! Upgrade process: something like 1. cd ~/bin 2. fetch|wget https://files.basex.org/releases/BaseX.zip 3. mv basex basex-old 4. unzip BaseX.zip 5. rm -rf basex/src basex/data 6. ln -s ~/bin/basex-src ~/bin/basex/src; ln -s ~/bin/basex-data ~/bin/basex/data ... Usage note: I have a line in my shell RC file (~/.zshrc) that helps my shell/command line know where to find the various basex executables: `export PATH="$PATH:$HOME/bin/basex/bin"`

6 5

Datamining MS Windows eventlogs converted to xml
by Joris Lambrecht 13 Apr '21

13 Apr '21

Dear, For the longest time a good tool to datamine ms windows eventlogs escaped me. BaseX appears to offer the toolkit which could permit to do so after an affordable conversion to XML. Now i seek to build a set of queries to extract information from multiple converted eventlog files at once. Are there people on this list who have experience or are open to building experience on this topic ? Br, Joris

3 3

anti bug
by Hans-Juergen Rennau 13 Apr '21

13 Apr '21

Hello, I would like to let you know that just as there is matter and anti matter, there are bugs and anti bugs. An anti bug is when something works which cannot possibly work. I am not surprised that the first anti bug detected (by me) is a BaseX one. The following invocation basex "doc('result.xml')/*/page/id => distinct-values() => sort()" processes a file the size of which is more than 8 GB. It cannot be processed on my machine, unless processing is streaming. A streaming fn:doc? Amazing. (At least me.) Hans-Jürgen

2 4

Re: [basex-talk] Multi-platform lightweight C/C++ BaseX client API
by Christian Grün 12 Apr '21

12 Apr '21

A new C++ BaseX Client is available, based on the SDL2_net library [1]. Thanks to Karim Salama for the contribution! [1] https://github.com/bsella/BaseX_CppClient

1 0

MAXLEN
by Beck, Julia 07 Apr '21

07 Apr '21

Dear all, The option MAXLEN tricked me today which is why I need some advice. I have the problem that I have a query that normally works smoothly but with one certain data set, it's really slow. I think it's because at some point in the query I am trying to match a value from the index that is cut due to MAXLEN of 96 (I made a test: the query is fast for values below 96, but 97 and above is slow). So I followed the documentation and created the database again with a MAXLEN of 200 instead. What tricked me was that whenever I optimize the database, MAXLEN is back to 96. The documentation says about MAXLEN: "Specifies the maximum length for strings to be stored in index structures<https://docs.basex.org/wiki/Indexes>. The value of this option will be assigned once to a new database, and can *only be changed* by creating a new database or doing a full optimization<https://docs.basex.org/wiki/Commands#OPTIMIZE>." I assumed "full optimization" means db:optimize with $all is true(). But I noticed that the value of MAXLEN also changes when I do db:optimize without true(). For Optimization, the documentation states: * If ALL is specified, all database structures are completely reconstructed. The database size will be reduced, and all orphaned data will be deleted. * Without ALL, only the outdated index structures and database statistics will be updated. If the database is completely up-to-date, nothing will be done. * Database options will be adopted from the original database. Only AUTOOPTIMIZE<https://docs.basex.org/wiki/Options#AUTOOPTIMIZE> and (if ALL is specified) UPDINDEX<https://docs.basex.org/wiki/Options#UPDINDEX> will be adopted from the current options. If I understand correctly from the last bullet point, it says that the MAXLEN option is not considered from the original database as it's an index option. For the second bullet point I would not say though, that "nothing will be done", if the database is completely up-to-date. Because when I create a new database "test" with a MAXLEN of 200 and do a db:optimize("test"), MAXLEN goes back to 96. How to keep MAXLEN at the value I set? Do I have to set it every time, I do an optimization? Like db:optimize("test", map {"maxlen": 200})? That seems to work but it feels a bit over the top, considering that I have many queries with optimizations that I would need to adapt for this one data set then. What is the reason for resetting MAXLEN during optimization? Best wishes, Julia

3 3

BaseX 9.4.2: Tweaks & Fixes
by Christian Grün 02 Apr '21

02 Apr '21

Dear all, We have released another 9.4 late-summer maintenance release with new performance tweaks and minor bug fixes: http://basex.org In addition, we have started a new Wiki article on XQuery optimizations, which we plan to extend step by step: https://docs.basex.org/wiki/XQuery_Optimizations We are looking forward to your feedback; have fun, Your BaseX Team

2 7

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk April 2021