- BaseX-Talk - mailman.uni-konstanz.de

Full text and stopwords
by Ben Engbers 13 Feb '20

13 Feb '20

Hi Christian, According to the docs, a stopword list can be used to decrease the size of the full text index. I had no problems when using this list while creating a database. Is it also possible to use this list for other purposes? 1 According to XQueryX 3.1.pdf it is possible to use a sequence of stopwords in a query: /books/book[@number="1"]//p contains text "propagating of errors" using stop words ("a", "the", "of"). How can I use this list in BaseX while building querys? 2 Is it possible to add words to the list, after that is has been loaded? Suppose that it shows that my text contains a lot of names that I want to exclude. How can I add those names to the stopwords list? 3 If I want to create a Wordcloud, I want to use all the words that remain after tokenization and removing all the words from the stopwords list. (I found this item 'https://en.wikibooks.org/wiki/XQuery/Tag_Cloud'. It might be a good starting point for creating a wordcloud) Cheers, Ben

2 2

XDM metadata
by Ben Engbers 13 Feb '20

13 Feb '20

Hi Christian, According to the server protocol, when first sending \04 to the server, the resulting items from the query are returned as strings, prefixed by a single byte. With my RBaseX-package, the result from "for $i in 1 to 2 return <xml>Text { $i }</xml>" is "0b" "<xml>Text 1</xml>" "0b" "<xml>Text 2</xml>" When sending \1F the strings should be prefixed by the XDM metadata. In my case however, the output is the samen as with \04. Is it possible to get the same output for querys in the basexgui so that I can see which output should be expected? This is the last remaining problem in my package. If I can resolve it, I can upload a new version. Cheers, Ben

2 2

file:move() fails when moving to another file system
by Zimmel, Daniel 12 Feb '20

12 Feb '20

Hi, I am able to file:copy() or file:write() a file from a Linux system to a Windows mount (via cifs). I am not able to file:move() to the same target. The error says "file:io-error: Die Operation ist nicht erlaubt" BaseX version is 9.2.1 Is this a bug or a known restriction with moving things around in Java? Thanks, Daniel

3 6

basexclient "Failed to construct terminal"
by Ben Engbers 11 Feb '20

11 Feb '20

Hi, It probably has been asked before but I found nothing on this topic. Whenever I try to start basex or basexclient on my Fedora 31 linux distribution, I get this output: BaseX 9.3.1 [Client] Probeer 'help' om informatie te krijgen. [ERROR] Failed to construct terminal; falling back to unsupported java.lang.NumberFormatException: For input string: "0x100" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:580) at java.lang.Integer.valueOf(Integer.java:766) at jline.internal.InfoCmp.parseInfoCmp(InfoCmp.java:59) at jline.UnixTerminal.parseInfoCmp(UnixTerminal.java:233) at jline.UnixTerminal.<init>(UnixTerminal.java:64) at jline.UnixTerminal.<init>(UnixTerminal.java:49) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at jline.TerminalFactory.getFlavor(TerminalFactory.java:209) at jline.TerminalFactory.create(TerminalFactory.java:100) at jline.TerminalFactory.get(TerminalFactory.java:184) at jline.TerminalFactory.get(TerminalFactory.java:190) at jline.console.ConsoleReader.<init>(ConsoleReader.java:240) at jline.console.ConsoleReader.<init>(ConsoleReader.java:232) at jline.console.ConsoleReader.<init>(ConsoleReader.java:220) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.basex.util.ConsoleReader$JLineConsoleReader.<init>(ConsoleReader.java:146) at org.basex.util.ConsoleReader.get(ConsoleReader.java:55) at org.basex.BaseX.console(BaseX.java:166) at org.basex.BaseX.<init>(BaseX.java:152) at org.basex.BaseXClient.<init>(BaseXClient.java:35) at org.basex.BaseXClient.main(BaseXClient.java:22) 'basex help' gives this output: Gestopt bij /home/bengbers/, 1/5: [XPDY0002] element(help): Context is undeclared. Am I missing something? Cheers, Ben

3 9

ft search with wildcards does not work with higher Unicode characters
by Omar Siam 11 Feb '20

11 Feb '20

Hi, I just came across this strange behavior collection('dc_tunico')//*[text() contains text 'rwḥ' using wildcards] yields nothing vs collection('dc_tunico')//*[text() contains text 'rwḥ'] yields the correct result <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-vicav">rwḥ</gram> <gram xmlns="http://www.tei-c.org/ns/1.0" type="root" xml:lang="ar-aeb-x-tunis-vicav">rwḥ</gram> Any ideas why this is the case? Best regards Omar

2 14

Reg : XQUERY Performence
by chandra Sekhar 07 Feb '20

07 Feb '20

Dear Basex Team, we have one collection with 3,00,000(3 lakhs) xmls and the collection size is 4 gb,on this collection we are running xquery ,this query taking more than 20 minutes.Here we need to run xquery all 3 lakhs xmls(for report purpose we need to get data from all xmls) . Note : Same query running very quick on huge size of collction(size more than 30gb) but in this case number of xmls in that collection is less(nearly 350). Please suggest any clue to tune the query. Regards, Sekhar

1 0

how to count and remove "entities"
by thufir 04 Feb '20

04 Feb '20

Not sure of the correct lingo, but I'm building a database of tweets. As I run it, duplicate tweets are added to the database. I can see the duplicates with: for $tweets in db:open("twitter") return <tweet>{$tweets/json/id__str}</tweet> Firstly, how would I select the json node for a duplicate entity. But, before even selecting that node, recursively look to see if there's more than one result for that id__str value. How would I even generate a count of each occurrence for the data of a specific id__str? thanks, Thufir

2 5

No difference for output from 'FULL' or 'RESULTS'
by Ben Engbers 04 Feb '20

04 Feb '20

Hi, As far as I can remember when using early versions from my client-software, the main difference in output after sending \04 or \1F to the database, was that in the latter case the output was preceded with XDM Meta data. # Full query_txt <- "for $i in 1 to 2 return <xml>Text { $i }</xml>" query_obj <- Query(Session, query_txt) result <- Full(query_obj) resulted in: "0b" "<xml>Text 1</xml>" "0b" "<xml>Text 2</xml>" # Iterate over query query2 <- "for $i in 3 to 4 return <xml>Iter { $i }</xml>" query_iterate <- Query(Session, query2) # <== Alternative call to query-object while (More(query_iterate)) { cat(Next(query_iterate), "\n") } resulted in: <xml>Iter 3</xml> <xml>Iter 4</xml> Now, iterating over the same query gives: 0b <xml>Iter 3</xml> 0b <xml>Iter 4</xml> Did something change in the client/server protocol or did I introduce an error somewhere? Ben

2 2

Finalizing Query-Objects
by Ben Engbers 04 Feb '20

04 Feb '20

Hi, The people from CRAN strongly suggested to add tests (comparable to Unit-tests) to my package (RBaseX). Their request led me to take another critical look at my code. So far the tests do not give an error message. But after completing the last test, 'testthat' reports 1 failure without further explanation. After changing the order in which the tests are executed, the failure is always caused by the last test. Therefore I think that it are not the tests that cause an error, but the finalize-process. At this moment, my code is based upon 3 classes: 'RBaseXClient' creates a new client-session. This session use 'SocketClass' to communicate with basexserver. When used in query-mode, the session uses 'QueryClass' to create new query-objects. Due to this architecture, it is easy to explicitly close a regular query-object, but (at least in R) it is difficult to close query-objects when finalizing the session-object. How does the basexserver respond to closing the session without first explicitly closing all open querys? Does this result in an error? Ben

2 2

Add command: name of the input will be set as path?
by thufir 03 Feb '20

03 Feb '20

What does it mean that "if null, the name of input will be set as the path"? Javadoc: Add public Add(java.lang.String path, java.lang.String input) Constructor, specifying a target path and an input. Parameters: path - target path, optionally terminated by a new file name. If null, the name of the input will be set as path. input - input file or XML string I'm looking to add an xml file, so am using "null" for the path: https://stackoverflow.com/q/60035605/262852 but what are the implications? the "name of the input" will be "set as path"? Where is the "name of the input"? What is "path" in relation to a String which exists only in memory? Just pass a string like: new Add(null, stringXml).execute(context); and that should add to the currently open database? thanks, Thufir

2 4

convert JSON to XML to add to database
by thufir 03 Feb '20

03 Feb '20

Hi all, How is JSON converted to XML in order to ADD to a database? JSONObject jsonTweet = tweets.getJSONObject(Long.toString(id)); xmlStringTweet = XML.toString(jsonTweet); alternately, could the JSON simply get directly added to the database? see also: https://stackoverflow.com/q/60034291/262852 thanks, Thufir

2 2

BaseX and validating the entire database
by ERRINGTON Luke 03 Feb '20

03 Feb '20

Hello, We are evaluating moving from an RDBMS (Oracle), to BaseX as much of our source data originate in XML files and converting to tables in a relational schema is painful. In general BaseX looks great! However, one thing that we lose is referential integrity, and the ability to validate data in one XML file that is referring to data in another. Are there any possibilities within BaseX or an additional module that can do this? For example: * Can we validate using a schema that applies across a collection of documents, rather than just one? * Can we use Schematron (which looks cool) to apply its inteRnal XPaths to the entire collection of documents? * Or both? * Something else? We could try using XLinks, but that would involve changing our XML data/structure, and my understanding is that BaseX doesn't support (let alone validate) them, anyway. A situation I have in mind is something like (very, very simplified): A.xml <object id="1" name="One"> </object> B.xml <object id="2" name="Two"> </object> X.xml <mapping object_from_id="1" object_to_id="2" /> Is there any way to ensure that when X.xml is added to the database that the object IDs that it is referring to actually exist in the database too? I would also like to be able to ensure that all of the <object>s in the database have unique id attributes. A schema can do this within a file, but how can I ensure that when a new object xml file is added that it is not using an ID that already exists? Thanks for any answers, Luke

8 18

JSON to XML conversion
by thufir 03 Feb '20

03 Feb '20

Hi, How do I do something like: public void transform(String fileName) throws IOException { String content = new String(Files.readAllBytes(Paths.get(fileName)), StandardCharsets.UTF_8); org.json.JSONObject json = new org.json.JSONObject(content); log.info(org.json.XML.toString(json)); } but using BaseX as below? private void baseXparseJsonFile(String fileName) throws IOException { org.basex.build.json.JsonParser jsonParser = new org.basex.build.json.JsonParser(new IOFile(fileName), new MainOptions()); //where is the xml? } Unclear on how to use the different parsers (really, just need this one parser). See also: https://stackoverflow.com/q/60022419/262852 thanks, Thufir

2 1

db:optimize
by Ветошкин Владимир 30 Jan '20

30 Jan '20

3 14

Load LibreOffice- and Word-documents?
by Ben Engbers 28 Jan '20

28 Jan '20

Hi, While we were discussing possible usecases for basex, a colleague asked me if it is also possible to load libreoffice and Word documents into Basex and then perform full-text analysis on them. In essence, these are both XML files, so it should be possible. Does anybody have experience with this? Ben

3 2

NullPointerException parsing trivial query using XQJ (CLI works)
by Andreas Sewe 28 Jan '20

28 Jan '20

Hi, I get a weird NPE/parse error when interfacing with BaseX 9.3.1 via XQJ and even occurs for the following minimal query: <html>{.}</html> In other words, I just want to interpolate the context item. The gist of the XQJ calls is the following: > XQPreparedExpression expression = connection.prepareExpression("<html>{.}</html>"); > expression.bindDocument(XQConstants.CONTEXT_ITEM, "<context/>", null, null); > XQResultSequence results = expression.executeQuery(); > results.writeSequence(System.out, null); This results in the following message > XQJQS001 - Invalid XQuery syntax, syntax does not pass static validation. and a long and unfortunately obfuscated stack trace: > java.lang.NullPointerException > at net.xqj.basex.bin.bz.a(Unknown Source) > at net.xqj.basex.bin.bv.c(Unknown Source) > at net.xqj.basex.bin.bC.bE(Unknown Source) > at net.xqj.basex.bin.bC.by(Unknown Source) > at net.xqj.basex.bin.bC.bu(Unknown Source) > at net.xqj.basex.bin.bC.bj(Unknown Source) > at net.xqj.basex.bin.bC.bi(Unknown Source) > at net.xqj.basex.bin.bC.bh(Unknown Source) > at net.xqj.basex.bin.bC.aY(Unknown Source) > at net.xqj.basex.bin.bC.aX(Unknown Source) > at net.xqj.basex.bin.bC.aW(Unknown Source) > at net.xqj.basex.bin.bC.aV(Unknown Source) > at net.xqj.basex.bin.bC.aU(Unknown Source) > at net.xqj.basex.bin.bC.aT(Unknown Source) > at net.xqj.basex.bin.bC.aS(Unknown Source) > at net.xqj.basex.bin.bC.aR(Unknown Source) > at net.xqj.basex.bin.bC.aQ(Unknown Source) > at net.xqj.basex.bin.bC.aP(Unknown Source) > at net.xqj.basex.bin.bC.aO(Unknown Source) > at net.xqj.basex.bin.bC.aN(Unknown Source) > at net.xqj.basex.bin.bC.aM(Unknown Source) > at net.xqj.basex.bin.bC.aL(Unknown Source) > at net.xqj.basex.bin.bC.aK(Unknown Source) > at net.xqj.basex.bin.bC.R(Unknown Source) > at net.xqj.basex.bin.bC.Q(Unknown Source) > at net.xqj.basex.bin.bC.N(Unknown Source) > at net.xqj.basex.bin.bC.bS(Unknown Source) > at net.xqj.basex.bin.bC.bR(Unknown Source) > at net.xqj.basex.bin.bC.bM(Unknown Source) > at net.xqj.basex.bin.bC.bL(Unknown Source) > at net.xqj.basex.bin.bC.bK(Unknown Source) > at net.xqj.basex.bin.bC.by(Unknown Source) > at net.xqj.basex.bin.bC.bu(Unknown Source) > at net.xqj.basex.bin.bC.bj(Unknown Source) > at net.xqj.basex.bin.bC.bi(Unknown Source) > at net.xqj.basex.bin.bC.bh(Unknown Source) > at net.xqj.basex.bin.bC.aY(Unknown Source) > at net.xqj.basex.bin.bC.aX(Unknown Source) > at net.xqj.basex.bin.bC.aW(Unknown Source) > at net.xqj.basex.bin.bC.aV(Unknown Source) > at net.xqj.basex.bin.bC.aU(Unknown Source) > at net.xqj.basex.bin.bC.aT(Unknown Source) > at net.xqj.basex.bin.bC.aS(Unknown Source) > at net.xqj.basex.bin.bC.aR(Unknown Source) > at net.xqj.basex.bin.bC.aQ(Unknown Source) > at net.xqj.basex.bin.bC.aP(Unknown Source) > at net.xqj.basex.bin.bC.aO(Unknown Source) > at net.xqj.basex.bin.bC.aN(Unknown Source) > at net.xqj.basex.bin.bC.aM(Unknown Source) > at net.xqj.basex.bin.bC.aL(Unknown Source) > at net.xqj.basex.bin.bC.aK(Unknown Source) > at net.xqj.basex.bin.bC.R(Unknown Source) > at net.xqj.basex.bin.bC.Q(Unknown Source) > at net.xqj.basex.bin.bC.P(Unknown Source) > at net.xqj.basex.bin.bC.d(Unknown Source) > at net.xqj.basex.bin.bC.b(Unknown Source) > at net.xqj.basex.bin.bC.a(Unknown Source) > at net.xqj.basex.bin.bC.a(Unknown Source) > at net.xqj.basex.bin.bJ.<init>(Unknown Source) > at net.xqj.basex.bin.bd.a(Unknown Source) > at net.xqj.basex.bin.H.<init>(Unknown Source) > at net.xqj.basex.bin.H.a(Unknown Source) > at net.xqj.basex.bin.u.a(Unknown Source) > at net.xqj.basex.bin.u.a(Unknown Source) > at net.xqj.basex.bin.w.prepareExpression(Unknown Source) > at net.xqj.basex.bin.w.prepareExpression(Unknown Source) Executing the same query using "bin/basex -i context.xml query.xq" outputs the expected result: > <html> > <context/> > </html> I have a minimal (Maven-based) test project, in case you want to reproduce the bug. I can also file the issue on GitHub, if that's the preferred method, but if I read <https://github.com/BaseXdb/basex/issues/new> correctly, then I should post here first. Best wishes, Andreas

2 3

Test for changes in the data structure
by benjamin.tobler＠sbb.ch 28 Jan '20

28 Jan '20

Dear all I am extracting some nodes from a xml document. The structure of the xml document can change over time. Therefore, I would like to install a mechanism, which is testing for changes of the data structure of the relevant nodes. The schema is described in multiple xsd documents but one xsd document is referring to all others (screenshot). [cid:image003.jpg@01D5D502.4E0C7DB0] I was testing the complete xml against the this xsd, which is referring to all other xsds, but I received the error message: "Validation could not be started: Java heap space" by using the function "validate:xsd". However, actually I only want to test the data structure of specific nodes and not the complete xml for changes. Is it possible to do this with BaseX? Thank you for your help! Best Regards Benjamin

2 1

saxon.jar on docker image
by France Baril 26 Jan '20

26 Jan '20

Hi I'm using the public 9.3.1 image. I've been able to set up most of my app, but I get an error when trying to apply an xsl. Cannot compile and the error mention that Xalan is used. I just added the saxon9.jar in /srv/basex/lib/custom the way I would have on my normal local installation. FROM basex/basexhttp:9.3.1 ... COPY ./image-setup-files/bx/saxon9.jar /srv/basex/lib/custom/saxon9.jar 17:04:57.447 172.17.0.1:39752 admin 400 Stopped at /srv/basex/webapp/Accounting_Bilans.xqm, 301/25: [xslt:error] ERROR: 'Could not compile stylesheet' FATAL ERROR: 'Error checking type of the expression 'funcall(lower-case, [parameter-ref(legal-entity/reference)])'.' :Error checking type of the expression 'funcall(lower-case, [parameter-ref(legal-entity/reference)])'. javax.xml.transform.TransformerConfigurationException: Error checking type of the expression 'funcall(lower-case, [parameter-ref(legal-entity/reference)])'. at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl.newTemplates(TransformerFactoryImpl.java:988) at org.basex.query.func.xslt.XsltTransform.transform(XsltTransform.java:122) at org.basex.query.func.xslt.XsltTransform.transform(XsltTransform.java:62) at org.basex.query.func.xslt.XsltTransform.item(XsltTransform.java:40) at org.basex.query.expr.ParseExpr.value(ParseExpr.java:50) at org.basex.query.expr.gflwor.GFLWOR.value(GFLWOR.java:80) at org.basex.query.func.StaticFunc.invValue(StaticFu... 5305.15 ms -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

1 1

Shoud 'addraw' work as parameter with db:create?
by France Baril 25 Jan '20

25 Jan '20

Hi, I have this function that loads all the indexable content, but leaves the .png behind. Am I missing something? declare %rest:path("/demo/create-db") %rest:GET %rest:query-param('db', '{$db}', 'new-name') %rest:query-param('dir-src', '{$dir-src}', '') %output:method("html") %output:html-version("5.0") updating function democlean:create-db($db as xs:string, $dir-src as xs:string){ let $params := map {'updindex': true(), 'language': 'fr', 'addraw': true(), 'chop': false(), 'intparse': true(), 'createfilter': '*.xml, *.ditamap, *.dita'} return ( if (db:exists($db)) then db:drop($db) else (), if ($dir-src = '') then db:create($db, (), (), $params) else db:create($db, $dir-src, '/', $params), db:output(<div>DB ready</div>) ) } I also tried 'serializer: ''indent=no' as a parameter, which was unrecognized. I assume that since addraw is not unrecognized, it should work. -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

How to escape/encode a search term using BaseX REST XQ
by Ivan Kanakarakis 24 Jan '20

24 Jan '20

Hello everyone, I am using BaseX 8.44 and the REST XQ interface (ie, http://docs.basex.org/wiki/RESTXQ). I have an endpoint that, when invoked with GET, it does a full text search (using "$db-nodes[text() contains text { $term } all]"), gets the results, constructs a JSON response and sends it back. That's all fine and works great. However, I am not sure how I should be doing the queries I describe bellow. _Note: the query is initiated by a SPA javascript client, thus when I say encode/uri-escape, what I mean is that I invoke the encodeURIComponent function (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob…). _Note 2: for the sake of conversation let's consider the example endpoint declared as: %rest:GET %rest:path("/search/{$term}") 1. I want to search for "tea". That is the basic query. A single term, no problem. curl -s "https://example.com/search/tea" 2. I want to search for "tea time". Now, this query has a space in between the two words. What I expect to get back, is any node that contains both words (thus I have used "contains text" with "all"), even if they may be a few words apart. - Should I be sending an encoded/uri-escape version of this, ie, "tea%20time"? - Or, should I be replacing the space with "+", ie "tea+time"? - Or, some other advice? curl -s "https://example.com/search/tea%20time" curl -s "https://example.com/search/tea+time" 3. I want to search for "tea/time". This is even trickier. What I expect to get back, is any node that contains "tea/time", ie a search result for a single term. How do I do this? - If I do not do anything, the slash is treated as part of the URL, thus not matching a route. - If I encoded/uri-escape this term, I get "tea%2Ftime". But, when I invoke the endpoint I get the same as if there was a slash. - I am not sure how I should deal with the slash. How should I escape/encode this? curl -s "https://example.com/search/tea/time" curl -s "https://example.com/search/tea%2Ftime" Thank you,

3 5

Re: [basex-talk] HTTPServer + gzip compression
by Stefan Koch 22 Jan '20

22 Jan '20

2 1

HTTPS requests using the HTTP Client Module with self signed (invlaid) certificates
by Omar Siam 22 Jan '20

22 Jan '20

Hi, I just searched for a way to turn off certificate verification for the HTTP Client xquery module. Turns out I have to use Client/Server Architecture option IGNORECERT. Although that is somewhat logical I miss a clear statement in the docs. Thanks for the great software! Omar Siam

2 1

Server-side XQuery scripts understanding their "context"
by ERRINGTON Luke 22 Jan '20

22 Jan '20

Hi all, I'm trying to use the REST API to call server-side XQuery scripts. As I understand it, these scripts are not part of a database, but reside in the filesystem where BaseX is "installed" (or where RESTPATH points to). However, when I execute an XQuery I would like it to run on a particular database. Is there anyway to achieve this without passing the database as an argument to the query? For example, http://localhost:8984/rest?run=find.xq http://localhost:8984/rest/Test?run=find.xq ... both execute the same query, and that query needs to open a database to do its searching, so currently the database to open is hard-coded. It would be nice if the query could work out what database or "collection" was its context and automatically use that. Additionally, if I do http://localhost:8984/rest?run=Test/find.xq ... then I run the query that is in the Test folder, but this still does not have any context and so still needs to know to open the Test database. Is there a way to achieve this without having to pass the name of the database to every XQuery? Or am I not understanding things and is there a better way to do this sort of thing? Thanks, Luke

2 3

Help with a Query/Performance
by Tom Rauchenwald (UNIFITS) 22 Jan '20

22 Jan '20

Hi list, I'm struggling with a query. We have XML documents with a structure similar to this: <Container> <ContainerMetaData1>FOO</ContainerMetaData1> <ContainerMetaData2>FOO</ContainerMetaData2> <MessageA> <MessageAMetaData> <MessageMetaData1>FOO</MessageMetaData1> <MessageMetaData2>FOO</MessageMetaData2> </MessageAMetaData> <MessageADetail> <DetailData1>FOO</DetailData1> <DetailData2>FOO</DetailData2> </MessageADetail> <MessageADetail> <DetailData1>FOO</DetailData1> <DetailData2>FOO</DetailData2> </MessageADetail> </MessageA> <MessageB> <MessageBMetaData> <MessageMetaData1>FOO</MessageMetaData1> <MessageMetaData2>FOO</MessageMetaData2> </MessageBMetaData> <MessageBDetail> <DetailData1>FOO</DetailData1> <DetailData2>FOO</DetailData2> </MessageBDetail> </MessageB> <MessageC> <MessageCMetaData> <MessageMetaData1>FOO</MessageMetaData1> <MessageMetaData2>FOO</MessageMetaData2> </MessageCMetaData> <MessageCDetail> <DetailData1>FOO</DetailData1> <DetailData2>FOO</DetailData2> </MessageCDetail> </MessageC> </Container> Messages are bundled in a container (up to n times for each message), and each message has details (also up to n times). Container, Message contain data that is the same for all details (it's basically a grouping). I'd like to retrieve a Detail with all corresponding data associated with it, so basically a MessageADetail, MessageA (without all the other MessageADetails), Container (without all the other Messages). I know the position of the message (i.e., I know that I want the second MessageA for example), and I know the position of the Detail (i.e., I know that I want the 3rd Detail). The use case is to show the detail in context in a UI. The query to do this I came up with is (here I want to get the 2nd detail from the third MessageA): let $fh := (copy $x := /*:Container modify ( delete node $x/*:MessageA[position() != 3] , delete node $x/*:MessageA[3]/*:MessageADetail[position() != 2] , delete node $x/*:MessageB , delete node $x/*:MessageC ) return $x) return $fh This works well for small documents. For large documents it can take a couple of seconds to evaluate the query (our real-life documents do have more data/elements in Details and Message). I'm wondering if there's a better/more efficient way to do this. I tried formulating a query that doesn't do deletes, but I couldn't come up with a solution that performs better and is correct. Any pointers would be very much appreciated. Here's a function to generate sufficiently large test data: declare function local:sample($numberOfMessages, $numberOfDetails) { <Container> <ContainerMetaData1>FOO</ContainerMetaData1> <ContainerMetaData2>FOO</ContainerMetaData2> {for $i in 1 to $numberOfMessages return <MessageA> <MessageAMetaData> <MessageMetaData1>FOO {$i}</MessageMetaData1> <MessageMetaData2>FOO {$i}</MessageMetaData2> </MessageAMetaData> {for $j in 1 to $numberOfDetails return <MessageADetail> <DetailData1>FOO {$j}</DetailData1> <DetailData2>FOO {$j}</DetailData2> </MessageADetail> } </MessageA> } <MessageB> <MessageBMetaData> <MessageMetaData1>FOO</MessageMetaData1> <MessageMetaData2>FOO</MessageMetaData2> </MessageBMetaData> <MessageBDetail> <DetailData1>FOO</DetailData1> <DetailData2>FOO</DetailData2> </MessageBDetail> </MessageB> <MessageC> <MessageCMetaData> <MessageMetaData1>FOO</MessageMetaData1> <MessageMetaData2>FOO</MessageMetaData2> </MessageCMetaData> <MessageCDetail> <DetailData1>FOO</DetailData1> <DetailData2>FOO</DetailData2> </MessageCDetail> </MessageC> </Container> }; db:create('tr-test', local:sample(20, 100000), 'test.xml') Thanks, Tom Rauchenwald

3 4

Help
by Pedro Sousa 20 Jan '20

20 Jan '20

Hello guys. I need to develop one API that must compare prices of laptops of two websites Fnac and Worten. I don't know to much about XML ( it's a college degree and it's one of last summarys that's i had left) Here it is what i need to do: It is intended to develop an application that allows you to make a price comparison of Laptops sold at Fnac and Worten online stores. Information Extraction, Structuring and Storage To preserve and share product data from each online store, such as price history, create an XML vocabulary that can capture all storage-related requirements. Thus, communication between applications is supported by a specific vocabulary and independent of the tools used to record prices used in the mentioned online stores. Always keep in mind that this vocabulary may be distributed in the future to various “partners” so that they can communicate or price their products and price comparator developed. The following vocabulary objectives were created: ● Represent practical store-identifying information, including the name, primary web address, and web addresses used for information extraction; ● Represent useful information with the products involved in the price comparator, namely: name, characteristics, brand, link to an offer page and their prices over time, including data / time when this information was collected; ● For each store, display aggregate data information where you can use: average price of each product (considering various amounts of data) not only considering the entire price history, but also a specific period (eg month); ● Represent practical information with partner pricing, including: the name of the online store and previously used product data (for example, if any online store wants to publish their pricing on the platform, it will use a used component of vocabulary). ● You can add new data elements to the vocabulary in order to enrich the whole process (this component will be enhanced). Information Processing and Availability The documents generated and valid according to the developed vocabulary must be stored in a database and made available through a REST API using the tool B aseX. You can use the Postman tool to extract data. The data must be extracted periodically (for example, hourly) from the referred addresses, and must later be stored in an XML database with BaseX. The data stored in the database must then be exposed through a REST API designed to support integration with applications developed by third parties. The API should (at a minimum) provide: ● data on a specific product with or without a price history (it should be possible to select one of the options); ● data about the products of a specific brand; ● data about a specific online store for one or more products; ● aggregated data for the online store (including the average price of products and maximum and minimum prices in a given period of time); ● price communication for partner products; ● it should also be possible to share information via twitter (for example, what is the best deal taking into account the price differences of a particular laptop). The number of endpoints, such as the form of interaction with the API, that can be parsed more expressively and consistently, but it is not possible to identify resources, but also in the REST verbs used in HTTP requests. Identifying common products between different stores is an extremely important process since without this identification it will not be possible to make a price comparison. In addition to adopting techniques to automate the process of identifying common products, it can provide manual mechanisms. For example, you can create a document that stores as mailings that 100% of common products could not be detected and allow a user to annotate or validate a document (yes or no) from the mailing. Data visualization It should also include a specific endpoint to provide an HTML document with a set of views related to vocabulary aggregated data components. To do this, select an HTML document to objectively display the required information. You can run quickchart.io to generate views and integrate documents into the available HTML (s). Overview and tools You should take advantage of the tools studied throughout the semester to use each step. The combination of tools to achieve the proposed objectives is at the discretion of each working group. As an example, Figure 1 provides an overview of a possible adjustment of different related concepts and technologies. must be delivered: ● A set of XML schemas capable of validating all syntax rules defined for the language and associated types; ● Evidence of the API developed using Postman, as well as its documentation. You can deliver a GIT repository; ● XQuery files that define the REST API developed in the BaseX tool; ● Examples of documents that allow you to test the solution (for example: sample document that features a typical response / request from each of the endpoints). ● A detailed report (which may be in .pdf format or made available through the GIT repository) where you must identify: ○ Documentation (example) of the vocabulary that justifies the decisions made as well as possible changes (with due justification) to the vocabulary throughout the project. For requirements that you consider ambiguous, you should justify the approach followed; ○ Identify the approach followed for data storage in the BaseX database, as well as all procedures / queries used to manipulate / adapt the data entered; ○ Description and documentation of the services developed; ○ Critical appreciation of the work developed; I'm asking for somebody that can help me or atleast give me some lights to start the project.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk