Hi Alexander,

The attached 4 files that should let you reproduce the issue. The XML files are the minimal imports (fail, work) and the XQ files are the queries (fail, work). Below is the output from the query that works. The one that fails returns nothing.

Thanks, Ron

<drug>   <ATC5>Lepirudin</ATC5>   <ATC4>Direct thrombin inhibitors</ATC4>   <ATC3>ANTITHROMBOTIC AGENTS</ATC3>   <ATC2>ANTITHROMBOTIC AGENTS</ATC2>   <ATC1>BLOOD AND BLOOD FORMING ORGANS</ATC1> </drug>

On September 1, 2017 at 11:13:58 AM, Alexander Holupirek (alex@holupirek.de) wrote:

On 1. Sep 2017, at 17:06, Ron Katriel rkatriel@mdsol.com wrote:

Hi,

I ran into a peculiar issue with importing an XML file from DrugBank into BaseX. It involves the file header, which looks like this:

<?xml version="1.0" encoding="UTF-8"?>

<drugbank xmlns="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.drugbank.ca&d=DwIFAg&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=4gh7MwX9kgAiuOIjM8N3S_zHFOutxVbKozwC0xz90BA&s=JdGaEkE3BkQuY6eLHh9eY-oEZ9HB2ZW9J3k6Hft-eaU&e= " xmlns:xsi="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.w3.org_2001_XMLSchema-2Dinstance&d=DwIFAg&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=4gh7MwX9kgAiuOIjM8N3S_zHFOutxVbKozwC0xz90BA&s=Tepbq62aOfIniXVpKLDYbXjyTd36uY8BK1aN4VG-UkQ&e= " xsi:schemaLocation="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.drugbank.ca&d=DwIFAg&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=4gh7MwX9kgAiuOIjM8N3S_zHFOutxVbKozwC0xz90BA&s=JdGaEkE3BkQuY6eLHh9eY-oEZ9HB2ZW9J3k6Hft-eaU&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__www.drugbank.ca_docs_drugbank.xsd&d=DwIFAg&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=4gh7MwX9kgAiuOIjM8N3S_zHFOutxVbKozwC0xz90BA&s=e9xaYijF4iXv_DD0kkUyk2njpqeFntG6_tYvv5rwZNQ&e= " version="5.0" exported-on="2017-08-31">

The XML file (which is quite large ~3GB) loads in cleanly but queries executed against it produce no output (no errors reported).

When I change the header to the following and re-import the file everything works fine:

<?xml version="1.0" encoding="UTF-8"?>

<drugbank>

I had this issue for a while (go thought this every month) so thought it might be good to share. I am currently using BaseX 8.6.4.

Any idea why this is happening? Attached is the console trace for both cases (they show subtle differences in the query plans).

Thanks, Ron

Hi Ron,

is it possible to reproduce the error on a reduced, smaller dataset and a simplified query that you can post to the list? Maybe a namespace issue?

Thanks Alex

Thanks Martin. Below is the modified query with the namespace specification and usage. It works but is cumbersome.

Is there a way simpler way around this - other than modifying the input header to remove the namespace declaration?

Thanks, Ron

declare namespace xs="http://www.drugbank.ca";

for $drug in db:open('DrugBankFail')/xs:drugbank/xs:drug where not(empty($drug/xs:atc-codes/xs:atc-code)) return <drug> {   <ATC5> { string-join(distinct-values($drug/xs:name), ' | ') } </ATC5>,   <ATC4> { string-join(distinct-values(for $level in $drug/xs:atc-codes/xs:atc-code/xs:level return if (fn:string-length($level/@code) = 5) then $level/text() else ()), ' | ') } </ATC4>,   <ATC3> { string-join(distinct-values(for $level in $drug/xs:atc-codes/xs:atc-code/xs:level return if (fn:string-length($level/@code) = 4) then $level/text() else ()), ' | ') } </ATC3>,   <ATC2> { string-join(distinct-values(for $level in $drug/xs:atc-codes/xs:atc-code/xs:level return if (fn:string-length($level/@code) = 3) then $level/text() else ()), ' | ') } </ATC2>,   <ATC1> { string-join(distinct-values(for $level in $drug/xs:atc-codes/xs:atc-code/xs:level return if (fn:string-length($level/@code) = 1) then $level/text() else ()), ' | ') } </ATC1> } </drug>

On September 1, 2017 at 12:48:21 PM, Martin Honnen (martin.honnen@gmx.de) wrote:

On 01.09.2017 18:03, Ron Katriel wrote:

The attached 4 files that should let you reproduce the issue. The XML files are the minimal imports (fail, work) and the XQ files are the queries (fail, work). Below is the output from the query that works. The one that fails returns nothing.

It appears to be the usual problem of dealing with a default namespace in the input XML, your path drugbank/drug would select elements in no namespace unless you set up a default element namespace (which however with XQuery would also change the namespace of the result elements) or declare a prefix in your query for the input XML namespace and use that or as a third option use a wildcard *:drugbank/*:drug.

On 1. Sep 2017, at 19:41, Ron Katriel rkatriel@mdsol.com wrote: Is there a way simpler way around this - other than modifying the input header to remove the namespace declaration?

Hi Ron,

declaring a default element in your XQuery might help?

```xquery declare default element namespace "http://www.drugbank.ca";

for $drug in doc('drugbank.Lepirudin.ATC.fail.xml')/drugbank/drug where not(empty($drug/atc-codes/atc-code)) return <drug> { <ATC5> { string-join(distinct-values($drug/name), ' | ') } </ATC5>, <ATC4> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 5) then $level/text() else ()), ' | ') } </ATC4>, <ATC3> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 4) then $level/text() else ()), ' | ') } </ATC3>, <ATC2> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 3) then $level/text() else ()), ' | ') } </ATC2>, <ATC1> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 1) then $level/text() else ()), ' | ') } </ATC1> } </drug>

```

Cheers, Alex

On 9/1/17, 1:04 PM, "basex-talk-bounces@mailman.uni-konstanz.de on behalf of Martin Honnen" <basex-talk-bounces@mailman.uni-konstanz.de on behalf of martin.honnen@gmx.de> wrote:

On 01.09.2017 22:01, Alexander Holupirek wrote: >> On 1. Sep 2017, at 19:41, Ron Katriel rkatriel@mdsol.com wrote: >> Is there a way simpler way around this - other than modifying the input header to remove the namespace declaration?

> declaring a default element in your XQuery might help? > > ```xquery > declare default element namespace "https://urldefense.proofpoint.com/v2/url?u=http-3A__www.drugbank.ca&d=Dw... "; > > for $drug in doc('drugbank.Lepirudin.ATC.fail.xml')/drugbank/drug > where not(empty($drug/atc-codes/atc-code)) > return <drug> {

But that would also put the result elements into that namespace, not sure whether that is wanted.

If all of the documents are similar and there is enough code to justify the effort of pre-processing the documents to change the default namespace to no namespace, you could do that:

declare namespace e = "http://example.com";

declare function e:strip-namespaces($node as node()) as node() { typeswitch ($node) case $node as document-node() return document { $node/node()/e:strip-namespaces(.) } case $node as element() return element {local-name($node)} { $node/@*, $node/node()/e:strip-namespaces(.) } default return $node };

for $drug in e:strip-namespaces(db:open('DrugBankFail'))/drugbank/drug where not(empty($drug/atc-codes/atc-code)) return <drug> { <ATC5> { string-join(distinct-values($drug/name), ' | ') } </ATC5>, <ATC4> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 5) then $level/text() else ()), ' | ') } </ATC4>, <ATC3> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 4) then $level/text() else ()), ' | ') } </ATC3>, <ATC2> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 3) then $level/text() else ()), ' | ') } </ATC2>, <ATC1> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 1) then $level/text() else ()), ' | ') } </ATC1> } </drug>

Hi Kendall,

Yes, your solution works too (see query below). Really appreciate your help!

Best, Ron

declare namespace e = "http://example.com";

declare function e:strip-namespaces($node as node()) as node() {    typeswitch ($node)    case $node as document-node() return document { $node/node()/e:strip-namespaces(.) }    case $node as element() return element {local-name($node)} { $node/@*, $node/node()/e:strip-namespaces(.) }    default return $node  };

for $drug in e:strip-namespaces(db:open('DrugBankFail'))/drugbank/drug where not(empty($drug/atc-codes/atc-code)) return <drug> {   <ATC5> { string-join(distinct-values($drug/name), ' | ') } </ATC5>,   <ATC4> { string-join(distinct-values($drug/atc-codes/atc-code/level[string-length(@code) = 5]), ' | ') } </ATC4>,   <ATC3> { string-join(distinct-values($drug/atc-codes/atc-code/level[string-length(@code) = 4]), ' | ') } </ATC3>,   <ATC2> { string-join(distinct-values($drug/atc-codes/atc-code/level[string-length(@code) = 3]), ' | ') } </ATC2>,   <ATC1> { string-join(distinct-values($drug/atc-codes/atc-code/level[string-length(@code) = 1]), ' | ') } </ATC1> } </drug>

On September 1, 2017 at 5:21:11 PM, Kendall Shaw (kendall.shaw@workday.com) wrote:

I think my mail client altered my post to move ‘.’ characters to the end of what it thinks is a sentence.

This:

e:strip-namespaces().

Is supposed to be this:

e:strip-namespaces(.)

Hi Ron,

Adding another opinion the board (I meant to write this already on Saturday) ;-) I don’t think there is much to optimize w.r.t. to the XQuery update solution, as your whole database will be held in memory during the update… YET: if you do not care for the namespaces you could still strip them when creating your database:

``` db:create( 'mydb-no-ns', '<xml xmlns="foo"><_></_></xml>', 'mydoc2.xml', map {"stripns": "yes"} )```

http://docs.basex.org/wiki/Database_Module#db:create http://docs.basex.org/wiki/Database_Module#db:create => • parsing options will only impact string input (URIs, XML strings), because nodes have already been parsed.

Maybe this helps, I am not sure if someone already mentioned it! Best

Michael

Am 02.09.2017 um 00:38 schrieb Ron Katriel rkatriel@mdsol.com:

I can go with the default namespace declaration so no followup is needed, unless you are curious and have the time to investigate.

Kendall/Alex,

Adding the default namespace solved the issue (see the modified query below).

The output indeed has the namespace

<drug xmlns="http://www.drugbank.ca">   <ATC5>Lepirudin</ATC5>   <ATC4>Direct thrombin inhibitors</ATC4>   <ATC3>ANTITHROMBOTIC AGENTS</ATC3>   <ATC2>ANTITHROMBOTIC AGENTS</ATC2>   <ATC1>BLOOD AND BLOOD FORMING ORGANS</ATC1> </drug>

However, this is not a problem for me as I serialize it into tab-separated format.

So I am good now.

Thanks! Ron

declare default element namespace "http://www.drugbank.ca";

for $drug in db:open('DrugBankFail')/drugbank/drug where not(empty($drug/atc-codes/atc-code)) return <drug> {   <ATC5> { string-join(distinct-values($drug/name), ' | ') } </ATC5>,   <ATC4> { string-join(distinct-values($drug/atc-codes/atc-code/level[string-length(@code) = 5]), ' | ') } </ATC4>,   <ATC3> { string-join(distinct-values($drug/atc-codes/atc-code/level[string-length(@code) = 4]), ' | ') } </ATC3>,   <ATC2> { string-join(distinct-values($drug/atc-codes/atc-code/level[string-length(@code) = 3]), ' | ') } </ATC2>,   <ATC1> { string-join(distinct-values($drug/atc-codes/atc-code/level[string-length(@code) = 1]), ' | ') } </ATC1> } </drug>

On September 1, 2017 at 5:07:35 PM, Kendall Shaw (kendall.shaw@workday.com) wrote:

On 9/1/17, 1:04 PM, "basex-talk-bounces@mailman.uni-konstanz.de on behalf of Martin Honnen" <basex-talk-bounces@mailman.uni-konstanz.de on behalf of martin.honnen@gmx.de> wrote:

On 01.09.2017 22:01, Alexander Holupirek wrote:

...

On 1. Sep 2017, at 19:41, Ron Katriel rkatriel@mdsol.com wrote: Is there a way simpler way around this - other than modifying the input header to remove the namespace declaration?

declaring a default element in your XQuery might help?

declare default element namespace "https://urldefense.proofpoint.com/v2/url?u=http-3A__www.drugbank.ca&d=DwICaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=txjXxCSiNw_iW8D4o67JHNWxnyvM5vySI1NZIbu5_aI&s=53cXA-gr8txDllT9Vth7fcG5TfJLR7SbBp_PXu-POYg&e= ";  

for $drug in doc('drugbank.Lepirudin.ATC.fail.xml')/drugbank/drug  
where not(empty($drug/atc-codes/atc-code))  
return <drug> {  

But that would also put the result elements into that namespace, not sure whether that is wanted.

If all of the documents are similar and there is enough code to justify the effort of pre-processing the documents to change the default namespace to no namespace, you could do that:

declare namespace e = "https://urldefense.proofpoint.com/v2/url?u=http-3A__example.com&d=DwIGaQ... ";

declare function e:strip-namespaces($node as node()) as node() { typeswitch ($node) case $node as document-node() return document { $node/node()/e:strip-namespaces(). } case $node as element() return element {local-name($node)} { $node/@*, $node/node()/e:strip-namespaces(). } default return $node };

for $drug in e:strip-namespaces(db:open('DrugBankFail'))/drugbank/drug where not(empty($drug/atc-codes/atc-code)) return <drug> { <ATC5> { string-join(distinct-values($drug/name), ' | ') } </ATC5>, <ATC4> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 5) then $level/text() else ()), ' | ') } </ATC4>, <ATC3> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 4) then $level/text() else ()), ' | ') } </ATC3>, <ATC2> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 3) then $level/text() else ()), ' | ') } </ATC2>, <ATC1> { string-join(distinct-values(for $level in $drug/atc-codes/atc-code/level return if (fn:string-length($level/@code) = 1) then $level/text() else ()), ' | ') } </ATC1> } </drug>