New subject: Stopword-related NullPointerException (in FTWords.java)

30 Aug 2015


      Hi,
I encountered a peculiar error with a query using a stopwords file in the context of a full text search. The query joins two XML databases: CT.gov (containing 86635503 nodes) and the 2015 MeSH dictionary (containing 12064461 nodes). I am debugging using CT.gov trial NCT00303472, hardcoded in the ‘where' clause of the following query:
let $trees := db:open('MeSH')/DescriptorRecordSet/DescriptorRecord
for $article in db:open('CTGov')/clinical_study
where $article/id_info/nct_id = 'NCT00303472'
let $mesh := $article/condition_browse/mesh_term
let $tn1 := $trees[DescriptorName/String contains text { $mesh }]
let $tn2 := $trees[DescriptorName/String contains text { $mesh } using stop words at "/Volumes/Extra/Documents/Standards/MeSH/stopwords.txt"]/TreeNumberList/TreeNumber
return <match> { $article/id_info/nct_id, $mesh, $tn2 } </match>
When the return clause contains the variable $tn2 (i.e., using stopwords - as shown above) a Java NullPointerException is generated (see the stack trace below). However, when only $tn1 is returned there is no problem (the code for $tn2 is removed by the optimizer).
The issue is related to a specific stopword (“syndrome”). When the stopword is removed from the file the exception does not occur. Surprisingly, when the stopword is in uppercase (“Syndrome”) the issue does not occur - even though the target MeSH term in this CT.gov trial is in uppercase, that is
<mesh_term>Syndrome</mesh_term>
Am I doing something wrong, or is this a real bug in BaseX? If the former, please suggest a workaround as I would like to filter out generic MeSH terms that match the stopwords before any further processing (I removed a lot of code from the above query to make it easier to debug).
Thanks,
Ron
Error:
Improper use? Potential bug? Your feedback is welcome:
Contact: basex-talk@mailman.uni-konstanz.de
Version: BaseX 8.2
Java: Oracle Corporation, 1.8.0_20
OS: Mac OS X, x86_64
Stack Trace: 
java.lang.NullPointerException
    at org.basex.query.expr.ft.FTWords$1.next(FTWords.java:166)
    at org.basex.query.expr.ft.FTIndexAccess$1.next(FTIndexAccess.java:48)
    at org.basex.query.expr.ft.FTIndexAccess$1.next(FTIndexAccess.java:45)
    at org.basex.query.iter.Iter.value(Iter.java:53)
    at org.basex.query.expr.ParseExpr.value(ParseExpr.java:67)
    at org.basex.query.QueryContext.value(QueryContext.java:421)
    at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:41)
    at org.basex.query.expr.path.CachedPath.iter(CachedPath.java:22)
    at org.basex.query.QueryContext.iter(QueryContext.java:410)
    at org.basex.query.expr.List$1.next(List.java:133)
    at org.basex.query.expr.constr.Constr.add(Constr.java:70)
    at org.basex.query.expr.constr.CElem.item(CElem.java:92)
    at org.basex.query.expr.constr.CElem.item(CElem.java:23)
    at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:43)
    at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99)
    at org.basex.query.MainModule$1.next(MainModule.java:114)
    at org.basex.query.QueryContext.cache(QueryContext.java:660)
    at org.basex.query.QueryProcessor.cache(QueryProcessor.java:103)
    at org.basex.core.cmd.AQuery.query(AQuery.java:83)
    at org.basex.core.cmd.XQuery.run(XQuery.java:22)
    at org.basex.core.Command.run(Command.java:398)
    at org.basex.core.Command.execute(Command.java:100)
    at org.basex.gui.GUI.exec(GUI.java:472)
    at org.basex.gui.GUI.access$400(GUI.java:43)
    at org.basex.gui.GUI$7.run(GUI.java:412)
Compiling:
- pre-evaluating db:open("MeSH")
- pre-evaluating db:open("CTGov")
- inlining $trees_0
- applying full-text index for { $mesh_2 } using language 'English'
- applying full-text index for { $mesh_2 } using language 'English'
- inlining $tn2_4
- removing variable $tn1_3
- applying text index for "NCT00303472"
- rewriting where clause(s)
Query:
let $trees := db:open('MeSH')/DescriptorRecordSet/DescriptorRecord for $article in db:open('CTGov')/clinical_study where $article/id_info/nct_id = 'NCT00303472' let $mesh := $article/condition_browse/mesh_term let $tn1 := $trees[DescriptorName/String contains text { $mesh }] let $tn2 := $trees[DescriptorName/String contains text { $mesh } using stop words at "/Volumes/Extra/Documents/Standards/MeSH/stopwords.txt"]/TreeNumberList/TreeNumber return <match> { $article/id_info/nct_id, $mesh, $tn2 } </match>
Optimized Query:
for $article_1 in db:text("CTGov", "NCT00303472")/parent::*:nct_id/parent::*:id_info/parent::*:clinical_study let $mesh_2 := $article_1/*:condition_browse/*:mesh_term return element match { (($article_1/*:id_info/*:nct_id, $mesh_2, ft:search("MeSH", { $mesh_2 } using language 'English')/parent::*:String/parent::*:DescriptorName/parent::*:DescriptorRecord/TreeNumberList/TreeNumber)) }
Query plan:
<QueryPlan compiled="true">
  <GFLWOR>
    <For>
      <Var name="$article" id="1"/>
      <IterPath>
        <ValueAccess data="CTGov" type="TEXT" name="*:nct_id">
          <Str value="NCT00303472" type="xs:string"/>
        </ValueAccess>
        <IterStep axis="parent" test="*:id_info"/>
        <IterStep axis="parent" test="*:clinical_study"/>
      </IterPath>
    </For>
    <Let>
      <Var name="$mesh" id="2"/>
      <IterPath>
        <VarRef>
          <Var name="$article" id="1"/>
        </VarRef>
        <IterStep axis="child" test="*:condition_browse"/>
        <IterStep axis="child" test="*:mesh_term"/>
      </IterPath>
    </Let>
    <CElem>
      <QNm value="match" type="xs:QName"/>
      <List>
        <IterPath>
          <VarRef>
            <Var name="$article" id="1"/>
          </VarRef>
          <IterStep axis="child" test="*:id_info"/>
          <IterStep axis="child" test="*:nct_id"/>
        </IterPath>
        <VarRef>
          <Var name="$mesh" id="2"/>
        </VarRef>
        <CachedPath>
          <FTIndexAccess data="MeSH">
            <FTWords>
              <VarRef>
                <Var name="$mesh" id="2"/>
              </VarRef>
            </FTWords>
          </FTIndexAccess>
          <IterStep axis="parent" test="*:String"/>
          <IterStep axis="parent" test="*:DescriptorName"/>
          <IterStep axis="parent" test="*:DescriptorRecord"/>
          <IterStep axis="child" test="TreeNumberList"/>
          <IterStep axis="child" test="TreeNumber"/>
        </CachedPath>
      </List>
    </CElem>
  </GFLWOR>
</QueryPlan>
Ron Katriel, Ph.D. | Senior Data Scientist | Medidata Solutions
350 Hudson Street, 7th Floor, New York, NY 10014
rkatriel@mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800