Paging takes time. - BaseX-Talk - mailman.uni-konstanz.de

20 Jan 2014


      Hi,
I have huge XML document containing around 70000000 child nodes. I want to
implement paging on this, so that I can fetch a chunk of nodes at time, but
as the start position value increases from 10000000 the execution time
takes more than a second to execute, is there a any optimized way I can
follow.
I have written following code:
public class XQJ
{
  public static void main(String[] args) throws XQException
  {
    XQDataSource xqs = new BaseXXQDataSource();
    xqs.setProperty("serverName", "localhost");
    xqs.setProperty("port", "1984");
// Change USERNAME and PASSWORD values
    XQConnection conn = xqs.getConnection("admin", "admin");
XQPreparedExpression xqpe =
    conn.prepareExpression("declare namespace xbrli='
http://www.xbrl.org/2003/instance';" +
            " declare variable $doc as xs:string external;" +
            " declare variable $start as xs:integer external;" +
            " declare variable $pageSize as xs:integer external;" +
            " let $allMatches := doc($doc)/xbrli:xbrl/*" +
            " return subsequence($allMatches,$start,$pageSize)");
//            " return $matches");
xqpe.bindString(new QName("doc"),
"1389962424906/1389962424906/facts.xml", null);
    int totalRecords = 70000000;
    int pageSize = 10000;
    int noOfPages = totalRecords/pageSize;
    long startTime = 0;
    int start = 0;
    for(int i=0; i<noOfPages; i++){
        startTime = System.currentTimeMillis();
        start = i*pageSize;
xqpe.bindInt(new QName("start"), start, null);
        xqpe.bindInt(new QName("pageSize"), pageSize, null);
XQResultSequence rs = xqpe.executeQuery();
//        while(rs.next()){}
//      System.out.println(rs.getItemAsString(null));
//        System.out.println(start + " : " +
(System.currentTimeMillis()-startTime));
    }
    System.out.println(start + " : " +
(System.currentTimeMillis()-startTime));
    conn.close();
  }