I found the proximate cause: Not having UPDINDEX set to true() on the failing server.

When I set UPDINDEX to true on the target database, then the load succeeded.

Tracking through the Java code it looks like a failure to initialize the idmap member of the DiskData object. This code looks like it could be the issue:

// open data and indexes
init();
if(meta.updindex) {
idmap = new IdPreMap(meta.dbFile(DATAIDP));
if(meta.textindex) textIndex = new UpdatableDiskValues(this, IndexType.TEXT);
if(meta.attrindex) attrIndex = new UpdatableDiskValues(this, IndexType.ATTRIBUTE);
if(meta.tokenindex) tokenIndex = new UpdatableDiskValues(this, IndexType.TOKEN);
} else {
if(meta.textindex) textIndex = new DiskValues(this, IndexType.TEXT);
if(meta.attrindex) attrIndex = new DiskValues(this, IndexType.ATTRIBUTE);
if(meta.tokenindex) tokenIndex = new DiskValues(this, IndexType.TOKEN);
}
if(meta.ftindex) ftIndex = new FTIndex(this);

The idmap is only set of meta.updIndex is true.

Cheers,

_____________________________________________

Eliot Kimber

Sr Staff Content Engineer

O: 512 554 9368

M: 512 554 9368

servicenow.com

LinkedIn | Twitter | YouTube | Facebook

From: Eliot Kimber <eliot.kimber@servicenow.com>
Date: Tuesday, January 23, 2024 at 11:51 AM
To: basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de>
Subject: Consistent NPE loading content on one server, other server works

I have two servers running the same code, both on 10.7. I have a REST API handler that takes data as input and stores it in a database, applying some preprocessing to the data first.

The data successfully loads on my dev server and consistently fails on the production server, with this Java failure:

[INFO] Posting "/Users/eliot.kimber/git-basex/product-content-analytics/analytics/adobe/adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv"

[INFO] as filename "adobe-analytics-Utah_Pages_Viewed_Aug2023-clean.csv"

[INFO] to Mirabel server "http://mirabel.corp.service-now.com:9984"...

Unexpected error: Improper use? Potential bug? Your feedback is welcome:

Contact: basex-talk@mailman.uni-konstanz.de

Version: BaseX 10.7

Java: Oracle Corporation, 17.0.8

OS: Linux, amd64

Stack Trace:

java.lang.NullPointerException: Cannot invoke "org.basex.index.IdPreMap.write(org.basex.io.IOFile)" because "this.idmap" is null

at org.basex.data.DiskData.write(DiskData.java:151)

at org.basex.data.DiskData.close(DiskData.java:160)

at org.basex.core.Datas.unpin(Datas.java:52)

at org.basex.core.cmd.Close.close(Close.java:45)

at org.basex.core.cmd.OptimizeAll.optimizeAll(OptimizeAll.java:124)

at org.basex.query.up.primitives.db.DBOptimize.apply(DBOptimize.java:119)

at org.basex.query.up.DataUpdates.applyDbUpdates(DataUpdates.java:213)

at org.basex.query.up.DataUpdates.apply(DataUpdates.java:172)

at org.basex.query.up.ContextModifier.apply(ContextModifier.java:120)

at org.basex.query.up.Updates.apply(Updates.java:179)

at org.basex.query.QueryContext.update(QueryContext.java:663)

at org.basex.query.QueryContext.lambda$iter$4(QueryContext.java:357)

at org.basex.query.QueryContext.run(QueryContext.java:766)

at org.basex.query.QueryContext.iter(QueryContext.java:357)

at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:78)

at org.basex.http.web.WebResponse.create(WebResponse.java:58)

at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72)

at org.basex.http.BaseXServlet.service(BaseXServlet.java:69)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)

at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)

at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319)

at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273)

at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)

at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

at org.eclipse.jetty.server.Server.handle(Server.java:516)

at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)

at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

at java.base/java.lang.Thread.run(Thread.java:833)[INFO] Data loaded.

The two servers are essentially identical Linux servers, although the production server has more resources.

Any idea what might cause this failure or what I can do to diagnose it?

There must be some non-obvious difference in these two servers but I don’t know what to look for.

The code doing the load to a database is:

let $reportPath as xs:string :=

``[/`{$analyticsmgmt:csvPathRoot}`/`{$source}`/`{$reportType}`/`{$reportName}`.xml]``

let $msg := prof:dump(``[[INFO] Storing analytics report "`{$reportPath}`"]``)

let $csv := $csv transform with { insert node attribute {'timestamp'} {$timeStampStr} into ./* }

let $csv := $csv transform with { analyticsmgmt:dispatchAnalysticsStorePreProcessing($reportName, ., $debug) }

return

try {

(db:put($analyticsDb, $csv, $reportPath),

db:optimize($analyticsDb,

true(),

map{

'attrindex' : true(),

'tokenindex' : true(),

'updindex' : true()

})

)

} catch * {

util:logToConsole(

'analyticsmgmt:storeAnalyticsCsvReport',

``[`{$err:code}`: `{$err:description}`]``,

'error'

)

}

Thanks,