Hi, Neven,

Yes, that's what I'll probably end up doing. My real motive for trying to process the file as text was to test a BaseX extension function for parsing n-triples, using one of Gunther Rademacher's REx Parser Generator parsers[1].

I added a Java module with the parser to BaseX, and it works very well with small to medium sized files, generating a parse tree based on the EBNF for the format. I was just curious to see how it would perform with very large files.

Best,

Tim

[1] http://www.bottlecaps.de/rex/

--
Tim A. Thompson
Metadata Librarian (Spanish/Portuguese Specialty)
Princeton University Library

On Fri, Nov 25, 2016 at 12:47 PM, Neven Jovanović <filologanoga@gmail.com> wrote:

Hi Tim,

may I suggest that you convert the n-triples file to RDF/XML format
(I'm using Pythong rdflib for such tasks,
<http://rdflib.readthedocs.io/en/stable/>)? Perhaps it would be easier
for BaseX to ingest the XML instead of text format (which it thinks
the n-triples are).

Best,

Neven

Neven Jovanovic, Zagreb

On 25 November 2016 at 18:21, Christian Grün <christian.gruen@gmail.com> wrote:
> Hi Tim,
>
> In BaseX, texts/strings are internally represented as byte arrays. Due
> to the 32 bit limitation of Java arrays, the file will be too large to
> be oped as single text in main-memory.
>
> To be honest, I didn’t have a similar use case before, so I guess the
> best solution for now will be to split the file into smaller chunks
> before processing it with BaseX.
>
> Cheers,
> Christian
>
>
>
> On Fri, Nov 25, 2016 at 6:07 PM, Tim Thompson <timathom@gmail.com> wrote:
>> Hello,
>>
>> I have a large file[1] (3.5G unzipped) in the n-triples RDF format that I
>> would like to work with in BaseX. When I try to read in the file using
>> file:read-text(), I get the following error:
>>
>> Error:
>> Version: BaseX 8.6 beta 8fa97ca
>> Java: Oracle Corporation, 1.8.0_73
>> OS: Linux, amd64
>> Stack Trace:
>> java.lang.NegativeArraySizeException
>> at java.util.Arrays.copyOf(Arrays.java:3236)
>> at org.basex.util.TokenBuilder.addByte(TokenBuilder.java:247)
>> at org.basex.util.TokenBuilder.add(TokenBuilder.java:176)
>> at org.basex.io.in.TextInput.cache(TextInput.java:143)
>> at org.basex.io.in.TextInput.content(TextInput.java:132)
>> at org.basex.query.value.item.StrStream.materialize(StrStream.java:71)
>> at org.basex.query.value.item.StrStream.string(StrStream.java:44)
>> at org.basex.query.expr.ParseExpr.toToken(ParseExpr.java:273)
>> at org.basex.query.expr.ParseExpr.toEmptyToken(ParseExpr.java:261)
>> at org.basex.query.func.fn.FnSubstring.item(FnSubstring.java:22)
>> at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44)
>> at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99)
>> at org.basex.query.scope.MainModule$1.next(MainModule.java:122)
>> at org.basex.query.QueryContext.cache(QueryContext.java:648)
>> at org.basex.query.QueryProcessor.cache(QueryProcessor.java:116)
>> at org.basex.core.cmd.AQuery.query(AQuery.java:87)
>> at org.basex.core.cmd.XQuery.run(XQuery.java:22)
>> at org.basex.core.Command.run(Command.java:255)
>> at org.basex.core.Command.execute(Command.java:93)
>> at org.basex.gui.GUI.exec(GUI.java:479)
>> at org.basex.gui.GUI.access$3(GUI.java:433)
>> at org.basex.gui.GUI$7.run(GUI.java:421)
>>
>> When I try to create a text database using the GUI, I get an error stating
>> that the file could not be parsed.
>>
>> Is it possible to work with text files that are this large using BaseX?
>>
>> Thank you,
>> Tim
>>
>> [1] Available for download here:
>> http://www.bne.es/es/Inicio/Perfiles/Bibliotecarios/DatosEnlazados/DescargaFicheros/
>> (http://datos.bne.es/datadumps/autoridades.nt.bz2)
>>
>> --
>> Tim A. Thompson
>> Metadata Librarian (Spanish/Portuguese Specialty)
>> Princeton University Library
>>