Hello !
We have to load many files (>20.000), large size (>30Gb) into BaseX. Today, that's done vis a BXS script which accesses files thru a network share.
I would like to optimize network transfert ; tar / compression at source, uncompress / untar at destination. But I would like to avoid to create temporary files (at source, on NAS server, and on destination server, BaseX host).
So, is there an API which I could use to load a "file" thru a stream ? I think I can write an inputstream that redirects data to outpustreams, cutting input data in many output streams, one per file. But may I chain this outputstream to an inputstream usable by BaseX loader ?
Best regards, Christophe
Hello Christophe,
Are you using java ? Maybe you should have a look at the add() method in :
https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/or...
I can also remember that command line tools can read standard input stream with the '-' argument.
Cordialement,
Fabrice Etanchaud Horanet Smart Objects
-----Message d'origine----- De : basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] De la part de Christophe Marchand Envoyé : mardi 19 janvier 2016 12:20 À : basex-talk@mailman.uni-konstanz.de Objet : [basex-talk] Streaming file loads
Hello !
We have to load many files (>20.000), large size (>30Gb) into BaseX. Today, that's done vis a BXS script which accesses files thru a network share.
I would like to optimize network transfert ; tar / compression at source, uncompress / untar at destination. But I would like to avoid to create temporary files (at source, on NAS server, and on destination server, BaseX host).
So, is there an API which I could use to load a "file" thru a stream ? I think I can write an inputstream that redirects data to outpustreams, cutting input data in many output streams, one per file. But may I chain this outputstream to an inputstream usable by BaseX loader ?
Best regards, Christophe
basex-talk@mailman.uni-konstanz.de