Re: [basex-talk] [EXTERNAL] - Re: Re. Encoding support for the BaseX server - BaseX-Talk - mailman.uni-konstanz.de

7 Sep 2020

      Hi Srikumar,
...

Without "<?xml version="1.0" encoding="ISO-8859-1"?>" encoding line in XML file, we are failing store XML file(i.e., ISO-8859-1 encoded file) with below error:
                     [FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence.

Have you revised the way how you export files from BaseX? Are you
still using String.getBytes() ?
If you want your exported docs to be ISO-8859-1-encoded, and if you
want to output the XML declaration, you can declare serialization
options in the prolog of your query [1].
Best,
Christian
[1] https://docs.basex.org/wiki/Serialization
On Mon, Sep 7, 2020 at 9:47 AM Srikumar Choudhury schoudhu@opentext.com wrote:
...
Hi Christian,
               1. Currently by default, we are able to store UTF-8 encoding XML files.
               2. Storing of different encoding files are succeeding if we specify the below line:
                    <?xml version="1.0" encoding="ISO-8859-1"?>
               3. Without "<?xml version="1.0" encoding="ISO-8859-1"?>" encoding line in XML file, we are failing store XML file(i.e., ISO-8859-1 encoded file) with below error:
                    [FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence.
                  How do we make basex server to store certain format of encoded file(In above example, it is ISO-8859-1 encoded file)?

Thanks,
Srikumar

From: Christian Grün christian.gruen@gmail.com
Sent: Sunday, September 6, 2020 2:05 PM
To: Sharath Babu Pulumati spulumat@opentext.com
Cc: James Ball basex-talk@jamesball.co.uk; BaseX basex-talk@mailman.uni-konstanz.de; Rajiv Pandya rpandya@opentext.com; Srikumar Choudhury schoudhu@opentext.com
Subject: Re: [EXTERNAL] - Re: Re. Encoding support for the BaseX server
Please note that your code has various flaws that might lead to the
erroneous result. For example, you should avoid calling
String.getBytes, as it will use the default character encoding of your
platform, not necessarily UTF-8 (see [1]).
I have attached the output that I get for your updated example (it
doesn’t trigger any errors). Did you check the encoding of your
document that’s output by your Java code
(outputxml_06000001800029e2p.xml)?
[1] https://urldefense.com/v3/__https://docs.oracle.com/javase/7/docs/api/java/l...
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
createDatabase: started creating database=DEMO_ENCODING
[10:29:20.124]  INFO: XQConnection-1 created, the databaseName
property was never specified at the XQDataSource level.
xqs.getConnection completed
[10:29:20.124] TRACE: XQConnection.getStaticContext() invoked.
[10:29:20.125] TRACE: XQConnection-1.createExpression() invoked.
[10:29:20.183] TRACE: XQConnection.getStaticContext() invoked.
[10:29:20.270] DEBUG: XQExpression-1.close() invoked.
[10:29:20.270] DEBUG: XQExpression-1.close() invoked.
createDatabase: completed creating database=DEMO_ENCODING
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
xqs.getConnection completed
[10:29:20.278] TRACE: XQConnection.getStaticContext() invoked.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
StoreDocument: BaseX test store ANSI encoded xml doc
[10:29:20.279] TRACE: XQConnection-2.createExpression() invoked.
[10:29:20.279] TRACE: XQConnection.getStaticContext() invoked.
[10:29:20.308] TRACE: XQExpression-2.executeQuery(), raw XQuery
resolved to: db:replace('DEMO_ENCODING',
'6300000180000114/06000001800029e2p.xml','c:\Users\user\Desktop\p.xml')
xqpe.executeQuery completed
[10:29:20.844] DEBUG: XQExpression-2.close() invoked.
StoreDocument: BaseX content push for ANSI encoding is completed
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
RetrieveDocument: BaseX test retrieve xml doc
[10:29:20.844] TRACE: XQConnection-2.createExpression() invoked.
[10:29:20.844] TRACE: XQConnection.getStaticContext() invoked.
[10:29:20.844] TRACE: XQExpression-3.executeQuery(), raw XQuery
resolved to: db:open('DEMO_ENCODING',
'6300000180000114/06000001800029e2p.xml')
xqpe.executeQuery completed
RetrieveDocument: BaseX test retrieve xml doc completed
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ReStoreDocument: BaseX test store ANSI encoded xml doc
[10:29:20.851] TRACE: XQConnection-2.createExpression() invoked.
[10:29:20.851] TRACE: XQConnection.getStaticContext() invoked.
[10:29:20.851] TRACE: XQExpression-4.executeQuery(), raw XQuery
resolved to: db:replace('DEMO_ENCODING',
'6300000180000114/06000001800029e2p_re_store.xml','c:\Users\user\Desktop\outputxml_06000001800029e2p.xml')
xqpe.executeQuery completed
[10:29:20.858] DEBUG: XQExpression-4.close() invoked.
ReStoreDocument: BaseX content push for ANSI encoding is completed
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On Sun, Sep 6, 2020 at 10:19 AM Sharath Babu Pulumati
spulumat@opentext.com wrote:
...
Hi Christian,
Below is the error stack trace's stating the error during the file uploading using the both BaseX GUI & Sample program:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
By Using the BaseX GUI:
Error:
Stopped at ., 1/11:
[FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence.
Optimized Query:
db:replace("DEMO_ENCODING", "6300000180000114/06000001800029e2p_re_store.xml", "C:\Sharath\outputxml_06000001800029e2p.xml")
Query:
db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p_re_store.xml','C:\Sharath\outputxml_06000001800029e2p.xml')
Query Plan:
<QueryPlan compiled="true" updating="false">
  <DbReplace name="db:replace" type="empty-sequence()" size="0">
    <Str type="xs:string" size="1">DEMO_ENCODING</Str>
    <Str type="xs:string" size="1">6300000180000114/06000001800029e2p_re_store.xml</Str>
    <Str type="xs:string" size="1">C:\Sharath\outputxml_06000001800029e2p.xml</Str>
  </DbReplace>
</QueryPlan>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
By Using the Sample Program:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
createDatabase: started creating database=DEMO_ENCODING
[13:30:33.103]  INFO: XQConnection-1 created, the databaseName property was never specified at the XQDataSource level.
xqs.getConnection completed
[13:30:33.103] TRACE: XQConnection.getStaticContext() invoked.
[13:30:33.103] TRACE: XQConnection-1.createExpression() invoked.
[13:30:33.135] TRACE: XQConnection.getStaticContext() invoked.
[13:30:33.432] DEBUG: XQExpression-1.close() invoked.
[13:30:33.432] DEBUG: XQExpression-1.close() invoked.
createDatabase: completed creating database=DEMO_ENCODING
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
xqs.getConnection completed
[13:30:33.432] TRACE: XQConnection.getStaticContext() invoked.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
StoreDocument: BaseX test store ANSI encoded xml doc
[13:30:33.447] TRACE: XQConnection-2.createExpression() invoked.
[13:30:33.447] TRACE: XQConnection.getStaticContext() invoked.
[13:30:33.510] TRACE: XQExpression-2.executeQuery(), raw XQuery resolved to: db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p.xml','C:\Sharath\p.xml')
xqpe.executeQuery completed
[13:30:34.369] DEBUG: XQExpression-2.close() invoked.
StoreDocument: BaseX content push for ANSI encoding is completed
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
RetrieveDocument: BaseX test retrieve xml doc
[13:30:34.385] TRACE: XQConnection-2.createExpression() invoked.
[13:30:34.385] TRACE: XQConnection.getStaticContext() invoked.
[13:30:34.400] TRACE: XQExpression-3.executeQuery(), raw XQuery resolved to: db:open('DEMO_ENCODING', '6300000180000114/06000001800029e2p.xml')
xqpe.executeQuery completed
RetrieveDocument: BaseX test retrieve xml doc completed
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ReStoreDocument: BaseX test store ANSI encoded xml doc
[13:30:34.416] TRACE: XQConnection-2.createExpression() invoked.
[13:30:34.416] TRACE: XQConnection.getStaticContext() invoked.
[13:30:34.416] TRACE: XQExpression-4.executeQuery(), raw XQuery resolved to: db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p_re_store.xml','C:\Sharath\outputxml_06000001800029e2p.xml')
javax.xml.xquery.XQException:
  BXCE001 - Stopped at ., 1/11:
[FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence.
  at com.test.BasexTest.ReStoreDocument(BasexTest.java:159)
  at com.test.BasexTest.main(BasexTest.java:32)
ReStoreDocument: BaseX content push for ANSI encoding is completed
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
FYI, I have attached the new sample program along with the snippet collected from BaseX GUI while reproducing the issue.
My query is regarding, why i'm unable to import the document which is exported from basex?
Thanks,
Sharath.

From: Christian Grün christian.gruen@gmail.com
Sent: Saturday, September 5, 2020 10:44 PM
To: Sharath Babu Pulumati spulumat@opentext.com
Cc: James Ball basex-talk@jamesball.co.uk; BaseX basex-talk@mailman.uni-konstanz.de; Rajiv Pandya rpandya@opentext.com; Srikumar Choudhury schoudhu@opentext.com
Subject: Re: [EXTERNAL] - Re: Re. Encoding support for the BaseX server
Please find the attached zip, which contains the sample program for replicating the encoding issue with the provided datafiles.
Could you please review the provided sample and  help me in resolving this issue.
After having created "testenvSSL1" database, and after having fixed your local paths, I ran your example, and it didn’t trigger any errors.
Please give me more information on what you believe goes wrong.