Did you specify "text/comma-separated-values" as Content-Type? Am 12.06.2016 8:05 nachm. schrieb "Alexander Shpack" shadowkin@gmail.com:
Thank you, Christian!
But it doesn't work as expected
declare %rest:path("/upload") %rest:POST("{$csv}") %input:csv("header=false,encoding=CP1251,separator=semicolon") %output:method("html") function page:uploadFile($csv as document-node()) { $csv };
[XPTY0004] Cannot cast xs:string to document-node(): "?????????? ?....
On Sun, Jun 12, 2016 at 6:48 PM, Christian Grün <christian.gruen@gmail.com
wrote:
Hi Alex,
The following annotation should do what you want:
%input:csv("header=true,encoding=CP1252")You were completely right, the existing documentation was a bit scrappy. I have slightly rewritten it (see [1] and outgoing links).
Hope thie helps, Christian
[1] http://docs.basex.org/wiki/RESTXQ#Input_options
On Sun, Jun 12, 2016 at 4:03 PM, Alexander Shpack shadowkin@gmail.com wrote:
Hi,
I have nonstandard CSV file with semicolon instead of comma as a
separator
and cp1251 encoded. Documentation is not clear in this case, and the
sample
doesn't work properly
declare %rest:path("/store.csv") %rest:POST("{$csv}") %input:csv("header=true") function page:store-csv($csv as document-node()) { "Number of rows: " || count($csv/csv/record) };
Even if I changed the encoding manually the next code does nothing %input:csv("separator=semicolon") and returns text instead of document-node()
Could you check it?
It would be nice if you add additional encoding parameter to csv module.
Thanks!
-- s0rr0w
No, browser doesn't want to send file content with this mime type. The other thing, RFC describes "text/csv" mime type. Are you sure that "text/comma-separated-values" is the correct one?
On Sun, Jun 12, 2016 at 9:41 PM, Christian Grün christian.gruen@gmail.com wrote:
Did you specify "text/comma-separated-values" as Content-Type? Am 12.06.2016 8:05 nachm. schrieb "Alexander Shpack" <shadowkin@gmail.com
:
Thank you, Christian!
But it doesn't work as expected
declare %rest:path("/upload") %rest:POST("{$csv}") %input:csv("header=false,encoding=CP1251,separator=semicolon") %output:method("html") function page:uploadFile($csv as document-node()) { $csv };
[XPTY0004] Cannot cast xs:string to document-node(): "?????????? ?....
On Sun, Jun 12, 2016 at 6:48 PM, Christian Grün < christian.gruen@gmail.com> wrote:
Hi Alex,
The following annotation should do what you want:
%input:csv("header=true,encoding=CP1252")You were completely right, the existing documentation was a bit scrappy. I have slightly rewritten it (see [1] and outgoing links).
Hope thie helps, Christian
[1] http://docs.basex.org/wiki/RESTXQ#Input_options
On Sun, Jun 12, 2016 at 4:03 PM, Alexander Shpack shadowkin@gmail.com wrote:
Hi,
I have nonstandard CSV file with semicolon instead of comma as a
separator
and cp1251 encoded. Documentation is not clear in this case, and the
sample
doesn't work properly
declare %rest:path("/store.csv") %rest:POST("{$csv}") %input:csv("header=true") function page:store-csv($csv as document-node()) { "Number of rows: " || count($csv/csv/record) };
Even if I changed the encoding manually the next code does nothing %input:csv("separator=semicolon") and returns text instead of document-node()
Could you check it?
It would be nice if you add additional encoding parameter to csv
module.
Thanks!
-- s0rr0w
No, browser doesn't want to send file content with this mime type.
If you use the browser to send requests, it may be better to convert your data to CSV via XQuery:
csv:parse($input, map { "header": false(), "separator": "semicolon" })
How does your html look like, which you use for sending the request via the browser?
"header=false,encoding=CP1251,separator=semicolon"
RFC describes "text/csv" mime type. Are you sure that "text/comma-separated-values" is the correct one?
True, text/csv should work as well!
Not really up to speed with this but I would expect the separator to be the symbol, no?
"header=false,encoding=CP1251,separator=;" or map{"header": false(), "separator": ";"} /Andy
On 12 June 2016 at 20:26, Christian Grün christian.gruen@gmail.com wrote:
No, browser doesn't want to send file content with this mime type.
If you use the browser to send requests, it may be better to convert your data to CSV via XQuery:
csv:parse($input, map { "header": false(), "separator": "semicolon" })
How does your html look like, which you use for sending the request via the browser?
"header=false,encoding=CP1251,separator=semicolon"
RFC describes "text/csv" mime type. Are you sure that "text/comma-separated-values" is the correct one?
True, text/csv should work as well!
Hi Andy,
Not really up to speed with this but I would expect the separator to be the symbol, no?
That’s possible as well. Currently, you can also pass on some keywords [1].
Cheers, Christian
[1] http://docs.basex.org/wiki/CSV_Module#Options
"header=false,encoding=CP1251,separator=;" or map{"header": false(), "separator": ";"} /Andy
On 12 June 2016 at 20:26, Christian Grün christian.gruen@gmail.com wrote:
No, browser doesn't want to send file content with this mime type.
If you use the browser to send requests, it may be better to convert your data to CSV via XQuery:
csv:parse($input, map { "header": false(), "separator": "semicolon" })
How does your html look like, which you use for sending the request via the browser?
"header=false,encoding=CP1251,separator=semicolon"
RFC describes "text/csv" mime type. Are you sure that "text/comma-separated-values" is the correct one?
True, text/csv should work as well!
On Sun, Jun 12, 2016 at 10:26 PM, Christian Grün christian.gruen@gmail.com wrote:
No, browser doesn't want to send file content with this mime type.
If you use the browser to send requests, it may be better to convert your data to CSV via XQuery:
csv:parse($input, map { "header": false(), "separator": "semicolon" })
I've tried it, it works. Almost fine. Now I have an issue with encoding. Seems like 'encoding' parameter was not applied
let $csv := csv:parse($csv, map{ 'separator': ';', 'header': false(), *'encoding': 'CP1251'* } )
How does your html look like, which you use for sending the request
via the browser?
<form action="/upload" method="POST" enctype="multipart/form-data"> <input type="file" name="csv"/> <button type="submit">Send</button> </form>
I've tried it, it works. Almost fine. Now I have an issue with encoding. Seems like 'encoding' parameter was not applied
let $csv := csv:parse($csv, map{ 'separator': ';', 'header': false(), 'encoding': 'CP1251' } )
This should work:
declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body) { let $string := convert:binary-to-string($body, 'CP1251') return csv:parse($string, map{ 'separator': ';', 'header': false(), };
No luck with you code
[XPTY0004] Binary expected, xs:string found.
On Sun, Jun 12, 2016 at 10:50 PM, Christian Grün christian.gruen@gmail.com wrote:
I've tried it, it works. Almost fine. Now I have an issue with encoding. Seems like 'encoding' parameter was not applied
let $csv := csv:parse($csv, map{ 'separator': ';', 'header': false(), 'encoding': 'CP1251' } )
This should work:
declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body) { let $string := convert:binary-to-string($body, 'CP1251') return csv:parse($string, map{ 'separator': ';', 'header': false(), };
Weird. I used the attached RESTXQ functions. Maybe you did something differently?
On Sun, Jun 12, 2016 at 9:55 PM, Alexander Shpack shadowkin@gmail.com wrote:
No luck with you code
[XPTY0004] Binary expected, xs:string found.
On Sun, Jun 12, 2016 at 10:50 PM, Christian Grün christian.gruen@gmail.com wrote:
I've tried it, it works. Almost fine. Now I have an issue with encoding. Seems like 'encoding' parameter was not applied
let $csv := csv:parse($csv, map{ 'separator': ';', 'header': false(), 'encoding': 'CP1251' } )
This should work:
declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body) { let $string := convert:binary-to-string($body, 'CP1251') return csv:parse($string, map{ 'separator': ';', 'header': false(), };
-- s0rr0w
OMG.... Basex 8.4.4 returns different error for the browsers
Chrome [bxerr:BXCO0001] String conversion: Invalid XML character (#18).
FF and IE11 [XPTY0004] Binary expected, xs:string found.
OMG.... Basex 8.4.4 returns different error for the browsers
Interestingly. I created a file with a single #18 character, and all 3 browsers you mentioned give me the same error message (Invalid XML character). Did you always upload the same file?
As the error indicates, it seems that your CSV input seems to contain characters that are not valid in XML. There are various ways to tackle this; one looks as follows:
(: interpret client data as Base64 :) declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { (: replace invalid characters with a question mark :) let $input := bin:from-octets( bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else 63) ) (: convert to XQuery Unicode string; convert to XML :) let $string := bin:decode-string($input, 'CP1251') return csv:parse($string) };
There may be easier solutions as well (I’ll give you an update once I remember them ;).
Did you always upload the same file?
Yeah
As the error indicates, it seems that your CSV input seems to contain characters that are not valid in XML. There are various ways to tackle this; one looks as follows:
(: interpret client data as Base64 :) declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { (: replace invalid characters with a question mark :) let $input := bin:from-octets( bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else 63) ) (: convert to XQuery Unicode string; convert to XML :) let $string := bin:decode-string($input, 'CP1251') return csv:parse($string) };
There may be easier solutions as well (I’ll give you an update once I remember them ;).
Strange solution... Ok, I'll wait for a better one.
One more thing. I've written the next code in GUI, and it works perfect.
let $fName := "test.csv" let $content := file:read-text($fName, "cp1251") let $csv := csv:parse($content, map{ 'separator': ';', 'header': false() } ) return $csv
What the difference between post data and raw file?
On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack shadowkin@gmail.com wrote:
Did you always upload the same file?
Yeah
As the error indicates, it seems that your CSV input seems to contain characters that are not valid in XML. There are various ways to tackle this; one looks as follows:
(: interpret client data as Base64 :) declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { (: replace invalid characters with a question mark :) let $input := bin:from-octets( bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else 63) ) (: convert to XQuery Unicode string; convert to XML :) let $string := bin:decode-string($input, 'CP1251') return csv:parse($string) };
There may be easier solutions as well (I’ll give you an update once I remember them ;).
Strange solution... Ok, I'll wait for a better one.
-- s0rr0w
This line tells you the difference:
let $content := file:read-text($fName, "cp1251")
You are reading the file contents as text (i.e., string). You could also use file:read-binary. Using the browser, you can also upload images or other binary contents.
let $csv := csv:parse($content, map{ 'separator': ';', 'header': false() } ) return $csv
What the difference between post data and raw file?
On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack shadowkin@gmail.com wrote:
Did you always upload the same file?
Yeah
As the error indicates, it seems that your CSV input seems to contain characters that are not valid in XML. There are various ways to tackle this; one looks as follows:
(: interpret client data as Base64 :) declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { (: replace invalid characters with a question mark :) let $input := bin:from-octets( bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else 63) ) (: convert to XQuery Unicode string; convert to XML :) let $string := bin:decode-string($input, 'CP1251') return csv:parse($string) };
There may be easier solutions as well (I’ll give you an update once I remember them ;).
Strange solution... Ok, I'll wait for a better one.
-- s0rr0w
-- s0rr0w
Oh.. right. I forgot about it
On Mon, Jun 13, 2016 at 12:02 AM, Christian Grün christian.gruen@gmail.com wrote:
This line tells you the difference:
let $content := file:read-text($fName, "cp1251")
You are reading the file contents as text (i.e., string). You could also use file:read-binary. Using the browser, you can also upload images or other binary contents.
let $csv := csv:parse($content, map{ 'separator': ';', 'header': false()
} )
return $csv
What the difference between post data and raw file?
On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack shadowkin@gmail.com wrote:
Did you always upload the same file?
Yeah
As the error indicates, it seems that your CSV input seems to contain characters that are not valid in XML. There are various ways to tackle this; one looks as follows:
(: interpret client data as Base64 :) declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { (: replace invalid characters with a question mark :) let $input := bin:from-octets( bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then .
else
- ) (: convert to XQuery Unicode string; convert to XML :) let $string := bin:decode-string($input, 'CP1251') return csv:parse($string)
};
There may be easier solutions as well (I’ll give you an update once I remember them ;).
Strange solution... Ok, I'll wait for a better one.
-- s0rr0w
-- s0rr0w
Hi Alex,
I decided to add another function signature to the convert:binary-to-string function [1]: If you set the third parameter to true(), invalid XML characters will automatically be replaced with \uFFFD (the so-called Unicode replacement character):
declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { let $string := convert:binary-to-string($input, 'CP1251', true()) return csv:parse($string) };
A new stable snapshot is available [2]; the final version of BaseX 8.5 will be released in June or July.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Conversion_Module#convert:binary-to-string [2] http://files.basex.org/releases/latest/
On Sun, Jun 12, 2016 at 11:06 PM, Alexander Shpack shadowkin@gmail.com wrote:
Oh.. right. I forgot about it
On Mon, Jun 13, 2016 at 12:02 AM, Christian Grün christian.gruen@gmail.com wrote:
This line tells you the difference:
let $content := file:read-text($fName, "cp1251")
You are reading the file contents as text (i.e., string). You could also use file:read-binary. Using the browser, you can also upload images or other binary contents.
let $csv := csv:parse($content, map{ 'separator': ';', 'header': false() } ) return $csv
What the difference between post data and raw file?
On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack shadowkin@gmail.com wrote:
Did you always upload the same file?
Yeah
As the error indicates, it seems that your CSV input seems to contain characters that are not valid in XML. There are various ways to tackle this; one looks as follows:
(: interpret client data as Base64 :) declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { (: replace invalid characters with a question mark :) let $input := bin:from-octets( bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else 63) ) (: convert to XQuery Unicode string; convert to XML :) let $string := bin:decode-string($input, 'CP1251') return csv:parse($string) };
There may be easier solutions as well (I’ll give you an update once I remember them ;).
Strange solution... Ok, I'll wait for a better one.
-- s0rr0w
-- s0rr0w
-- s0rr0w
Super! Thank you!
On Mon, Jun 13, 2016 at 12:49 AM, Christian Grün christian.gruen@gmail.com wrote:
Hi Alex,
I decided to add another function signature to the convert:binary-to-string function [1]: If you set the third parameter to true(), invalid XML characters will automatically be replaced with \uFFFD (the so-called Unicode replacement character):
declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { let $string := convert:binary-to-string($input, 'CP1251', true()) return csv:parse($string) };
A new stable snapshot is available [2]; the final version of BaseX 8.5 will be released in June or July.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/Conversion_Module#convert:binary-to-string [2] http://files.basex.org/releases/latest/
On Sun, Jun 12, 2016 at 11:06 PM, Alexander Shpack shadowkin@gmail.com wrote:
Oh.. right. I forgot about it
On Mon, Jun 13, 2016 at 12:02 AM, Christian Grün <
christian.gruen@gmail.com>
wrote:
This line tells you the difference:
let $content := file:read-text($fName, "cp1251")
You are reading the file contents as text (i.e., string). You could also use file:read-binary. Using the browser, you can also upload images or other binary contents.
let $csv := csv:parse($content, map{ 'separator': ';', 'header':
false()
} ) return $csv
What the difference between post data and raw file?
On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack <
shadowkin@gmail.com>
wrote:
Did you always upload the same file?
Yeah
As the error indicates, it seems that your CSV input seems to
contain
characters that are not valid in XML. There are various ways to
tackle
this; one looks as follows:
(: interpret client data as Base64 :) declare %rest:path("/upload") %rest:POST("{$body}") function local:store-csv($body as xs:base64Binary) { (: replace invalid characters with a question mark :) let $input := bin:from-octets( bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else 63) ) (: convert to XQuery Unicode string; convert to XML :) let $string := bin:decode-string($input, 'CP1251') return csv:parse($string) };
There may be easier solutions as well (I’ll give you an update once
I
remember them ;).
Strange solution... Ok, I'll wait for a better one.
-- s0rr0w
-- s0rr0w
-- s0rr0w
basex-talk@mailman.uni-konstanz.de
-
Alexander Shpack -
Andy Bunce -
Christian Grün