Is there a function to change output method dynamically ?
Yes, that’s possible: You can use web:response-header in order to assign a dynamic Content-Type header [1]. The chapter on Content Negotiation [2] may be relevant to you as well, if you want to use a single URL path but multiple RESTXQ functions for it.
[1] http://docs.basex.org/wiki/Web_Module#web:response-header [2] http://docs.basex.org/wiki/RESTXQ#Content_Negotiation