Streaming large (binary) data

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Streaming large (binary) data

Martin Renner
Hi,

I have to export lots of objects from a database into a XML file. These objects contain large BLOBs,
so I started to implement a "BlobConverter". As the BLOBs are very large, I cannot use the following
approach:

   public void marshal(Object source, HierarchicalStreamWriter writer, MarshallingContext context) {
      Blob blob = (Blob) source;
      context.convertAnother(blob.getBytes(1, (int) blob.length()));

This would read the whole BLOB into a byte array and delegate to EncodedByteArrayConverter, which in
turn would write a base64 encoded string. This creates two arrays in the heap (the byte array and
the char array with the base64 encoded data), which are way too large to fit into the available heap
space.

So I am looking for a solution which would allow to stream the binary data into a streaming base64
encoder and to stream that into the output stream. However, I could not find any method in
HierarchicalStreamWriter which would allow to stream character data. There is just "void
setValue(String text)", which requires the whole data to be in the heap.

Is it possible to stream character data to the output stream?


Regards,
Martin


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Streaming large (binary) data

Jörg Schaible-4
Hi Martin,

sorry for the long delay. However, you should use the user's list for
questions.

Martin Renner wrote:

> Hi,
>
> I have to export lots of objects from a database into a XML file. These
> objects contain large BLOBs, so I started to implement a "BlobConverter".
> As the BLOBs are very large, I cannot use the following approach:
>
>    public void marshal(Object source, HierarchicalStreamWriter writer,
>    MarshallingContext context) {
>       Blob blob = (Blob) source;
>       context.convertAnother(blob.getBytes(1, (int) blob.length()));
>
> This would read the whole BLOB into a byte array and delegate to
> EncodedByteArrayConverter, which in turn would write a base64 encoded
> string. This creates two arrays in the heap (the byte array and the char
> array with the base64 encoded data), which are way too large to fit into
> the available heap space.
>
> So I am looking for a solution which would allow to stream the binary data
> into a streaming base64 encoder and to stream that into the output stream.
> However, I could not find any method in HierarchicalStreamWriter which
> would allow to stream character data. There is just "void setValue(String
> text)", which requires the whole data to be in the heap.
>
> Is it possible to stream character data to the output stream?

XStream has no direct support writing or reading an element's text with
streams (XSTR-532). However, although outside of the XStream specification,
it should be possible to call setValue multiple times for stream-based XML
drivers. Therefore you could use a Base64 stream (e.g. from Apache commons-
codec) to write chunks into a buffer and write that one with setValue before
continuing with the next chunk.

- Jörg


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Streaming large (binary) data

Tatu Saloranta
One related thing worth noting is that Woodstox and Aalto Stax parsers implement Stax2 extension API, and support both typed access to base64 encoded content, and allow that to be done incrementally (block by block). Both for reading and writing. So this is a very efficient way of dealing with large binary content in this context (probably more so than trying to use attachments).

Interfaces used are XMLStreamReader2 / XMLStreamWriter2, implemented by readers/writers that Woodstox and Aalto return.

-+ Tatu +-


-----


On Friday, September 26, 2014 12:27 AM, Jörg Schaible <[hidden email]> wrote:


Hi Martin,

sorry for the long delay. However, you should use the user's list for
questions.

Martin Renner wrote:

> Hi,
>
> I have to export lots of objects from a database into a XML file. These
> objects contain large BLOBs, so I started to implement a "BlobConverter".
> As the BLOBs are very large, I cannot use the following approach:
>
>    public void marshal(Object source, HierarchicalStreamWriter writer,
>    MarshallingContext context) {
>      Blob blob = (Blob) source;
>      context.convertAnother(blob.getBytes(1, (int) blob.length()));
>
> This would read the whole BLOB into a byte array and delegate to
> EncodedByteArrayConverter, which in turn would write a base64 encoded
> string. This creates two arrays in the heap (the byte array and the char
> array with the base64 encoded data), which are way too large to fit into
> the available heap space.
>
> So I am looking for a solution which would allow to stream the binary data
> into a streaming base64 encoder and to stream that into the output stream.
> However, I could not find any method in HierarchicalStreamWriter which
> would allow to stream character data. There is just "void setValue(String
> text)", which requires the whole data to be in the heap.
>
> Is it possible to stream character data to the output stream?

XStream has no direct support writing or reading an element's text with
streams (XSTR-532). However, although outside of the XStream specification,
it should be possible to call setValue multiple times for stream-based XML
drivers. Therefore you could use a Base64 stream (e.g. from Apache commons-
codec) to write chunks into a buffer and write that one with setValue before
continuing with the next chunk.




- Jörg


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email