gphilip gphilip - 1 year ago 34
Java Question

Xuggle can't open in-memory input

I am working on a program that integrates Hadoop's MapReduce framework with Xuggle. For that, I am implementing a

class that reads and writes from and to in-memory Hadoop data objects.

You can see the relevant code here:

The idea is to register each
object in the
class with a UUID so that when I later refer to that name while opening the file it returns a
instance that is attached to that
object and I can read and write from and to memory.

The problem is that I get an exception like this:

java.lang.RuntimeException: could not open: byteswritable:d68ce8fa-c56d-4ff5-bade-a4cfb3f666fe

(see also under the posted link)

When debugging I see that the objects are correctly found in the factory, what's more, they are even being read from in the protocol handler. If I remove the listeners from/to the output file, the same happens, so the problem is already with the input. Digging deeper in the code of Xuggle I reach the JNI code (which tries to open the file) and I can't get further than this. This apparently returns an error code.


I would really appreciate some hint where to go next, how should I continue debugging. Maybe my implementation has a flaw, but I can't see it.

Answer Source

I think the problem you are running into is that a lot of the types of inputs/outputs are converted to a native file descriptor in the IContainer JNI code, but the thing you are passing cannot be converted. It may not be possible to create your own IURLProtocolHandler in this way, because it would, after a trip through, just end up calling IContainer again and then into the IContainer JNI code which will probably try to get a native file descriptor and call avio_open().

However, there may be a couple of things that you can open in IContainer which are not files/have no file descriptors, and which would be handled correctly. The things you can open can be seen in the IContainer code, namely and (and the corresponding inputs). I recommend making your DataInput/DataOutput implementation which wraps around BytesReadable/BytesWriteable, and opening it in IContainer.

If that doesn't work, then write your inputs to a temp file and read the outputs from a temp file :)