Noxalus Noxalus -4 years ago 147
Android Question

Unable to render an Android WebView into an external texture shared between C++ and Java code

I'm currently trying to render an Android WebView content into a texture that can be used in a C++ application using JNI and the NDK. I don't figure out why it doesn't work as I expect.

I've read a lot of documentation all over the web, and here is what I have for now:

C++ side:

// Create the texture
uint texture;
glGenTextures(1, &texture);

// Bind the texture with the proper external texture target
glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);

// Create the EGLImage object that maps the GraphicBuffer
int usage = GraphicBuffer::USAGE_HW_TEXTURE | GraphicBuffer::USAGE_SW_READ_OFTEN | GraphicBuffer::USAGE_SW_WRITE_OFTEN;
auto gralloc = new GraphicBuffer(width, height, PIXEL_FORMAT_RGBA_8888, usage);
EGLClientBuffer clientBuffer = (EGLClientBuffer) gralloc->getNativeBuffer();

EGLDisplay display = eglGetDisplay(EGL_DEFAULT_DISPLAY);
EGLint attrs[] = { EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE };

auto eglImage = eglCreateImageKHR(display, EGL_NO_CONTEXT, EGL_NATIVE_BUFFER_ANDROID, clientBuffer, attrs);
glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, eglImage);


In the pixel shader, I use the new type of sampler specific to external textures:

// See https://developer.android.com/reference/android/graphics/SurfaceTexture.html
#extension GL_OES_EGL_image_external : require
precision mediump float;

uniform samplerExternalOES uDiffuseMap;
varying vec2 vVertexUV;

void main(void)
{
gl_FragColor = texture2D(uDiffuseMap, vVertexUV);
}


To check that I can properly sample the texture, I've already did some tests filling the GraphicBuffer with specific data doing like that:

unsigned char *pixels = nullptr;
gralloc->lock(GraphicBuffer::USAGE_SW_READ_OFTEN | GraphicBuffer::USAGE_SW_WRITE_OFTEN, (void **) &pixels);
std::memcpy(pixels, data, width * height * 4);
gralloc->unlock();


And I confirm that it's work as expected, the data written in the GraphicBuffer is the data retrieved when I sample the external texture in the pixel shader.

Now, let's see how it's done Java side to render the WebView in the same texture:

Java side:

CustomRenderer.java (implements GLSurfaceView.Renderer):

@Override
public void onSurfaceChanged(GL10 gl, int width, int height)
{
// glTextureId is given by the C++ through JNI and correspond
// to the ID returns by glGenTextures() call
surfaceTexture = new SurfaceTexture(glTextureId);
surfaceTexture.setDefaultBufferSize(width, height);
surface = new Surface(surfaceTexture);
}

@Override
public void onDrawFrame(GL10 gl)
{
synchronized(this)
{
surfaceTexture.updateTexImage();
}
}

public Canvas onDrawViewBegin()
{
surfaceCanvas = surface.lockCanvas(null);
}

public void onDrawViewEnd()
{
surface.unlockCanvasAndPost(surfaceCanvas);
}


CustomWebView.java (extends WebView)

@Override
public void draw(Canvas canvas) {
if (customRenderer == null)
{
super.draw(canvas);
return;
}

// Returns canvas attached to OpenGL texture to draw on
Canvas glAttachedCanvas = customRenderer.onDrawViewBegin();

if (glAttachedCanvas != null)
{
// Draw the view to provided canvas
super.draw(glAttachedCanvas);
}
else
{
super.draw(canvas);
return;
}

// Notify the canvas is updated
customRenderer.onDrawViewEnd();
}


I willingly removed all error checks to have the simplest code ever.

The result that I get with this code let me think that the texture is not written by the Java side.

Do you know what happens? Did I do something wrong?

Answer Source

I finally found the answer a few days ago. I assume the problem was that the Surface and SurfaceTexture objects were not instanciated (Java side) in the same OpenGL context than the texture creation (C++ side).

I found this project in which SurfaceTexture and Surface objects were created directly from C++ using JNI. Then, instead of passing the texture name, I pass the JNI Surface and use it to retrieve a Canvas and render the WebView into it.

C++ side

auto textureName = 1; // Should be returned by glGenTextures() call
auto textureWidth = 512;
auto textureHeight = 512;

// Retrieve the JNI environment, using SDL, it looks like that
auto env = (JNIEnv*)SDL_AndroidGetJNIEnv();

// Create a SurfaceTexture using JNI
const jclass surfaceTextureClass = env->FindClass("android/graphics/SurfaceTexture");
// Find the constructor that takes an int (texture name)
const jmethodID surfaceTextureConstructor = env->GetMethodID(surfaceTextureClass, "<init>", "(I)V" );
jobject surfaceTextureObject = env->NewObject(surfaceTextureClass, surfaceTextureConstructor, sharedTexture->id());
jobject jniSurfaceTexture = env->NewGlobalRef(surfaceTextureObject);

// To update the SurfaceTexture content
jmethodId updateTexImageMethodId = env->GetMethodID(surfaceTextureClass, "updateTexImage", "()V");
// To update the SurfaceTexture size
jmethodId setDefaultBufferSizeMethodId = env->GetMethodID(surfaceTextureClass, "setDefaultBufferSize", "(II)V" );

// Create a Surface from the SurfaceTexture using JNI
const jclass surfaceClass = env->FindClass("android/view/Surface");
const jmethodID surfaceConstructor = env->GetMethodID(surfaceClass, "<init>", "(Landroid/graphics/SurfaceTexture;)V");
jobject surfaceObject = env->NewObject(surfaceClass, surfaceConstructor, jniSurfaceTexture);
jobject jniSurface = env->NewGlobalRef(surfaceObject);

// Now that we have a globalRef, we can free the localRef
env->DeleteLocalRef(surfaceTextureObject);
env->DeleteLocalRef(surfaceTextureClass);
env->DeleteLocalRef(surfaceObject);
env->DeleteLocalRef(surfaceClass);

// Don't forget to update the size of the SurfaceTexture
env->CallVoidMethod(jniSurfaceTexture, setDefaultBufferSizeMethodId, textureWidth, textureHeight);

// Get the method to pass the Surface object to the WebView
jmethodId setWebViewRendererSurfaceMethod = env->GetMethodID(webViewClass, "setWebViewRendererSurface", "(Landroid/view/Surface;)V");
// Pass the JNI Surface object to the Webview
env->CallVoidMethod(webView, setWebViewRendererSurfaceMethod, jniSurface);

It's important to call the updateTexImage() on the JNI SurfaceTexture object in the same thread to update the OpenGL texture content. Thus, we don't even need to have a GLSurfaceView anymore (that, in my case, was only used to update the texture content regularly):

env->CallVoidMethod(jniSurfaceTexture, updateTexImageMethodId);

Java side, you only need to store the Surface object provided from the C++ code and use it in the overriden draw() function to draw into it.

Java side

public class CustomWebView extends WebView
{
    private Surface _webViewSurface;

    public void setWebViewSurface(Surface webViewSurface)
    {
        _webViewSurface = webViewSurface;
    }

    @Override
    public void draw(Canvas canvas) {
        if (_webViewSurface == null)
        {
            super.draw(canvas);
            return;
        }

        // Returns canvas attached to OpenGL texture to draw on
        Canvas glAttachedCanvas = _webViewSurface.lockCanvas(null);

        if (glAttachedCanvas != null)
        {
            // Draw the view to provided canvas
            super.draw(glAttachedCanvas);
        }
        else
        {
            super.draw(canvas);
            return;
        }

        _webViewSurface.unlockCanvasAndPost(glAttachedCanvas);
    }
}

It's that simple. You don't need to use an EGL image neither (so, no GraphicBuffer anymore!), if you bind the texture with the correct target (GL_TEXTURE_EXTERNAL_OES) and use the correct sampler (samplerExternalOES), it should work as expected.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download