Wednesday, June 26, 2013

ETC1 textures loading and alpha handling with Android NDK



 In this post I will describe how to work with ETC1 textures.

 What are compressed textures good for?

 You are probably familiar with image formats like .png and .jpg. The first one employs lossless compression while the second uses lossy compression. Usually the .jpg files are smaller for the same image for price of decreased quality.
 Regardless the way the image is stored compressed on the disk, when you load and extract it into memory it bloats to its uncompressed size. For RGBA 8888 it is 32 bits per each one pixel. So when your 1024x1024 image as a texture may eat 4MB of precious video memory
  Beside these well known image formats there are also formats that compress the texture and it stays compressed on GPU. These formats can drastically reduce number of bits required for each pixel.
 Unfortunately, when coding for Android these formats are vendor specific. Currently there are four of them: ETC1, PVRTC, ATITC, S3TC. See more here.

 Only ETC1 is available on all Android devices so the rest of the article is about this format. But it lacks one important thing - it does not support textures with alpha. Fortunately, there are ways how to bypass this. As mentioned before, these format means that texture stays compressed on GPU being decompresssed on the fly which will reduce texture space used as well as increase performance of your openGL game (reduced data bandwidth).

 Another atractiviy of these texture formats comes from fact that you do not need to include any image unpacking library like libpng into your game. Nor you need to load images on the java side like I was describing in my older post (Load images under Android with NDK and JNI).

 The drawback is that textures in its compressed form are bigger than .jpg or .png files with the same image, so more disk space will be consumed. Further in the article I will also describe way how I solved it.


Creating ETC1 texture

 Every GPU vendor has its own set of tools including tool for compressing textures. I am using tool from ARM (load it here). After you run the tool you will get initial screen. Open image file with you texture (do not forget that open GL ES needs height and width to be power of 2) and you should get screen like this:

 Select the texture(s) in left panel and press "Compress" icon (). The compression parameters panel will pop up:


 Choose ETC1/ETC2 tab (1.) and select PKM as output format. PKM is very simple format that ads small header to compressed data. The header is this:

+0: 4 bytes header "PKM "
+4: 2 bytes version "10"
+6: 2 bytes data type (always zero)
+8: 2 bytes extended width
+10: 2 bytes extended height
+12: 2 bytes original width
+14: 2 bytes original height
+16: compressed texture data

 In ETC1 format each 4x4 pixel bloc is compressed into 64 bits. So the extended width and height are the original dimensions rounded up to multiple of four. If you are using power of 2 textures then the original and extended dimensions are the same.

 From these parameters you can calculate the size of compressed data like this:

(extended width / 4) * (extended height / 4) * 8

 This formula just says: there is so many 4x4 pixel blocks and each of them is 8 bytes long (64 bits).

 Parameters marked 2. and 5. on the picture will affect quality of compression. The compression takes quite a lot of time. So during development you can use worse quality if you do not want to wait. But be sure that when finishing your game you use maximum quality - the size of output remains the same.

 Under 3. do not forget to check that ETC1 is chosen and udder 4. choose to create separate texture for alpha channel. This texture will have the same dimensions as the original one. But in the red channel of it there will be stored alpha instead of color. The green and blue channels are unused so theoretically you can put any additional information there (but not with the tool - it would be up to you how to do it).


Loading ETC1 texture

  Now when you have the texture compressed it is time to load it into GPU.


//------------------------------------------------------------------------
u16 TextureETC1::swapBytes(u16 aData)
{
 return ((aData & 0x00FF) << 8) | ((aData & 0xFF00) >> 8);
}

//------------------------------------------------------------------------
void TextureETC1::construct(SBC::System::Collections::ByteBuffer& unpacked)
{
 // check if data is ETC1 PKM file - should start with text "PKM " (notice the space in the end)
 // read byte by byte to prevent endianness problems
 u8 header[4];
 header[0] = (u8) unpacked.getChar();
 header[1] = (u8) unpacked.getChar();
 header[2] = (u8) unpacked.getChar();
 header[3] = (u8) unpacked.getChar();

 if (header[0] != 'P' || header[1] != 'K' || header[2] != 'M' || header[3] != ' ')
  LOGE("data are not in valid PKM format");

 swapBytes is just help method the real work is done in construct method. ByteBuffer is our simple wrapper around array of bytes holding not only data but also its size. This is not important here it just increases readability.
 We start with header to check whether the input data are really PKM file.


 // read version - 2 bytes. Should be "10". Just skipping
 unpacked.getShort();
 // data type - always zero. Just skip
 unpacked.getShort();

 // sizes of texture follows: 4 shorts in big endian order
 u16 extWidth = swapBytes((u16) unpacked.getShort());
 u16 extHeight = swapBytes((u16) unpacked.getShort());
 u16 width = swapBytes((u16) unpacked.getShort());
 u16 height = swapBytes((u16) unpacked.getShort());

 // calculate size of data with formula (extWidth / 4) * (extHeight / 4) * 8
 u32 dataLength = ((extWidth >> 2) * (extHeight >> 2)) << 3;

 In next step we skip additional information on version and data type as we do not use them. Then dimensions are read and with swapByte it is converted from big endian to little endian. The size of compressed data is calculated with already mentioned formula.


  // openGL part
  // create and bind texture - all next texture ops will be related to it
  glGenTextures(1, &mTextureID);
  glBindTexture(GL_TEXTURE_2D, mTextureID);
  // load compressed data (skip 16 bytes of header)
  glCompressedTexImage2D(GL_TEXTURE_2D, 0, GL_ETC1_RGB8_OES,
    extWidth, extHeight, 0, dataLength, unpacked.getPositionPtr());

  // set texture parameters
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

  // save size
  mWidth = extWidth;
  mHeight = extHeight;
}

 Now it is time to create texture with OpenGL calls. We first allocate texture ID and then we sent data for it. Note that internal format is GL_ETC1_RGB8_OES. This says that we are using OES_compressed_ETC1_RGB8_texture extension that ads support for ETC1 textures.

 Now we can use this texture as any other loaded from .png.


Handling alpha with ETC1 textures

 As previously said, ETC1 does not support alpha. During texture creation time we exported alpha into separate texture. This texture has the same dimension as the one with colors. You can employ fragment shader to compose the final color containing alpha from these two textures.
 The simple fragment shader doing this may look like this:


#ifdef GL_ES
precision mediump float;
#endif

uniform lowp sampler2D u_map[2];
varying mediump vec2 v_texture;

void main(void)
{
  gl_FragColor = vec4(texture2D(u_map[0], v_texture).rgb, texture2D(u_map[1], v_texture).r);
}

 It took me less than 1 hour to rewrite parts of our game to support ETC1 with alpha in separate texture instead of previously used textures from .png files. And I got really sweet reward in increasing the frame rate by about 10% on my Samsung Galaxy Tab.



Compression to the power of two

 The last thing that I did not liked first was that ETC1 textures without alpha were 2 times bigger than .jpg files (in our prepared game Shards we are using .jpg for backgrounds - so no need for alpha for these). I finally solved this with additional compression of .pkm file with zlib library. It took me some time to find how to use it on Android, but you can read it here (Using zlib compression library in Adroid NDK) - it is really worth of it.
 Now I am almost on the size of .jpg file. When creating texture I first uncompress the file with zlib and the uncompressed .pkm is sent to previously described routine.


Conclusion

  ETC1 texture compression is the only one supported by all Android devices (having OpenGL ES 2.0 of course). It lacks alpha so small overhead is present when bypassing this. It pays as you are saving texture memory and increasing your frame rate - 10% in case of my Samsung Galaxy Tab.




13 comments:

  1. Hi,

    I am trying to use this shader but I get errors

    10-05 10:54:56.118: D/cocos2d-x debug info(29246): L0007: Cannot find vertex shader varying for declared fragment shader varyingv_texture

    10-05 11:49:30.638: D/cocos2d-x debug info(30657): OpenGL error 0x0501 in cocos2dx/textures/CCTextureAtlas.cpp mapBuffers 306

    I am using cocos2dx
    can u detail on how can i use this shader

    ReplyDelete
  2. Hi!, I have no experience with cocos2D, but you are probably missing v_texture definition in your vertex shader. The simple one can be like this:


    uniform mediump mat4 u_projection;

    attribute mediump vec2 a_vertex;
    attribute mediump vec2 a_texture;

    varying mediump vec2 v_texture;

    void main(void)
    {
    mediump vec4 position = vec4(a_vertex, 0.0, 1.0);
    gl_Position = u_projection * position;

    v_texture = a_texture;
    }


    Notice, there are:
    1) one uniform for projection matrix,
    2) 2 attributes (a_vertex, a_texture) you are sending to GPU for every vertex,
    3) one varying v_texture (this one is missing in your vertex shader according to error). Varyings are there to provide communication between vertex and fragment shader. Here it is very simple: I just pass texture U,V attribute for vertex into fragment shader (v_texture = a_texture)

    ReplyDelete
  3. Hey

    Thanks, it works now with no error. But I cant see the image at all. This is how I have loaded the shaders cc_MonoChromeV vertex , cc_MonoChrome fragment

    CCTextureCache::sharedTextureCache()->addETCImage("SpinWheel-hd.pkm");

    MainScreen1 = CCSprite::create("SpinWheel-hd.pkm");
    MainScreen1->setScale(0.3); //DEVSCALE4(1);
    CCSize size1 = CCDirector::sharedDirector()->getWinSize();
    MainScreen1->setPosition(ccp(size1.width/2, size1.height/2));
    this->addChild(MainScreen1);

    CCGLProgram *shader = CCShaderCache::sharedShaderCache()->programForKey("monochromeShader");
    shader->reset();
    /// Same code as before to initialize the shader
    shader->initWithVertexShaderByteArray(cc_MonoChromeV, cc_MonoChrome);
    MainScreen1->setShaderProgram(shader);

    MainScreen1->getShaderProgram()->addAttribute(kCCAttributeNamePosition, kCCVertexAttrib_Position);
    MainScreen1->getShaderProgram()->addAttribute(kCCAttributeNameTexCoord, kCCVertexAttrib_TexCoords);
    MainScreen1->getShaderProgram()->link();
    MainScreen1->getShaderProgram()->updateUniforms();

    _maskLocation = glGetUniformLocation( MainScreen1->getShaderProgram()->getProgram(),"u_map[1]");
    glUniform1i(_maskLocation, 1);

    alphaTex = CCTextureCache::sharedTextureCache()->addETCImage("SpinWheel1-hd.pkm");
    alphaTex->setAliasTexParameters();

    MainScreen1->getShaderProgram()->use();

    glActiveTexture(GL_TEXTURE1);

    glBindTexture(GL_TEXTURE_2D,alphaTex->getName());

    glActiveTexture(GL_TEXTURE0);

    What I am be missing? I am alos following this guide:
    http://www.raywenderlich.com/10862/how-to-create-cool-effects-with-custom-shaders-in-opengl-es-2-0-and-cocos2d-2-x
    http://www.raywenderlich.com/4428/how-to-mask-a-sprite-with-cocos2d-2-0

    ReplyDelete
    Replies
    1. I have no experience with cocos2D. But ... in the code above you have some initialization - no real drawing. You just say: this is my texture and this is my shader, but "and now draw my sprite" is not there. You must have somewhere in your code also set of commands like glVertexAttribPointer, glDrawElements, ... These commands are called on every frame and this is what exactly says what are vertices and texture coordinates

      In article you mentioned it is in the "-(void)blit" method.

      The best for you would be to download example from that article. Run it and then step by step replace or play with shaders and textures (including replacement of single-texturing with multi-texturing, because keeping alpha in separate texture results in multitexturing)

      Also in your code above, there are som strange settings like: "_maskLocation = glGetUniformLocation( MainScreen1->getShaderProgram()->getProgram(),"u_map[1]");
      glUniform1i(_maskLocation, 1);"
      I am not sure, if this will work and if yes I do not see anywhere setting of u_map[0]...

      In my code I am setting both textures at once like this:

      glActiveTexture(GL_TEXTURE_0);
      glBindTexture(GL_TEXTURE_2D, myRGBTexture);
      glActiveTexture(GL_TEXTURE_1);
      glBindTexture(GL_TEXTURE_2D, myAlphaTexture);

      GLint textureUnits[2] = {0, 1};
      glUniform1iv(location_OfMapUniform, 2, &textureUnits);

      Delete
    2. Hi,

      In your code : what is location_OfMapUniform?

      also, the drawing code is in the draw of cocos2dx , if you look at first article, there is no custom drawing, on every frame cocos2dx calls the usual draw method.

      in the shader i have u_map[2], so I was trying to load rgb in u_map[0] and alpha in u_map[1]

      rgbTex = CCTextureCache::sharedTextureCache()->addETCImage("SpinWheel-hd.pkm");

      MainScreen1 = CCSprite::create("SpinWheel-hd.pkm");

      MainScreen1->setScale(0.3); //DEVSCALE4(1);

      CHECK_GL_ERROR_DEBUG();
      CCSize size1 = CCDirector::sharedDirector()->getWinSize();
      MainScreen1->setPosition(ccp(size1.width/2, size1.height/2));
      this->addChild(MainScreen1);

      CCGLProgram *shader = CCShaderCache::sharedShaderCache()->programForKey("monochromeShader");
      shader->reset();
      /// Same code as before to initialize the shader
      shader->initWithVertexShaderByteArray(cc_MonoChromeV, cc_MonoChrome);
      CHECK_GL_ERROR_DEBUG();

      MainScreen1->setShaderProgram(shader);
      CHECK_GL_ERROR_DEBUG();
      MainScreen1->getShaderProgram()->addAttribute(kCCAttributeNamePosition, kCCVertexAttrib_Position);
      MainScreen1->getShaderProgram()->addAttribute(kCCAttributeNameTexCoord, kCCVertexAttrib_TexCoords);
      CHECK_GL_ERROR_DEBUG();
      MainScreen1->getShaderProgram()->link();
      CHECK_GL_ERROR_DEBUG();
      MainScreen1->getShaderProgram()->updateUniforms();
      CHECK_GL_ERROR_DEBUG();

      _maskLocation = glGetUniformLocation( MainScreen1->getShaderProgram()->getProgram(),"u_map[1]");
      glUniform1i(_maskLocation, 1);

      _textureLocation = glGetUniformLocation( MainScreen1->getShaderProgram()->getProgram(),"u_map[0]");
      glUniform1i(_textureLocation, 0);


      alphaTex = CCTextureCache::sharedTextureCache()->addETCImage("SpinWheel1-hd.pkm");
      alphaTex->setAliasTexParameters();

      MainScreen1->getShaderProgram()->use();

      glActiveTexture(GL_TEXTURE0);
      glBindTexture(GL_TEXTURE_2D, rgbTex->getName());
      glActiveTexture(GL_TEXTURE1);
      glBindTexture(GL_TEXTURE_2D, alphaTex->getName());

      In the article 1 above, there is not custom drawing, check colorramp shader

      Delete
    3. Hi,
      - location_OfMapUniform is just uniform location. In your code you are calling glGetUniformLocation(....) to retrieve it. I just did not want to increase complexity of my example,
      - loading into u_map[0] and then u_map[1] is probably OK (I never used it like this before - I always loaded both values at once),
      - I do not know cocos2D, but it must somewhere inside send vertices to GPU on every frame. It is "-(void)blit" method in the article. But it seems that Cocos is doing it for you? Again, I have no experience with Cocos,
      - is the example in article without changes working for you? If yes, try to adjust it step by step like this:
      -- display non-alfa texture in middle of screen (actually no changes at all),
      -- try the same with alfa texture (just change texture name in loading - as alpha is in red channel you should see some red image),
      -- change code to load both textures and play for while with attaching texture unit 0 or 1 to shader. Both should work,
      -- try to adjust your fragment shader to take both textures but do not change drawing code. Again, play with it for a while,
      -- adjust shader to use both textures...

      You can also play with fragment shader like this:
      replace gl_FragColor = vec4(texture2D(u_map[0], v_texture).rgb, texture2D(u_map[1], v_texture).r);
      with gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0); and you should see red square,
      or gl_FragColor = vec4(texture2D(u_map[1], v_texture).r, 0.0, 0.0, 1.0) and you should see only alpha texture drawn in red color - this is how you can check if alpha data were passed as second texture.

      But again, I have no experience with cocos... so I am running out of ideas :-(

      Delete
  4. display non-alfa texture in middle of screen (actually no changes at all), works

    try the same with alfa texture i load the aplha texture normally as an etc image (my aplha is a pkm file) but i see a black and white image and not red?

    when i do this
    gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);

    i see red square

    when i do this
    gl_FragColor = vec4(texture2D(u_map[1], v_texture).r, 0.0, 0.0, 1.0)

    i see nothing

    Any clues with this info?

    ReplyDelete
  5. - black & white seems to be OK. I checked generated alpha PKM and it looks like the tool is placing alpha channel into all three channels not only into red one (so in shader you can take alpha from r, g or b channel),

    - from tests it looks like there is something wrong in attaching the second texture. Maybe try to change order of your commands: 1) load all textures (also the one with alpha) on the top (together with non-alpha one) 2) use shader before setting its uniforms,

    - try http://discuss.cocos2d-x.org forum focused on cocos. I beleive they will have some working sample - it is forum with high traffic.

    ReplyDelete
  6. rgbTex = CCTextureCache::sharedTextureCache()->addETCImage("SpinWheel-hd.pkm");
    MainScreen1 = CCSprite::create("SpinWheel-hd.pkm");
    CCSize size1 = CCDirector::sharedDirector()->getWinSize();
    MainScreen1->setPosition(ccp(size1.width/2+70, size1.height/2+70));
    this->addChild(MainScreen1);
    alphaTex = CCTextureCache::sharedTextureCache()->addETCImage("SpinWheel1-hd.pkm");

    alphaTex->setAliasTexParameters();

    CCGLProgram *shader = CCShaderCache::sharedShaderCache()->programForKey("monochromeShader");
    shader->reset();
    /// Same code as before to initialize the shader
    shader->initWithVertexShaderByteArray(cc_MonoChromeV, cc_MonoChrome);

    MainScreen1->setShaderProgram(shader);

    MainScreen1->getShaderProgram()->addAttribute(kCCAttributeNamePosition, kCCVertexAttrib_Position);
    MainScreen1->getShaderProgram()->addAttribute(kCCAttributeNameTexCoord, kCCVertexAttrib_TexCoords);

    MainScreen1->getShaderProgram()->link();

    MainScreen1->getShaderProgram()->updateUniforms();


    _maskLocation = glGetUniformLocation( MainScreen1->getShaderProgram()->getProgram(),"u_colorRampTexture");
    glUniform1i(_maskLocation,1);

    _textureLocation = glGetUniformLocation( MainScreen1->getShaderProgram()->getProgram(),"u_texture");
    glUniform1i(_textureLocation, 0);


    MainScreen1->getShaderProgram()->use();


    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, rgbTex->getName());
    glActiveTexture(GL_TEXTURE1);
    glBindTexture(GL_TEXTURE_2D,alphaTex->getName());
    glActiveTexture(GL_TEXTURE0);

    Fragment Shader:

    "\n\
    #ifdef GL_ES \n\
    precision mediump float;\n\
    #endif\n\
    uniform sampler2D u_texture;\n\
    uniform sampler2D u_colorRampTexture;\n\
    varying mediump vec2 v_texture;\n\
    void main(void)\n\
    {\n\
    gl_FragColor = vec4(texture2D(u_texture, v_texture).rgb, texture2D(u_colorRampTexture, v_texture).r);\n\
    }\n\
    ";

    Vertical Shader:

    "\n\
    attribute vec4 a_position; \n\
    attribute mediump vec2 a_texture;\n\
    \n\
    #ifdef GL_ES\n\
    varying mediump vec2 v_texture;\n\
    #else \n\
    varying vec2 v_texture; \n\
    #endif \n\
    \n\
    void main(void)\n\
    {\n\
    gl_Position = CC_MVPMatrix * a_position;\n\
    v_texture = a_texture;\n\
    }\n\
    ";

    ReplyDelete
  7. With this code, I see a white image on screen. I had to alter the vertex shader. Please check now.
    Also I posted on cocos2d forum, no response yet.

    ReplyDelete
  8. - in my opinion you should use shader before working with it uniforms. There is pretty big chance that your shader is put in use after link, but to be sure alway use it before setting uniforms,
    - what does white image mean? Is it white sqaure or does it have proper shape you are expocting it to have?,
    - play with fragment shader:
    -- try: gl_FragColor = vec4(texture2D(u_texture, v_texture).rgb, 1.0); ... and you should see image without alpha (to check your texture data are ok),
    -- try: gl_FragColor = vec4(texture2D(u_colorRampTexture, v_texture).r, 0.0, 0.0 1.0); ... and you should see your alpha mask in red (again to check if texture data are available for shader);

    ... and just noticed, that you are using CC_MVPMatrix in your vertex shader, but there is no definition of this uniform! This is vertex shader that works for me. Notice, there is u_projection uniform. OpenGL has view with coordinates -1 ... 1 and your projection matrix has to map your screen coordinates into this range. I also do not see you setting this uniform in you code, but as I already mentioned: I have no experience with cocos2D. Maybe cocos is handling projection for you. But what is surely bad is missing its definition in vertex shader.

    uniform mediump mat4 u_projection;

    attribute mediump vec2 a_vertex;
    attribute mediump vec2 a_texture;

    varying mediump vec2 v_texture;

    void main(void)
    {
    mediump vec4 position = vec4(a_vertex, 0.0, 1.0);
    gl_Position = u_projection * position;

    v_texture = a_texture;
    }

    Are you working with Android or iOS?

    ReplyDelete
  9. I am working on Android.

    It finally works for me, Just a few tweaks in the naming of textures in the shaders acc to cocos2dx.

    The only trouble is tht i get rough edges after i use the shader. As i see Mali tool is also decreasing quality of the texture. Is there any other tool I can use which does less lossy compression?

    ReplyDelete
    Replies
    1. Happy to hear it wokrs :-)

      For "production" quality use "Compression mode" : slow, "Error mertic" : perceptual in tool's settings. The loss of quality will still be noticable but is not issue in my opinion (see our games Fruit Dating: https://play.google.com/store/apps/details?id=com.sbcgames.fruitdatingl or Shards: https://play.google.com/store/apps/details?id=com.sbcgames.shardsl - both are using it)

      The quality is not tool related. It is technology related. All texture compressions are lossy, but your GPU will love it.

      The real benefit is speed. If speed is not issue for you, you can use PNGs or JPEGs. JPEGs eat less space on disk while having also artifactcs and missing alpha while PNGs give best quality. For GPU texture compression like ETC is best as it sqeezes 16 pixels into 64 bits. Good is that ETC1 is on all Android (OpenGL ES 2.0) devices.

      Delete