ETC1 textures loading and alpha handling with Android NDK

 Previous parts

• Simple cross-platform game engine - Introduction
• Universal Box2D debug draw for OpenGL ES 1.x and OpenGL ES 2.0
• Loading images under Windows 
• Load images under Android with NDK and JNI
• Using JNI_OnLoad() in Adroid NDK development 
• Simple streaming audio mixer for Android with OpenSL ES - part 1  
• Simple streaming audio mixer for Android with OpenSL ES - part 2
• Crunch - very lightweight unpacker used in our Android and bada games
• Using zlib compression library in Adroid NDK

 In this post I will describe how to work with ETC1 textures.

 What are compressed textures good for?

 You are probably familiar with image formats like .png and .jpg. The first one employs lossless compression while the second uses lossy compression. Usually the .jpg files are smaller for the same image for price of decreased quality.

 Regardless the way the image is stored compressed on the disk, when you load and extract it into memory it bloats to its uncompressed size. For RGBA 8888 it is 32 bits per each one pixel. So when your 1024x1024 image as a texture may eat 4MB of precious video memory

  Beside these well known image formats there are also formats that compress the texture and it stays compressed on GPU. These formats can drastically reduce number of bits required for each pixel.

 Unfortunately, when coding for Android these formats are vendor specific. Currently there are four of them: ETC1, PVRTC, ATITC, S3TC. See more here.

 Only ETC1 is available on all Android devices so the rest of the article is about this format. But it lacks one important thing - it does not support textures with alpha. Fortunately, there are ways how to bypass this. As mentioned before, these format means that texture stays compressed on GPU being decompresssed on the fly which will reduce texture space used as well as increase performance of your openGL game (reduced data bandwidth).

 Another atractiviy of these texture formats comes from fact that you do not need to include any image unpacking library like libpng into your game. Nor you need to load images on the java side like I was describing in my older post (Load images under Android with NDK and JNI).

 The drawback is that textures in its compressed form are bigger than .jpg or .png files with the same image, so more disk space will be consumed. Further in the article I will also describe way how I solved it.

Creating ETC1 texture

 Every GPU vendor has its own set of tools including tool for compressing textures. I am using tool from ARM (load it here). After you run the tool you will get initial screen. Open image file with you texture (do not forget that open GL ES needs height and width to be power of 2) and you should get screen like this:

 Select the texture(s) in left panel and press "Compress" icon (). The compression parameters panel will pop up:

 Choose ETC1/ETC2 tab (1.) and select PKM as output format. PKM is very simple format that ads small header to compressed data. The header is this:

+0: 4 bytes header "PKM "
+4: 2 bytes version "10"
+6: 2 bytes data type (always zero)
+8: 2 bytes extended width
+10: 2 bytes extended height
+12: 2 bytes original width
+14: 2 bytes original height
+16: compressed texture data

 In ETC1 format each 4x4 pixel bloc is compressed into 64 bits. So the extended width and height are the original dimensions rounded up to multiple of four. If you are using power of 2 textures then the original and extended dimensions are the same.

 From these parameters you can calculate the size of compressed data like this:

(extended width / 4) * (extended height / 4) * 8

 This formula just says: there is so many 4x4 pixel blocks and each of them is 8 bytes long (64 bits).

 Parameters marked 2. and 5. on the picture will affect quality of compression. The compression takes quite a lot of time. So during development you can use worse quality if you do not want to wait. But be sure that when finishing your game you use maximum quality - the size of output remains the same.

 Under 3. do not forget to check that ETC1 is chosen and udder 4. choose to create separate texture for alpha channel. This texture will have the same dimensions as the original one. But in the red channel of it there will be stored alpha instead of color. The green and blue channels are unused so theoretically you can put any additional information there (but not with the tool - it would be up to you how to do it).

Loading ETC1 texture

  Now when you have the texture compressed it is time to load it into GPU.

//------------------------------------------------------------------------
u16 TextureETC1::swapBytes(u16 aData)
{
 return ((aData & 0x00FF) << 8) | ((aData & 0xFF00) >> 8);
}

//------------------------------------------------------------------------
void TextureETC1::construct(SBC::System::Collections::ByteBuffer& unpacked)
{
 // check if data is ETC1 PKM file - should start with text "PKM " (notice the space in the end)
 // read byte by byte to prevent endianness problems
 u8 header[4];
 header[0] = (u8) unpacked.getChar();
 header[1] = (u8) unpacked.getChar();
 header[2] = (u8) unpacked.getChar();
 header[3] = (u8) unpacked.getChar();

 if (header[0] != 'P' || header[1] != 'K' || header[2] != 'M' || header[3] != ' ')
  LOGE("data are not in valid PKM format");

 swapBytes is just help method the real work is done in construct method. ByteBuffer is our simple wrapper around array of bytes holding not only data but also its size. This is not important here it just increases readability.

 We start with header to check whether the input data are really PKM file.

 // read version - 2 bytes. Should be "10". Just skipping
 unpacked.getShort();
 // data type - always zero. Just skip
 unpacked.getShort();

 // sizes of texture follows: 4 shorts in big endian order
 u16 extWidth = swapBytes((u16) unpacked.getShort());
 u16 extHeight = swapBytes((u16) unpacked.getShort());
 u16 width = swapBytes((u16) unpacked.getShort());
 u16 height = swapBytes((u16) unpacked.getShort());

 // calculate size of data with formula (extWidth / 4) * (extHeight / 4) * 8
 u32 dataLength = ((extWidth >> 2) * (extHeight >> 2)) << 3;

 In next step we skip additional information on version and data type as we do not use them. Then dimensions are read and with swapByte it is converted from big endian to little endian. The size of compressed data is calculated with already mentioned formula.

  // openGL part
  // create and bind texture - all next texture ops will be related to it
  glGenTextures(1, &mTextureID);
  glBindTexture(GL_TEXTURE_2D, mTextureID);
  // load compressed data (skip 16 bytes of header)
  glCompressedTexImage2D(GL_TEXTURE_2D, 0, GL_ETC1_RGB8_OES,
    extWidth, extHeight, 0, dataLength, unpacked.getPositionPtr());

  // set texture parameters
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

  // save size
  mWidth = extWidth;
  mHeight = extHeight;
}

 Now it is time to create texture with OpenGL calls. We first allocate texture ID and then we sent data for it. Note that internal format is GL_ETC1_RGB8_OES. This says that we are using OES_compressed_ETC1_RGB8_texture extension that ads support for ETC1 textures.

 Now we can use this texture as any other loaded from .png.

Handling alpha with ETC1 textures

 As previously said, ETC1 does not support alpha. During texture creation time we exported alpha into separate texture. This texture has the same dimension as the one with colors. You can employ fragment shader to compose the final color containing alpha from these two textures.

 The simple fragment shader doing this may look like this:

#ifdef GL_ES
precision mediump float;
#endif

uniform lowp sampler2D u_map[2];
varying mediump vec2 v_texture;

void main(void)
{
  gl_FragColor = vec4(texture2D(u_map[0], v_texture).rgb, texture2D(u_map[1], v_texture).r);
}

 It took me less than 1 hour to rewrite parts of our game to support ETC1 with alpha in separate texture instead of previously used textures from .png files. And I got really sweet reward in increasing the frame rate by about 10% on my Samsung Galaxy Tab.

Compression to the power of two

 The last thing that I did not liked first was that ETC1 textures without alpha were 2 times bigger than .jpg files (in our prepared game Shards we are using .jpg for backgrounds - so no need for alpha for these). I finally solved this with additional compression of .pkm file with zlib library. It took me some time to find how to use it on Android, but you can read it here (Using zlib compression library in Adroid NDK) - it is really worth of it.

 Now I am almost on the size of .jpg file. When creating texture I first uncompress the file with zlib and the uncompressed .pkm is sent to previously described routine.

Conclusion

  ETC1 texture compression is the only one supported by all Android devices (having OpenGL ES 2.0 of course). It lacks alpha so small overhead is present when bypassing this. It pays as you are saving texture memory and increasing your frame rate - 10% in case of my Samsung Galaxy Tab.

Shards - Work In Progress I

This is short info on our upcoming game - Shards - we are working on. The game will be clone of well known arcade classic with glassy look and very nice fractal backgrounds. The music in video is just a placeholder I found on the web - it sounds to make me sure the sound engine works.

 Making the game we are using these technologies:

• box2D for physics,
• notice the usage of our universal box2D debug draw that works for Open GL ES 1.x and 2.0,
• our cross platform engine -  the video is taken from desktop Windows while the game runs on Android as well,
• PicOpt - our sprite atlas creation tool that is free to download,
• for Android NDK: textures are loaded through this way, while sound (not too much of them in the game presently) is coded like this,
• for ads we plan to utilize our simple ads manager.

Load images under Android with NDK and JNI

Previous parts

• Simple cross-platform game engine - Introduction
• Universal Box2D debug draw for OpenGL ES 1.x and OpenGL ES 2.0
• Loading images under Windows

  This snippet will focus on loading images on Android NDK using JNI. Our cross-platform mobile engine runs on Android, bada and desktop Windows. In last part I described two ways how to load images on Windows platform. Similar to that I again wanted solution that will load .png image as well as .jpg image.

 The solution that works good for me is split into C and Java part. The actual image loading and texture creating takes part in Java part. It also works either in fully native app with its C main loop and using native_app_glue or in Java app with GLSurfaceView.Renderer calling native methods in onDrawFrame method.

 The java part not only loads image but also creates openGL texture for me. If you are interested only in loading the image you will have to adjust it a little for your needs.

Native part

  The initial part of the method is split depending on whether you are using native_app_glue or not. If yes then first call to getApplication is actually returning android_app* type. This structure is part of parameter list of native main method. In my engine it is typedefed to SBC::System::Application (typedef struct android_app Application;). If you are not writing fully native app then you will have to use some variables previously cached (gJavaVM, gJavaActivityClass). I am caching it in JNI_OnLoad which is called when native library is loaded (see another article on JNI_OnLoad).

void Texture::construct(u8* aFileName, u32 aIdx)
{
 JNIEnv *env;

#ifdef USE_NATIVE_APP_GLUE
 SBC::System::Application* app = &Game::getGame().getApplication();

 JavaVM* vm = app->activity->vm;
 vm->AttachCurrentThread (&env, NULL);
 jclass activityClass = env->GetObjectClass(app->activity->clazz);
#else
 bool shouldDetach = false;
 JavaVM* vm = gJavaVM;
 jint rc = vm->GetEnv((void **)&env, JNI_VERSION_1_6);
 if (rc != JNI_OK)
 {
  shouldDetach = true;
  vm->AttachCurrentThread(&env, NULL);
 }
 jclass& activityClass = gJavaActivityClass;
#endif

  Then main part of the method follows:

 jmethodID mid = env->GetStaticMethodID(activityClass, "loadTexture", "(Ljava/lang/String;I)I");
 jstring mystr = env->NewStringUTF(aFileName);
 jint ret = env->CallStaticIntMethod(activityClass, mid, mystr, aIdx);
 env->DeleteLocalRef(mystr);

 // store information on ID, width and height of texture
 mTextureID = ret;

 mid = env->GetStaticMethodID(activityClass, "getTextureWidth", "()I");
 mWidth = env->CallStaticIntMethod(activityClass, mid);
 mid = env->GetStaticMethodID(activityClass, "getTextureHeight", "()I");
 mHeight = env->CallStaticIntMethod(activityClass, mid);

 mTextureID, mWidth and mHeight are member variables of the Texture class. The ID is OpenGL ID of the texture. So I am simply calling java methods that will do all the work. After that I just "clean" with detaching if necessary:

#ifdef USE_NATIVE_APP_GLUE
 vm->DetachCurrentThread();
#else
 if (shouldDetach)
  vm->DetachCurrentThread();
#endif

 LOGI("texture ID %i, width %i, height %i", mTextureID, mWidth, mHeight);
} 

Java part

 The main things are going on in java class. I call it Tools as the image/texture loading is not its only purpose. But only this is subject of this article. In the beginning I just open file fname for reading. The second parameter beyond the file name (int id) is used when using compound files (single file made from multiple other files with some offsets header in the beginning). If using single file then this parameter contains -1.

package com.sbcgames.sbcengine;

import java.io.DataInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.util.Locale;

import android.content.res.AssetManager;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Canvas;
import android.opengl.GLES20;
import android.opengl.GLUtils;
import android.util.Log;

/*
 * Help methods for doing things I cannot do from my native code now...
 * Hope, this class will get smaller and smaller when it finally disappears...
 */

public class Tools
{
 //-----------------------------------------------------
 // BITMAP
 //-----------------------------------------------------
 
 // last loaded texture parameters
 static int txtID, width, height;
 
 //-----------------------------------------------------
 /* fname = asset file name
  * id >= 0 ... position in compound file
  * id < 0 ... single file (no compound)
  */
 public static int loadTexture(String fname, int id)
 {
  // clear last texture parameters
  txtID = width = height = -1;
  
  Log.d("Helper", "Loading texture from asset file " + fname + " with id " + id);

  final BitmapFactory.Options options = new BitmapFactory.Options();
  options.inScaled = false;    // No pre-scaling
  AssetManager am = SBCEngine.getSBCEngine().getAssets();
  Bitmap bitmap = null;

  try
  {
   InputStream stream = am.open(fname);

 After the file is opened I can start loading the image. Again there is split depending on whether the file is compound or not.

   // loading from compound file?
   if (id >= 0)
   {
    DataInputStream input = new DataInputStream(stream);
    
    // skip header
    input.skip(3);
    // skip to entry offset
    input.skip(id * 4);
    // read entry beginning
    int dataStart = input.readInt();
    // read data length
    int dataLen = input.readInt() - dataStart;
    // skip to start of subfile
    // offsets are without header (3) bytes
    // we already skipped id * 4 bytes
    // we already have read 2 offset by 4 bytes = 8 in total
    input.skip(dataStart - (id * 4) - 8);

    // get data from correct position
    byte[] data = new byte[dataLen];
    input.read(data);

    bitmap = BitmapFactory.decodeByteArray(data, 0, data.length, options);
   }
   else // no compound
   {
    Log.d("Helper", "Loading from stream");
    bitmap = BitmapFactory.decodeStream(stream, null, options);
   }

 Here we can test whether we are succesful and have bitmap loaded

   // test returned bitmap for success
   if (bitmap == null)
   {
    Log.e("Helper", "Failed to load texture " + fname + " with id " + id);
   }

 If yes, we can continue to creating OpenGL texture from it. First we also check whether the image height and width is power of 2. If not we put message into log and then we convert it to nearest power of two image.

   // check whether the loaded bitmap has width and height equal to power of 2
   int w = bitmap.getWidth();
   int h = bitmap.getHeight();
   if (getNearestPOT(w) != w || getNearestPOT(h) != h)
   {
    Log.w("Helper", "Texture " + fname + " with id " + id +
      " has not either width or height power of 2");
    
    // new dimensions
    w = getNearestPOT(w);
    h = getNearestPOT(h);
    
    // redraw bitmap into POT bitmap
    Bitmap newBitmap = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);
    Canvas canvas = new Canvas(newBitmap);
    canvas.drawBitmap(bitmap, 0.0f, 0.0f, null);
    bitmap.recycle();
    canvas = null;
    bitmap = newBitmap;
    
    Log.w("Helper", "Texture " + fname + " rebuilded into texture with POT");
   }

 From the bitmap (possibly rebuilt into new power of two image) we create OpenGL texture like this:

   // generate textureID
   int[] textures = new int[1];
   GLES20.glGenTextures(1, textures, 0);
   int textureID = textures[0];
   
   // create texture
   GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureID);
   GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
   GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
   GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
   GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
   GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap, 0);

 And finally we do some clean up:

   // destroy bitmap
   bitmap.recycle();
   
   txtID = textureID;
   width = w;
   height = h;
   
   Log.d("Helper", "Loaded texture ID:" + textureID + ", width:" + w + ", height:" + h);
   return textureID;
  }
  catch (IOException e)
  {
   Log.e("Helper", "Failed to load texture " + fname + " with id " + id);
   return 0;
  }
 }

 The other Java methods called from C or from within the Java loadTexture method are these:

        //------------------------------------------------------------------------
 public static int getTextureWidth()
 {
  return width;
 }
 
 //------------------------------------------------------------------------
 public static int getTextureHeight()
 {
  return height;
 }
 
 //------------------------------------------------------------------------
 private static int getNearestPOT(int val)
 {
  int newDim = 1;
  while(val > newDim) newDim *= 2;
  return newDim;
 }
 

 It is probably pretty stupid to call three methods to get texture first and then to get its width and height. Better way would be to return some kind of object with three members. But it works and as I will get deeper into NDK/JNI I will adjust it in future.

 I hope this article helped you to write your own image loading class. The final class can load OpenGL textures from .png, .jpg files in your asset directory and it can also adjust images that are not power of two to make OpenGL happy.