…code, cg, electronics, 3d printing…

Go to part 5…, part 7…

Processing makes it really easy to “Present” a sketch to your Android phone via it’s “Present” button.  This creates a special ‘debug’ apk that can run on your specific device, but isn’t distributable (and I can’t even find it on the phone) to others.    There is no easy way (yet) to get these sketches to other users via the apk format.  But thanks to the great user community (and one specific post in question) I was able to get a signed apk built and installed on my phone.  Follow-up post will expose this apk to the world 😉

How to do it?

First Export the sketch.  Then Sign the Sketch.  Sounds easy.  The first part is… 😉  Below is the process I took to get my “floater08″ sketch “apk’ed”, so it’s what I’ll use as an example:

Part 1:  Export the sketch

• From the Processing PDE, open the sketch (go into ‘Android -> Android Mode’), and press the “Export” button (arrow facing to right).  This will export the android data and put it in an \export folder under the sketch:
• C:\sketchbook\floater08\android

That’s the easy part.

Part 2:  Signing the sketch

Docs:

The Android-Processing wiki (under the ‘Distributing Apps’ section) links to these Android Docs for signing an APK.  But I followed the steps on this incredibly helpful post (the ones by user ‘ckamath‘ specifically) to get me going.  Below is an overview of my process working with this information.

Install Tools:

Like the post says, make sure you can run these commands from a shell: keytool, jarsigner, zipalign, ant.  I didn’t have ant, so I installed it from here, following these steps (specifically everything it says to do under “The Short Story” & “Setup”).  jarsigner and keytool are provided via the JDK which I’d already installed.

Like the post says, I also (apparently) needed cygwin which provides OpenSSL.  After downloading and running the setup.exe cygwin installer, I figured out I needed to filter the install list for ‘openssl’, and then I needed to select each entry for install.  It was all a bit clunky and cryptic, and the installer seemed to crash a couple times.  But it did install (with a bunch of other stuff too apparently).

Run Code:

These are my steps, which more or less follow ckamath’s tutorial in the above post:

• Open a shell.
• Browse to the \android folder where your export data is.  In my case that was:
  • C:\sketchbook\floater08\android
  • In the below examples, this is the presumed path from the shell that the code is being entered from.

• A: Make your secret key!
• Create the ‘secret key’ that is required for signing your application:
  • Here is a template of what to type:
    • \android>keytool -genkey -v -keystore <SKETCHNAME>-release-key.keystore -alias <YOURNAME> -keyalg RSA -keysize 2048 -validity 10000
  • This is what I typed:
    • \android>keytool -genkey -v -keystore floater08-release-key.keystore -alias AKEric -keyalg RSA -keysize 2048 -validity 10000
• It will then ask you several question like “What is your first and last name?”, What is the name of your City or Locality?”, etc.
• When its done (successfully) it will finish by printing:
  • [Storing floater08-release-key.keystore]
• And will store the corresponding file:  \android\floater08-release-key.keystore

• B: Create ‘unsigned .apk’
• Execute ant to build the unsiged apk:
  • \android>ant release
• This should spit out a bunch of stuff in the shell.  When complete you should see something like:
  • BUILD SUCCESSFUL
    Total time: 18 seconds
• If you didn’t have one before, you should now have a \bin folder with the unsigned apk in it:
  • C:\sketchbook\floater08\android\bin\floater08-unsigned.apk

• C: Sign the unsigned .apk with the secret key:
• Here is a template of what to type:
  • \android>jarsigner -verbose -keystore <SKETCHNAME>-release-key.keystore <FULL PATH TO>\android\bin\<SKETCHNAME>-unsigned.apk <YOUR NAME FROM SECRET KEY STEP>
• This is what I typed:
  • \android>jarsigner -verbose -keystore floater08-release-key.keystore C:\sketchbook\floater08\android\bin\floater08-unsigned.apk akeric
• The shell should have several lines of “adding” and “signing”.

• D: Verify that jarsigner did its thing:
• Here is a template of what to type:
  • \android>jarsigner -verify <FULL PATH TO>\android\bin\<SKETCHNAME>-unsigned.apk
• This is what I typed:
  • \android>jarsigner -verify C:\sketchbook\floater08\android\bin\floater08-unsigned.apk
• If this works you should see in the shell:
  • jar verified.

• E: Make signed (and distributable) apk file:
• Here is a template of what to type:
  • \android>zipalign -v 4 <FULL PATH TO>\android\bin\<SKETCHNAME>-unsigned.apk   <SKETCHNAME>.apk
• This is what I typed:
  • \android>zipalign -v 4 C:\sketchbook\floater08\android\bin\floater08-unsigned.apk   floater08.apk
• The shell will start by saying:
  • Verifying alignment of floater08.apk (4)…
• Print a bunch more stuff, and finally:
  • Verification successful
• You’ll now find a <SKETCHNAME>.apk file in your \android directory:
  • \android\floater08.apk

That apk is now directly consumable by an Android device.  As a test I put it on my ftp, browsed to it from my phone, and it installed immediately.

The next step would be to get it on the Android market, but I’ll work in that in another post.  Happy Holidays!

Go to part 5…, part 7…

Go to part 4…, part 6…

This became far more difficult than anything previous I’d tried to do with the hardware

I thought it would be a simple matter to access the camera’s pixel data in Processing, but that was not the case.   And I should point out I can’t take credit for everything below:  The camera passes back a byte stream encoded in YUV format, that my brain simply couldn’t\wouldn’t decode.  I’d already ran across the Ketai library before (here, & here), and discovered that they had written a YUV decoder function (since they’ve already completed this exercise I’m trying…), so my solution below uses a direct implementation of their code.  So a huge thank you to that project!

As well, I used concepts for my CameraSurfaceView class from examples in the book  Android Wireless Application Development, page 340.

At any rate, it works.  Camera pixel data is passed to Processing, and displayed as a PImage on the screen.  It’s not fast (1 fps?), which is a bit disappointing, but it’s a start!

/**
CameraPixelData
Eric Pavey - 2010-11-15

Set Sketch Permissions : CAMERA
Add to AndroidManifest.xml:
    uses-feature android:name="android.hardware.camera"
    uses-feature android:name="android.hardware.camera.autofocus"
*/

import android.content.Context;
import android.hardware.Camera;
import android.hardware.Camera.PreviewCallback;
import android.view.SurfaceHolder;
import android.view.SurfaceHolder.Callback;
import android.view.SurfaceView;
import android.view.Surface;

// Setup camera globals:
CameraSurfaceView gCamSurfView;
// This is the physical image drawn on the screen representing the camera:
PImage gBuffer;

void setup() {
  size(screenWidth, screenHeight, A2D);
}

void draw() {
  // nuttin'... onPreviewFrame below handles all the drawing.
}

//-----------------------------------------------------------------------------------------
//-----------------------------------------------------------------------------------------
// Override the parent (super) Activity class:
// States onCreate(), onStart(), and onStop() aren't called by the sketch.  Processing is entered
// at the 'onResume()' state, and exits at the 'onPause()' state, so just override them:

void onResume() {
  super.onResume();
  println("onResume()!");
  // Sete orientation here, before Processing really starts, or it can get angry:
  orientation(LANDSCAPE);

  // Create our 'CameraSurfaceView' objects, that works the magic:
  gCamSurfView = new CameraSurfaceView(this.getApplicationContext());
}

//-----------------------------------------------------------------------------------------
//-----------------------------------------------------------------------------------------

class CameraSurfaceView extends SurfaceView implements SurfaceHolder.Callback, Camera.PreviewCallback {
  // Object that accesses the camera, and updates our image data
  // Using ideas pulled from 'Android Wireless Application Development', page 340

  SurfaceHolder mHolder;
  Camera cam = null;
  Camera.Size prevSize;

  // SurfaceView Constructor:  : ---------------------------------------------------
  CameraSurfaceView(Context context) {
    super(context);
    // Processing PApplets come with their own SurfaceView object which can be accessed
    // directly via its object name, 'surfaceView', or via the below function:
    // mHolder = surfaceView.getHolder();
    mHolder = getSurfaceHolder();
    // Add this object as a callback:
    mHolder.addCallback(this);
  }

  // SurfaceHolder.Callback stuff: ------------------------------------------------------
  void surfaceCreated (SurfaceHolder holder) {
    // When the SurfaceHolder is created, create our camera, and register our
    // camera's preview callback, which will fire on each frame of preview:
    cam = Camera.open();
    cam.setPreviewCallback(this);

    Camera.Parameters parameters = cam.getParameters();
    // Find our preview size, and init our global PImage:
    prevSize = parameters.getPreviewSize();
    gBuffer = createImage(prevSize.width, prevSize.height, RGB);
  }  

  void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
    // Start our camera previewing:
    cam.startPreview();
  }

  void surfaceDestroyed (SurfaceHolder holder) {
    // Give the cam back to the phone:
    cam.stopPreview();
    cam.release();
    cam = null;
  }

  //  Camera.PreviewCallback stuff: ------------------------------------------------------
  void onPreviewFrame(byte[] data, Camera cam) {
    // This is called every frame of the preview.  Update our global PImage.
    gBuffer.loadPixels();
    // Decode our camera byte data into RGB data:
    decodeYUV420SP(gBuffer.pixels, data, prevSize.width, prevSize.height);
    gBuffer.updatePixels();
    // Draw to screen:
    image(gBuffer, 0, 0);
  }

  //  Byte decoder : ---------------------------------------------------------------------
  void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
    // Pulled directly from:
    // http://ketai.googlecode.com/svn/trunk/ketai/src/edu/uic/ketai/inputService/KetaiCamera.java
    final int frameSize = width * height;

    for (int j = 0, yp = 0; j < height; j++) {       int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
      for (int i = 0; i < width; i++, yp++) {
        int y = (0xff & ((int) yuv420sp[yp])) - 16;
        if (y < 0)
          y = 0;
        if ((i & 1) == 0) {
          v = (0xff & yuv420sp[uvp++]) - 128;
          u = (0xff & yuv420sp[uvp++]) - 128;
        }

        int y1192 = 1192 * y;
        int r = (y1192 + 1634 * v);
        int g = (y1192 - 833 * v - 400 * u);
        int b = (y1192 + 2066 * u);

        if (r < 0)
           r = 0;
        else if (r > 262143)
           r = 262143;
        if (g < 0)
           g = 0;
        else if (g > 262143)
           g = 262143;
        if (b < 0)
           b = 0;
        else if (b > 262143)
           b = 262143;

        rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
      }
    }
  }
}

Go to part 4…, part 6…

Go to part 2, part4…

The main reason I got my Android phone was to get Processing running on it, and start grabbing the sensor data.  There were no complete examples  (I could find) on how to actually do this though.  After two days of research I got it working.  Some takeaways:

• I learned that Processing is ran as an Android “Activity“:  It enters the Activity at the onResume() state, and exits it at the onPause() state.  You need to override these in your sketch to do what you want.   See the above link to a nice image that shows the state tree.
• All the code you need to author to setup your SensorManagers, SensorEventListeners, and Sensors,  needs to happen in the onResume() function:  Putting this stuff in the sketch’s setup() function won’t work.
• You can make a single SensorManager, but for each sensor you want to track you need to make a unique SensorEventListener, and Sensor.
• Once I figured it out I realized how easy it actually is.  Like most things 😉

Here is a list of resources I pulled from, in order of discovery/usefulness:

• http://wiki.processing.org/w/Android – The main ‘Processing Android’ wiki page.
• http://forum.processing.org/topic/reading-the-compass – Forum reply by Ben Fry that got me started.
• http://developer.android.com/resources/samples/ApiDemos/src/com/example/android/apis/graphics/Compass.html – Android API example.  Outdated (uses deprecated code), but good concepts.
• http://developer.android.com/reference/packages.html – the Android API docs.
• http://www.netmite.com/android/mydroid/cupcake/development/samples/Compass/src/com/example/android/compass/CompassActivity.java – Another example I pulled from.

A (cropped) screenshot off the Phone of the exciting final product!

And the code:

// android_sensorData
// Eric Pavey - 2010-10-10
// http://www.akeric.com
//
// Query the phone's accelerometer and magnetic field data, display on screen.
// Made with Android 2.1, Processing 1.2

//-----------------------------------------------------------------------------------------
// Imports required for sensor usage:
import android.content.Context;
import android.hardware.Sensor;
import android.hardware.SensorEvent;
import android.hardware.SensorManager;
import android.hardware.SensorEventListener;

//-----------------------------------------------------------------------------------------
// Screen Data:
float sw, sh;
// Font Data:
String[] fontList;
PFont androidFont;

// Setup variables for the SensorManager, the SensorEventListeners,
// the Sensors, and the arrays to hold the resultant sensor values:
SensorManager mSensorManager;
MySensorEventListener accSensorEventListener;
MySensorEventListener magSensorEventListener;
Sensor acc_sensor;
float[] acc_values;
Sensor mag_sensor;
float[] mag_values;

//-----------------------------------------------------------------------------------------

void setup() {
  size(screenWidth, screenHeight, A2D);
  sw = screenWidth;
  sh = screenHeight;
  // Set this so the sketch won't reset as the phone is rotated:
  orientation(PORTRAIT);
  // Setup Fonts:
  fontList = PFont.list();
  androidFont = createFont(fontList[0], 16, true);
  textFont(androidFont);
}

//-----------------------------------------------------------------------------------------

void draw() {
  fill(0);
  rect(0,0,sw,sh);
  fill(255);
  if (acc_values != null) {
    text(("Accelerometer: " + acc_values[0] + " " + acc_values[1] + " " + acc_values[2]), 8, 20);
  }
  else {
    text("Accelerometer: null", 8, 20);
  }
  if(mag_values != null) {
    text(("Magnetic Field: " + mag_values[0] + " " + mag_values[1] + " " + mag_values[2]), 8, 40);
  }
  else {
    text("Magnetic Field: null", 8, 40);
  }
}

//-----------------------------------------------------------------------------------------
// Override the parent (super) Activity class:
// States onCreate(), onStart(), and onStop() aren't called by the sketch.  Processing is entered at
// the 'onResume()' state, and exits at the 'onPause()' state, so just override them:

void onResume() {
  super.onResume();
  println("RESUMED! (Sketch Entered...)");
  // Build our SensorManager:
  mSensorManager = (SensorManager)getSystemService(Context.SENSOR_SERVICE);
  // Build a SensorEventListener for each type of sensor:
  magSensorEventListener = new MySensorEventListener();
  accSensorEventListener = new MySensorEventListener();
  // Get each of our Sensors:
  acc_sensor = mSensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);
  mag_sensor = mSensorManager.getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD);
  // Register the SensorEventListeners with their Sensor, and their SensorManager:
  mSensorManager.registerListener(accSensorEventListener, acc_sensor, SensorManager.SENSOR_DELAY_GAME);
  mSensorManager.registerListener(magSensorEventListener, mag_sensor, SensorManager.SENSOR_DELAY_GAME);
}

void onPause() {
  // Unregister all of our SensorEventListeners upon exit:
  mSensorManager.unregisterListener(accSensorEventListener);
  mSensorManager.unregisterListener(magSensorEventListener);
  println("PAUSED! (Sketch Exited...)");
  super.onPause();
} 

//-----------------------------------------------------------------------------------------

// Setup our SensorEventListener
class MySensorEventListener implements SensorEventListener {
  void onSensorChanged(SensorEvent event) {
    int eventType = event.sensor.getType();
    if(eventType == Sensor.TYPE_ACCELEROMETER) {
      acc_values = event.values;
    }
    else if(eventType == Sensor.TYPE_MAGNETIC_FIELD) {
      mag_values = event.values;
    }
  }
  void onAccuracyChanged(Sensor sensor, int accuracy) {
    // do nuthin'...
  }
}

Go to part 2, part4…