Digital image processing on Mobile


Skip to end of metadata

Go to start of metadata

Pixel = Picture element

NOTE!  All image processing preferably should be done on the GPU vs CPU.

In Video Streaming we are processing a YUV buffers of images that are coming from Camera.

On iOS the YUV420 format is used

NOTE! YUV and YCbCr are the same thing and may be used interchangeably in any article you may find

Y : Luminance ( fancy word for brightness )

U aka Cb aka Blue : Chrominance ( fancy word for color )

V aka Cr aka Red : Chrominance

Values on those planes ( buffers ) may be negative hence the need for uint8_t ( Unsigned Int )

Agora supplies us with 3 type of buffers for each captured frame :

yBuffer, uBuffer and vBuffer represented as uint8_t * ( thaaaats right, welcome to C ) 

Y′UV signals are typically created from RGB (red, green and blue) source. Weighted values of R, G, and B are summed to produce Y′, a measure of overall brightness or luminance. U and V are computed as scaled differences between Y′ and the B and R values.

Rasterized image example 

The smiley face in the top left corner is a raster image. 

When enlarged, individual pixels appear as squares. 

Enlarging in further, they can be analyzed, with their colors constructed by combining the values for red, green and blue.

Example : RGBA → YUV ( YCbCr )

Y = 0.2126*R + 0.7152*G + 0.0722*B

Chrominance (Cb and Cr) is the correction applied to the luminance to add color information. For ITU-R BT.709 they would be:

Cb  = 0.5389*(B-Y)
Cr  = 0.6350*(R-Y)

YUV Buffers in memory look like that : 

yBuffer : Y1 Y2 Y3 Y4 Y5 Y6 Y7 Y8 ....

uBuffer : U1234 U5678 ....

vBuffer : V1234 V5678 ...

And reads ( this is really fucked up ) as follows :

Each pixel of color is represented by = 4 yBuffer bytes + 1 uBuffer byte + 1 vBuffer byte

so, in our example, to read a single pixel ( top left pixel ) we will have to :

single color = yBuffer[0..3] + uBuffer[0] + vBuffer[0]

Accessing single pixel

to access a single pixel in memory = data_begin + y * stride_bytes + x * bytes_per_pixel.

C example for that : 

uint32_t *p = data_begin;
p += y * stride_bytes / sizeof(uint32_t);
p += x;

Allocating YUV Buffers

Y = width x height pixels (bytes)

Cb= Y / 4 pixels (bytes)

Cr= Y / 4 pixels (bytes)

Total num pixels (bytes)= width * height *3/2

i.e. for each 4 Y(brightness) pixels there will only be 1 Cb(blue) and 1Cr(red) pixel in YUV420 format

C code to allocate YUV buffers will look the following : 

size_t srcWidth  = image.size.width;size_t srcHeight = image.size.height;size_t yBufferSize = destWidth * destHeight * sizeof(uint8_t);size_t uBufferSize = yBufferSize / 4 * sizeof(uint8_t);size_t vBufferSize = yBufferSize / 4 * sizeof(uint8_t);uint8_t * yBuffer = calloc(yBufferSize, sizeof(uint8_t)); /// Alternatively you can use malloc(yBufferSize);uint8_t * uBuffer = calloc(uBufferSize, sizeof(uint8_t)); /// malloc(uBufferSize);uint8_t * vBuffer = calloc(vBufferSize, sizeof(uint8_t)); /// malloc(vBufferSize);

Converting RGB to YUV

void rgbToYuv(byte* src, byte* dst, int width,int height){ byte* src_planes[3] = {src,src + width*height, src+ (width*height*3/2)};int src_stride[3] = {width, width / 2, width / 2};byte* dest_planes[3] = {dst,NULL,NULL};int dest_stride[3] = {width*4,0,0};struct SwsContext *img_convert_ctx = sws_getContext(width,height,PIX_FMT_YUV420P,width,height,PIX_FMT_RGB32,SWS_POINT,NULL,NULL,NULL);sws_scale(img_convert_ctx, src_planes,src_stride,0,height,dest_planes,dest_stride); sws_freeContext(img_convert_ctx);}


Mountain biking app for riding, recording and sharing trails among our community. For any requests contact me at Thanks

Binary Trees

Binary Tree – Recursion Recursion • Recursion is the strategy for solving problems where a method calls itself. • Approach– If the problem is straightforward, solve it directly (base case – the last step to stop the recursion). – Else (recursive step) Simplify the problem into smaller problems. Solve the simpler problems using the same algorithm. Combine … Continue reading Binary Trees

Follow My Blog

Get new content delivered directly to your inbox.