Happy hump day! Today, I’ll be discussing some updates over the past two weeks along with our plans for the next two. Let’s jump right in:

Lower Latency Camera Streaming

Since introducing NatML, we’ve been putting a lot of focus on expanding our NatDevice media device API to enable developers build low-latency computer vision applications. We released NatDevice 1.2.0 last week, featuring direct access to raw camera pixel buffers with zero memory copies; EXIF metadata for camera images; support for manual exposure; and much more. And while performance has improved a good amount since the last version, we can do even better:

In the next update, we’ll be introducing the RenderTextureOutput camera device output. This camera device output will perform pixel buffer conversions on the GPU instead of on the CPU, drastically reducing the amount of processing time between receiving a camera image and displaying it on screen. The RenderTextureOutput is specifically intended for use cases where the camera preview data is not needed on the CPU, allowing us to keep as much of our processing on the GPU.

Cropping and Watermarking Videos

Another focus of ours with NatML is in providing the tools for developers to quickly implement the kinds of video editing effects you see on TikTok:

On top of that list are two things: cropping and watermarking. In the next update, we’ll be introducing the WatermarkTextureInput recorder input for adding watermarks to recorded videos with ease; and the CropTextureInput for recording only a region of the input frames, with optional letterboxing. I’m especially excited to create a demo that uses the crop texture input along with face detection from NatML to recreate the effect above.

Finally, we’re looking at adding a way to save only the last few seconds of recording. This has been a common request we’ve received from developers, for quite some time. We haven’t decided whether to release it in the next update, but once we decide, we’ll be announcing it on Discord.

And Other Developments

We’re planning to deprecate NatCorder Performance Extensions (NCPX). Before we do, we’ll be moving all of its functionality into the main NatCorder package. We’ve decided to keep provide these extensions as part of the core package because enough developers consider them an essential part of their development workflow that we can no longer justify maintaining a separate package.

We’ll also be deprecating our open-source NatReader video decoding library once the next NatML update is published. This is because NatML now features full video and audio decoding functionality, allowing for extracting pixel and sample buffers from MP4 videos and other supported formats. Once NatML 1.0.9 is published, we’ll be strongly recommending all developers who use NatReader to migrate to NatML.

And on a final note, we’ll be publishing a NatShare update that includes a permissions API when saving to the gallery. It’s a minor addition, but an important one in certain scenarios. Till next time, happy coding!

— Yusuf.