FORGE AHEAD?

I have seen many presentations in which a vendor claims that its platform is ideal for handheld application development that includes audio and video capabilities. During questioning, the conversation explores the vendor’s development tools and support infrastructure, including third-party integrators and library providers. I have often wondered whether it is even half as easy as I am told it is to implement these capabilities on these platforms. Eventually, such wondering became the seed concept for a hands-on project: Would these development support infrastructures allow someone without an expert knowledge of audio and video to include them in a design?

I approached Texas Instruments about using the Innovator Development kit for the OMAP (Open Mobile Application Processor) platform to implement some audio and video functions from the perspective of an application developer. The OMAP5910 device, one of the processors that the Innovator development kit supports, is the first OMAP with general availability.

It integrates an ARM925 micro controller core, a TMS320C55x DSP core, and shared on-chip resources and peripherals into a single device. Texas Instruments claims that ARM developers can use the device with the operating systems, software-development tools, and standard APIs (application-programming interfaces) that they are familiar with, without an intimate knowledge of DSP programming.

One key assumption that aims to allow application developers to develop with a DSP without learning the DSP internals is that someone else writes the DSP code and provides the interface code to interconnect that code into their preferred APIs. Looking at Texas Instruments’ network of independent OTCs (OMAP Technology Centers), which provide system-integration expertise; the third-party-network catalog offerings for compliant algorithms; and the way the DSP/BIOS Bridge works suggests that this goal may be realizable. The DSP/BIOS bridge is the interconnection mechanism between the cores for OMAPs, and it places the DSP in a secondary or coprocessor capacity.

Why consider a dual-heterogeneous core device for audio and video when the same capabilities exist in single-core micro processor systems, and programming for multiple cores, especially heterogeneous ones, is more complicated? Power consumption for handheld applications is one reason. An application that partitions well between the cores can allow lower clock rates, take advantage of the DSP’s computational efficiency, and allow support for more features than comparable single-core systems. However, developers tend to choose devices they are familiar with for new designs (Reference 1). Texas Instruments’ development-support infrastructure is one strategy that makes OMAPs more accessible to traditional micro processor developers.

A primary goal of this project was to put this development-support infrastructure—which includes the OTC, third-party networks, and Web resources — through its paces. Another goal was to determine how much effort adding recording and playback support for audio and video to an application would require — especially because I have little direct technical experience with these capabilities. If adding them proved as easy as, say, implementing them in a single line or small block of code, I would try more interesting tasks, such as merging and overlaying audio over a recorded movie.

A significant detail changed this project from a pure exercise in implementing audio and video capabilities to what I have dubbed an early-adopter exercise: The OMAP5910 and tools were still in the sampling stage and in pre production testing during the development of this article. Therefore, some components, such as the video driver, might be unavailable before the project’s conclusion. Nonetheless, I believe that the proposed project, even with a smaller set of technical goals, would still be valuable to you.

THE SETUP

WinCE .NET (WinCE 4.1) made the most sense as the operating system to accommodate the small window for this project, because it was probably the easiest operating system for me to get up to speed with and because of its support for audio and video. Selecting the audio and video codecs, AMR (adaptive-multirate) and MPEG-4, was somewhat arbitrary, because my main goal was not to implement for maximum performance, compression, or quality, but to explore how to implement audio and video manipulations through the DSP without detailed experience.

Because I would be working with members of Texas Instruments’ OTC companies and third-partyalgorithm providers, a key element for moving this project forward was establishing a working relationship with them. Stellcom, an OTC company, acted as an integrator, providing consulting support for development and application questions that would arise during the project and integrating the operating system, board-support package, and DSP/BIOS bridge that would run on the Innovator. Stellcom also built and provided the software - development kit necessary for the ARM micro processor core. Emuzed, a third-party algorithm provider, contributed the DSP algorithms that interfaced with the DSP/BIOS bridge for encoding and decoding AMR and MPEG-4.

The Innovator development kit is a handheld, expandable development platform that comprises a processor module, an interface module, an expansion module, a camera module, and an LCD / touchscreen panel (Figure 1). A breakout board accommodates these modules and provides Ethernet, keyboard, and mouse support (Figure 2). Besides the diagnostics bootable kernel, the kit includes two bootable user-flash areas.

The boot sequence for this project began with Microsoft’s boot loader in one userflash area and completed by loading the WinCE .NET image from the other user-flash area. I updated both flash areas several times throughout the project because each version integrated more capabilities into the system image, through a USB link with a host-boot utility on the desktop. I did not need to use Texas Instruments’ Code Composer Studio because I was not developing any DSP code; the DSP code was coming from Emuzed. I would be able to load the DSP code via cexec, a WinCE console application that is available on the microprocessor core as part of the DSP/BIOS Bridge component.

I did not need to use Microsoft’s Platform Builder, because Stellcom was using it to roll a software-development kit for the Innovator development kit. Instead, I needed to use only Microsoft’s EVC/C++ 4.0 (Embedded Visual C/C++), along with the software-development kit from Stellcom, to develop the application code.

PUTTING IT ALL TOGETHER

Because it was a work in pro g ress, the entire development environment did not arrive in one box. It came in pieces and from different sources. I received the Innovator kit from Stellcom after the company had loaded the boot kernel and initial operating-system image. The CD that came with the Innovator kit included the boot-host utility for subsequent updates to the user-flash areas. I downloaded the ActiveSync and EVC/C++ tools from Microsoft’s Web site. ActiveSync allowed me to work directly with the Innovator kit, including single-stepping through the code, from the EVC/C++ environment through the USB port. It would be sometime later that I would receive the DSP algorithms from Emuzed via an FTP site that Stellcom set up to facilitate transferring the latest version of the software-development kit and operating system image.

Unfortunately, EVC/C++ requires Windows XP or Windows 2000, so I had to upgrade my workstation operating system. It took two operating-sys temupgrade attempts to get my workstation fully functional. The reinstallation was predicated by a noncooperative virus-protection package that apparently did not completely uninstall itself before the upgrade. Fortunately, uninstalling the operating system restored the workstation to its former condition, and I was able to find the manual procedure for completely removing the virus - protection package. If you ever find yourself performing an operating-system upgrade, check the Web site for any applications, such as virus-protection software, that might have a tight coupling with the operating system. Simply uninstalling the software via the add/remove programs icon in the control panel may not be enough.

Products go through a pre production phase for a reason. There are probably many functions that developers have yet to completely implement, interface drivers that do not yet include robust error trapping and recovery, or even capabilities that developers have yet to test. I did not always know where to find the documentation, and I sometimes did not receive some of the documentation right away, specifically for the DSP/BIOS bridge. Initially, this lack of documentation was a source of frustration. As I quickly gained experience with the system, however, my frustration lessened, because I better understood how the system operated. Going to a workshop or class for the platform was unnecessary but would have been a good idea.

We quickly determined that e-mail was not a good way to transfer file updates, because the files were large—some as large as 50 Mbytes. Even when the files were small, if they resembled executable files, the antivirus scanner software on the e-mail server stripped them out. Instead, we used an FTP site. I began having problems accessing the FTP site when I tried to download the second update. It took a few tries and a little longer than a week to resolve things. Fortunately, we were able to rely on overnight “sneaker-net” to move the update files around on a CD.

Connecting over the USB port did not work for the first system image, but the second system image addressed it. Because I was not working directly with any of the hardware, I did not need to use the breakout board. Assembled, the Innovator kit looks like a PDA, so I made the mistake of thinking it would function like a consumer product. This assumption bit me when I tried to make my first USB connection with ActiveSync. I connected the USB cable to the Innovator kit and turned on the power, just like I might do with any consumer product. Instead, I needed to power up the Innovator kit and let the operating system complete loading before connecting the USB cable. The development power-up sequence differed from a production power-up sequence in that it relied on the operator to manually ensure that everything initialized properly and in the correct order before linking the two systems together.

When you are working with other teams, it is a good idea to know what software, including the version, each is working with. I was trying to use the latest version of the boot-host utility on the CD in the kit to update the user-flash areas, but it was not working. After speaking with Stellcom representatives, I found out that the company was using an earlier version. After I received that version, I was able to update the flash. From then on, I checked the time stamps on all files to identify discrepancies and avoid version incompatibility. Disconnecting another tool involved installing the software-development kit for EVC/C++. Because I was using EVC/C++ instead of Platform Builder to build my applications, I did not have all the files that Platform Builder installs. When I received the first software-development-kit-installation file, it did not properly link with the EVC/C++ e n v i ronment, because it assumed that certain files were already on my system. When building or receiving software-development-kit-installation files, determine which assumptions about the target system the install wizard uses, so that you deliver or receive all the necessary files.

When I received the first iteration of the DSP algorithms from Emuzed, I found a capability that had not yet been tested—namely, the DSP/BIOS bridge. The system image had so far not included it. This discovery caused some heartache. Adding the bridge required the removal of certain software-development-kit components, because the system image was already straining the available flash storage. The final version of the user-flash image was almost 15 Mbytes, and the runtime footprint was just less than 2 Mbytes.

The ideal picture of DSP transparency in OMAP platform development for using multimedia in a WinCE .NET environment is that the application developer has to work only at the high-level Direct X API. Therefore, any DSP code must also include applicable filters or wrapper filters that reside on the microprocessor side of the device. It must also register the filters and DSP codecs to the operating system, allowing application developers to use the DSP codecs in the same as way as they would use any normal codec installed on the micro processor. Because this hands-on project used tools that were still works in progress, we could not realize this scenario.

Only the audio driver became available during the project window; the video-camera driver would not be available until after the project concluded. Therefore, video capture would have to wait until the driver was available, or I would have to write directly to the low level control for the camera from the application code. Once I realized that synchronized video and audio would be such a complicated task, I chose to focus on implementing audio. Unless a desire to be the first to market drives your project schedule to conflict with driver availability, there is probably little return on the resources you’ll invest to directly drive the camera interface. You will not likely have to make the same trade-off when using production devices and mature development tools.

I was, however, able to use the audio driver. Implementing .wav audio was a straightforward process. The Waveform Audio API recognized .wav files, because the filters for .wav files were part of building the operating-system. Using the AMR encoder and decoder was a different story, because the filter support that would normally be available with a production package was not yet ready. When the filter support is available, you can use the simpler, synchronous, high level, multimedia streaming APIs to manage the resources. However, in this case, the lack of filter support meant that to implement AMR support, I would need to create my own filters or more directly manipulate the data in the application code.

Using the Waveform Audio API was straightforward and allowed me to insert DSP/BIOS-bridge calls as necessary. I needed to add a few functions calls to create, delete, and execute the DSP nodes for encoding and decoding. The application code handled capturing the data, routing it to the encoder, storing the file, routing the stored file to the decoder, and, finally, playing it back (Figure 3). Other than the DSPnode create- and delete-function calls, there was nothing to indicate that the DSP and not the micro processor executed these functions. Effectively, I manually implemented a custom filter graph and filtergraph manager in the application code. Taking the next step to convert the code into a set of filters that could register themselves with the operating system would allow other applications to use the new media format.


Reusing software across many projects continues to grow as a strategy for completing more complex software development in less time. This reuse is more complicated for multiprocessor applications and even more so for heterogeneous - core designs. The DSP/BIOS bridge reduces the interconnect variability that third party vendors must support when they offer algorithms for such devices—increasing the opportunity for developers to use these algorithms across many projects. On a whim, I called an ARM-tool provider that was not listed in the Texas Instruments marketing material. The company had recently begun exploring how to support the bridge with its operating system and tools, because its customers were asking it to.

If you do build your own tools and drivers because they are not available when you need them, be sure that you build them so that you can use them across multiple applications or that you can easily port to the production instantiations when they are finally available. Implementing the peripheral control and filter graphs as application code is quick and dirty and may support quick turn a round for your current project, but it may complicate your effort during the next generation development. Unless you want to be in the business of updating your drivers, you will want to know how to port to the production-supported drivers after they become available.

This hands-on project was made more challenging because it was a part-time activity and I am not an expert Windows programmer. I could not apply my own expert knowledge to develop missing or late drivers or filters as easily as someone doing a true development project using pre production silicon and tools might be able to. However, this project did illustrate some of the wait-for-or-build-it-yourself tradeoffs that design teams must make when choosing to use pre production products and support.

I can see how a micro processor- biased application developer can be spared of having to learn the intimate details of DSP programming when using a production-supported OMAP. I was able to demonstrate how to offload the audio encoding and decoding to the DSP while working entirely with the development tools that a Windows - application developer would normally use. Unfortunately, due to time and complexity constraints, I did not get to implement the more interesting data streaming, transformations, and synchronization that I had hoped to, because all of the support software necessary to simplify these capabilities was not yet available.

Implementing audio and video support in an embedded system is a less mature process than implementing them in a desktop environment. Embedded systems, by design, are more resource-constrained than desktop systems, complicating the implementation and support of audio and video. The operating systems for these systems generally cannot afford to include the same rich support for audio and video that desktop systems enjoy. However, I do see processor vendors and their development-support infrastructures making headway to bring these capabilities to handheld systems. This project would have been very different had we done it six to nine months later, but then, those few months can mean the difference between being the first to market and playing catch-up.

Sidebar: Bridging between processors

There are many mechanisms for interconnecting multiple processors (Re f e re n ce A). The interconnection mechanism is usually a per-design option for systems with heterogeneous processors. Texas Instruments’ DSP / BIOS bridge specifies a standard mechanism for interconnecting microprocessors and DSPs. By specifying this mechanism, third party vendors can design to that mechanism and offer plug-and-play algorithms that run on the DSP but are controlled by the microprocessor. DSP / BIOS bridge is software that resides on both processors and links the operating system of each processor. The microprocessor API (application-programming interface) enables application code to initiate signal processing on the DSP; exchange messages with DSP tasks; stream data buffers to and from DSP tasks; pause, resume, and delete DSP tasks; and perform resource-status queries. The DSP A PI enables mess age exchange and streaming large blocks of data between the processors. The microprocessor code can specify the inputs and outputs that a DSP task uses. Message objects are a primary
mechanism for passing control or status information or for streaming real-time data.

A DSP node groups related code and data into functional blocks. The resource manager is responsible for dynamically instantiating DSP resources to meet allocation requests, monitoring DSP resources, dynamically loading DSP code as needed, and implementing policies for managing DSP resources when there are conflicting requests. WinCE .NET and Linux exposes the high-level DSP / BIOS bridge A PI that builds on a lower level DSP - linkdriver. Nucleus and VxWorks expose the lower DSP link driver to minimize resource overhead at the expense of scalability. To use the bridge, at least under WinCE, you need to include the DSP / BIOS bridge header files and link the appropriate libraries in your application. There are two methods for loading and running DSP base images. The first method is to use the AutoStart feature that can automatically load the DSP base image upon WinCE boot-up. The other method is to use a WinCE console application, cexec, to load and start a new DSP image from the WinCE target processor. You must practice caution when using cexec, how ever, because it resets the DSP without notifying the WinCE applications that may be using the bridge API .

Sidebar: Sampling versus production

Sampling is a method by which a processor vendor can offer preproduction silicon to designers to allow them to evaluate and develop products before the processor begins full production. Sampling devices are functionally accurate, although they may still have bugs, but they have not gone through a full electrical characterization and qualification process. How ever, programmable processors are fairly useless unless there is some software support. The drivers, board support packages, operating systems, and other software-development tools must also undergo a qualification process.

OEM-system developers, integrators, and tools providers need early access to these devices. How else can they provide production-level services, add-on boards, and software-development tools when the production devices are fully released? Other developers, such as application developers, who choose to use devices that are in the sampling stage, often need access to leading-edge technology to meet first-to-market windows. The first-to-market window is not the same as the time-to-market budget. You will expend more time and more engineering resources to complete your design with devices that are in the sampling stage. You may need to spend more time experimenting with the device and tools, because the documentation is incorrect or incomplete. You might experience integration troubleshooting and workaround scenarios that production device users avoid. You need to bring more of your own expertise to bear on a design when using devices that are available for sampling. You may even need to build or modify your own drivers and other tools, because they may not exist in the timeframe during which you need them.

So, is this development premium worth it? It is essential for OEM - system developers, integrators, and tool providers to provide products to their customers—namely, the users of the production devices. For application developers, the payoff for incurring this development premium comes if their product is at the right time to market and if they build brand value before their competitors even get a product out.

Sidebar: Filters

A filter is a COM (component-objectmodel) object that supports DirectX interfaces and uses ”pins“ to allow connection to other filters. A filter graph is a set of filters connected by pins and synchronized to the same reference clock. A filter-graph manager oversees the connection of filters in a filter graph and controls the media stream’s data flow. It connects filters in the proper order and starts and stops the data stream in the proper order (Figure A).


DirectX supplies filters for playing and converting many media formats. DirectX APIs (application-programming interfaces) range from high-level, which, for example, allow you to build the entire graph, to low - level, which, for example, allow you to connect one pin to another. The most basic types of filters are source, transform, and rendering filters. A source filter introduces data from a source, such as a file, a microphone, or a camera, to the filter graph. A transform filter processes data by, for example, parsing it, decompressing it, or applying effects and then passes on the data to the next filter. A rendering filter feeds data to the output, such as a hardware device or a file system, but it could be to any location that accepts media input.

You can also build your own filters for handling custom and standard data formats. Filters are registered in the Windows registry with a GUID (globally unique identifier) and other information, such as which media types it supports. The filter -graph manager employs an intelligent-connection mechanism that can search for a set of filters to render a media type and build its filter graph. This process involves either using the file extension or searching for special “check bytes ” in the file. It generally results in the filter graph's finding a source filter that it creates and adds to the filter graph, as well as telling the source filter the name of the file. An application can also manually disconnect any filter in a graph and replace it with another filter. When an application starts, pauses, or stops the media stream; plays for a particular duration; or seeks a particular point in the stream, the filter-graph manager calls the appropriate methods on the filters to implement stream control.


Reference
1. Cravotta, Robert, “Control and signal processing: Can one processor do it all?” EDN, March 7, 2002, pg 63.

Acknowledgments
Special thanks to the people at Texas Instruments, Stellcom, and Emuzed for working with me during this project, even though the tools were still works in progress. Special thanks also to Christy Brunton from Texas Instruments, who coordinated the teams throughout the project.

You can contact Technical Editor Robert Cravotta at (1) 661-296-5096, Fax (1) 661-296-1087 E-mail rcravotta@edn.com