Leopard Developer Graphics & Media Overview

June 18, 2008

Mac OS X Leopard provides developers with an unsurpassed collection of cutting-edge graphics, imaging, and media technologies. This comprehensive set of tools allows you to create applications that amaze and intrigue your users with a high level of sophistication. The all-new Core Animation and Image Kit frameworks, the substantially enhanced QTKit and Quartz Composer, and the powerful technologies of Quartz 2D, Core Image and OpenGL all allow you to quickly and easily create rich user experiences in your applications.

When you take advantage of the graphics and media technologies in Leopard and integrate them into your application, you are able to add an extra dimension of detail that takes your application from normal to extraordinary. With Leopard, you can add the latest features to your application more easily, more quickly and with greater consistency. Furthermore, by creatively using these technologies, you’ll be able to further set your application apart from your competition.

32F1206B-286A-400C-8575-AA6742848D4C.jpg

This article, part of the Leopard Technology Series for Developers, provides an overview of Leopard’s graphics and media technologies from top to bottom. First, we’ll take a look at the higher-level frameworks, including Core Animation, Image Kit and QTKit, as well as the improvements to Quartz Composer. Then, we’ll consider the foundations of Leopard’s graphics and media capabilities that the higher-level frameworks are built on, so you can think in terms of accessing these technologies at a lower level for even more control.

Building Graphics Technologies into Your Application

Leopard gives you easy and powerful access to its graphics and media capabilities from your application through several high-level frameworks. Brand new in Leopard are the Core Animation and Image Kit frameworks, both of which can enrich your application with a high level of sophistication. As well, the QTKit framework and Quartz Composer feature several improvements. Let’s first take a look at the Core Animation framework.

Core Animation

Core Animation lets you create groundbreaking user experiences and data visualizations by combining and animating layers of graphics, text, and video. With Core Animation you can use common motion graphics techniques, such as keyframes, transformations, and transitions, to animate your application’s visual content and user interface elements. It allows you to create interfaces with high production values while offering an easy programming model for both Cocoa and Quartz 2D-based applications.

89AE1FEB-7EB9-4C57-93BF-FFE6F2B775B3.jpg

Core Animation is highly efficient. Animations, once set in motion, are controlled by a separate thread from the application’s main run loop. As a result, your application is free to process further events even as an animation is running. If an animation’s properties are changed while it is in flight, the Core Animation framework seamlessly retargets the animation to its new state.

Using Core Animation, your application can effortlessly provide a cinematic user experience to your users. You are able to smoothly move user interface elements around the screen, fade them in and out, and add effects with just a few lines of code. In addition to Quartz and Core Image, you can also leverage Core Video and Quartz Composer to achieve a wide range of visual results.

Image Kit

The Image Kit framework in Leopard is designed to give your application the ability to browse, view, adjust, and process images efficiently. It provides a platform-optimized set of image handling components, including a browser that can handle thousands of images without adversely affecting the memory consumption of your application. Image Kit attends to the details of handling large sets of images in your application, freeing you to focus on adding features that make your application unique.

E8EE6830-B431-4465-BAE6-E5B906B25245.jpg

The Image Kit browser supports multiple image selection, keyboard navigation, and drag and drop. Integration of the image browser into an application is straightforward. You can add the image browser component to your user interface using Interface Builder, and then you simply need to provide a set of callback methods that lets the image browser determine how many images it will display and fetch a particular image. Once you have provided the callback methods, the image browser can display the images you want and allow the user to work with them.

To work with individual images, Image Kit provides an image viewer that allows a user to resize, crop, and rotate images. Along with the image viewer comes an Image Edit panel that allows for many common image adjustments, such as exposure, saturation, contrast, and brightness.

6D4194E0-188D-4BCE-8D0B-62445E8BF675.jpg

Image Kit also provides components for capturing and presenting images. For example, there is a a picture taker component that lets your application access an iSight camera and grab images from it, just like iChat and PhotoBooth.

QTKit

QTKit provides a modern, object-oriented interface to QuickTime, Apple’s multimedia handling framework. Introduced in Mac OS X Tiger, QTKit in Leopard now provides a much more comprehensive set of tools for working with QuickTime in Cocoa.

One of its major new features is the QTKit Capture API that lets you capture video from DV cameras, HDV cameras, FireWire IIDC devices, and USB VDC cameras. It supports the ability to capture and record from multiple devices and can output to multiple devices, as well as directly to QuickTime movie files. The new capture classes are built on a framework that provides frame-accurate capture with timecodes and A/V synchronization. It provides access to the transport controls, such as fast-forward and rewind on a camcorder. And, it can write captured QuickTime data to a file while providing an onscreen preview.

Another key benefit of QTKit in Leopard is the ability to run in 64-bit mode. In fact, it’s the only way to access QuickTime in 64-bit applications. The 64-bit QTKit API is identical to the 32-bit API, with some restrictions on data types. The current C-based QuickTime API will only be supported in 32-bit applications.

Quartz Composer

Quartz Composer, the groundbreaking graphical development environment introduced in Tiger, has been significantly updated for Leopard. Quartz Composer allows you to easily work with units of graphics functionality, called patches, which are similar to subroutines in a text-based programming environment. The patches can be connected together to create a composition. Compositions can then be run in a variety of ways. You can execute them as part of your application in a QCView, in a web page via QuickTime, and in Dashboard widgets.

15DEAA5A-6BFC-40B1-A826-ED551DB9C633.jpg

The Quartz Composer development tool in Leopard introduces a wide assortment of new patches and improved versions of old ones. In Leopard, exciting examples include an expanded patch for programming Core Image filters, an improved JavaScript patch, and an all-new patch for creating OpenGL shaders. With the QC Plugin API, you can go even further and create your own custom patches by using Xcode. Leopard also includes a system-wide repository of motion graphics compositions, image filters, and image transitions, which your application can discover and use.

Relying on Deep Foundations

Mac OS X’s graphics and media capabilities are powered by some incredibly powerful foundation technologies including Quartz 2D, Core Image, Core Video, OpenGL, Core Audio, and QuickTime. These technologies power the Mac OS X user experience and the frameworks upon which Mac OS X applications are built. They are also available for your application to tap into directly. This means that in addition to taking advantage of top level frameworks like Core Animation, you can go directly down into the stack as low as needed in order to make your application work the way you want it to.

1252083D-A24D-4D91-8C60-A63D75363054.jpg

In Leopard, Quartz 2D features an opt-in hardware acceleration layer which offloads more of the work of screen drawing onto the GPU in modern video cards. This can result in impressive increases in drawing performance for your application. Quartz 2D also features a new 16-bit color depth pipeline which joins the existing 8-bit and floating-point pipelines. This pipeline is perfect for applications looking to take advantage of Leopard’s enhanced precision printing capability.

D5BC69F1-64C3-4CCF-BBC5-1E5266618A17.jpg

Core Image in Leopard has been optimized for faster performance and increased multi-core scalability. It now can support the application of hundreds of filters to an image. Filters can also be turned into a macro using the new CIFilterGenerator class. There’s a new automated user interface for browsing and applying filters to an image. As well, Core Image gains access to RAW processing functionality for processing RAW format images from professional digital cameras.

The OpenGL framework in Leopard can provide a dramatic increase in OpenGL performance by offloading processing onto a separate thread which can run on a different CPU core. In essence, this means that instead of performing OpenGL processing synchronously in the run loop of your application, the OpenGL framework quickly returns control of the run loop back and does the majority of its processing in a background thread while your application responds to the next user event. This means that your application quickly gets back to responding to user input and that the OpenGL framework can do everything possible to keep the GPU rendering pipeline efficiently filled.

In addition to performance improvements, the OpenGL framework now features full 64-bit support making a perfect platform for scientific visualization of extremely large data sets.

All of these improvements to Leopard’s graphics and media foundation translate into concrete improvements in the user experience of your application.


Leopard Technology Overview

June 18, 2008

Leopard is the sixth major version of Mac OS X and it will be the most advanced and powerful version yet. For users, it is full of new features and elegant user experience improvements that will make it a joy to use. For developers, things get even better. Leopard contains a cornucopia of cutting-edge new frameworks, streamlined developer tools, new application technologies, and strong system-level foundations. No matter what kind of developer we are, there’s something new in the system that will feel tailor made just for us.

An Operating System Designed for Innovation

The goal in adding so many new developer technologies into Leopard is to make it easier for us to create compelling, rich, and amazing new applications. Once upon a time, a great application was defined by the functionality it provided. Then, over time, the ease of use of an application became important—it wasn’t enough just to get the job done, it had to be easy to do so. In the last few years, the bar has been raised again. Now, a truly great application isn’t just defined by its features and how easy it is to use them, it’s also defined by how enjoyable it is to use.

By taking advantage of the technologies in Leopard, we can add that extra level of detail that makes our applications not just useful, but highly desirable. We can give them that “wow” factor that will bring our customers back to our application again and again.

Getting to Know the Leopard Technologies

These articles will take a deep look at Xcode, Objective-C 2.0, Core Animation, Image Kit, Instruments, and much more.

I’m starting the series with a broad overview of more than two dozen new and enhanced technologies, tools and APIs that are in Leopard.

Developer Tools

As a developer, the first thing we should do when we install Leopard is take a look at the new developer tools. We are in for a real treat. Apple has put a lot of time and effort into improving Xcode and Interface Builder so that it’s easier to create and maintain large projects. They didn’t stop there, however. In Leopard, Apple introduced the entirely new Instruments and Dashcode tools. Instruments will let us take a look at how our application runs like we’ve never been able to before. Dashcode lets us build Dashboard widgets in no time flat.

Xcode 3.0

Xcode is Apple’s Integrated Development Environment (IDE) for creating any code for Mac OS X. It’s the centerpiece application of our workday. Because it’s such an important part of the development process, Apple spent a lot of time making the latest version of Xcode streamlined and faster, in order to make us more productive. The first place we can notice Xcode’s improvements is in the text editor. Code folding, scope highlighting, and iChat-like message bubbles that clearly mark build warnings and errors as well as breakpoints, are some of the key editing enhancements.

8A36DC43-EBDA-4073-944E-E94DDCD47E30.jpg

To help us access documentation while we work, the new Research Assistant gives us a heads-up display of relevant information about symbols in our code in a convenient, small panel. If we need more information, we can simply click for more detail and the full documentation window will open. And, to help us experiment with our code, Project Snapshot lets us record the state of our project and snap back to that state at any point in the future. This lets us quickly try out an idea without having to check things back into our projects source code repository.

Interface Builder 3.0

Interface Builder is the intuitive, easy-to-use tool for creating and editing user interface resources files (nib files). These files represent a set of interface objects and their relationships, as well as their connections to our application’s logic. In Leopard, Interface Builder has been thoroughly modernized and updated, both to support the new interface technologies in Leopard as well as to make the applications we develop easier than ever to build.

0BFDC4E1-BC14-4E2E-84A5-CDF3A70824A2.jpg

In addition to laying out our user interface, Interface Builder now lets us add impressive Core Animation-based behaviors to our application, such as the iChat sliding list view or Keynote-style transitions, in a matter of minutes. And it even has features that improve the ability to see differences between versions of our interface. It also comes with a new Interface Builder Kit API for plug-in developers.

Instruments

Instruments is a brand-new, timeline-based performance visualization tool that gives us the ability to see how our application works like we’ve never been able to do before. It lets me watch CPU, disk I/O, memory usage, garbage collection, events, and more in the form of graphs tied to time.

For example, we can track UI events in real-time and see how they affect I/O activity and CPU load at the same time. And, once the data has been collected, we can scan back and forth through our application’s run and see exactly what occurred, and when.

1A8D413F-4EE1-400C-8A85-CB44C24BF7BD.jpg

Instruments builds on top of the open source DTrace utility. DTrace is a comprehensive command-line monitoring utility that can probe almost any aspect of system performance. It comes with a large set of tools that cover most general aspects of system performance, including both kernel and user code, with near-zero overhead. If there is an aspect of the system that we want to track that isn’t already supported, we can create our own tool using the D programming language.

Dashcode

Dashcode is Apple’s new Dashboard widget development environment. It offers a built-in debugger and all the text editing features of a professional IDE. We can set and view breakpoints as well as track our widget’s execution through its JavaScript source code.

AB40D5DB-DCDA-45B4-A527-08FA6EF83BF3.jpg
Dashcode is geared for all levels of widget developers. We can start out with a prebuilt template and modify it to our needs, or get deep into the JavaScript, CSS, and HTML that defines all the aspects of how a widget looks and works.

Application Technologies

Leopard brings a multitude of improvements and additions to the Cocoa application framework set. The most visible of these is the new Core Animation framework which makes creating compelling interfaces much easier. Not as obvious, but no less important is the addition of garbage collection to Objective-C, new graphical controls, and 64-bit capabilities up the entire Cocoa stack from the BSD core system up to graphics, sound and networking. In addition, Leopard brings a factor of ten increase in the performance of Core Data.

Core Animation

Core Animation is a Cocoa framework for creating animated, composited, and dynamic user interfaces with a simple and approachable programming model. The fluid animations it creates are enabled by Quartz, OpenGL, and Core Image, yet it exposes a familiar NSView-style hierarchy that mirrors that of the Application Kit view architecture. Many of the new user-level features in Leopard, such as Spaces and Time Machine, utilize Core Animation to perform their user interface transitions.

32D16ACC-E480-4A83-8166-53321CBDE3F5.jpg

Using Core Animation, our application can effortlessly provide a cinematic user experience to our users. We’ll be able to smoothly move user interface elements around the screen, fade them in and out, and add effects with just a few lines of code. In addition to Quartz and Core Image, we can also leverage Core Video and Quartz Composer to achieve a wide range of visual results. We can direct Core Animation from either code or by using Interface Builder.

Objective-C 2.0

Objective-C is an ANSI C compatible language with dynamic object-oriented extensions. It combines the power of Smalltalk-style message passing with the performance and native system-level access of C. From the start, the abilities of Obj-C have fueled the power of the Cocoa framework. Now in Leopard, the Objective-C runtime has been updated to include a thoroughly modern and high performance garbage collection system, making memory management a thing of the past.

Also new in Objective-C 2.0 are a set of syntax improvements, including property accessors and fast enumeration. These improvements bring the language up-to-date with the expectations of today’s programmers. And, since Objective-C 2.0 is compatible with existing Objective-C source, we can adopt it at our own pace.

iChat Integration

Instant messaging has become an essential tool for users to communicate and collaborate. In Leopard, the ability to collaborate has been taken to a new level. In an iChat session, users will be able to use iChat Theatre to present data from another application, such as iPhoto or Keynote, as part of a video conference. The applications we create will also be able to integrate with iChat in the same way using the Instant Messaging framework via the Core Video and Core Audio APIs. Once integrated, the Instant Messaging framework will automatically compress the content, using the H.264 codec, and send it across the network to the remote user’s display.

Another new feature in the Instant Messaging framework is the ability to manage presence. This means that our application can query the iChat status of the user, indicating whether they are available or busy and what their status message is, as well as manipulate it. As well, it will give access to the buddy list. If we are building an application which allows users to collaborate on content, we could use the APIs to notify users when their friends or co-workers were online.

Resolution Independence

The old assumption that displays are 72dpi has been rendered obsolete by advances in display technology. Macs now ship with displays that sport native resolutions of 100dpi or better. Furthermore, the number of pixels per inch will continue to increase dramatically over the next few years. This will make displays crisper and smoother, but it also means that interfaces that are pixel-based will shrink to the point of being unusable. The solution is to remove the 72dpi assumption that has been the norm. In Leopard, the system will be able to draw user interface elements using a scale factor. This will let the user interface maintain the same physical size while gaining resolution and crispness from high dpi displays.

The introduction of resolution independence may mean that there is work that we’ll need to do in order to make our application look as good as possible. For modern Cocoa applications, most of the work will center around raster-based resources. For older applications that use QuickDraw, more work will be required to replace QuickDraw-based calls with Quartz ones.

Calendar Store

The Calendar Store framework provides access to a user’s iCal calendar data. Using the framework, we can fetch calendars, events, and tasks from iCal’s data storage. This allows our application to work with the same data as iCal, Mail, and any other application that a manager uses to interact with their schedule and to-do items.

Graphics & Media

Each release of Mac OS X builds on and expands the capabilities of QuickTime, Quartz, OpenGL, and Core Audio, and Leopard is no exception. The Graphics & Media frameworks in Leopard enable all of the higher-level application technologies, and expand them in new and exciting ways.

Image Kit

Image Kit is a new and robust Cocoa-based framework powered by Core Image and Core Animation. It makes available powerful imaging services that can be easily incorporated into our application. It provides functionality for finding, browsing, and viewing images in a collection. We can rearrange images in a group view as well as rotate individual images. Image Kit also provides a gateway for browsing, previewing, and setting Core Image filters on an image.

OpenGL Improvements

OpenGL is the industry-standard API for developing portable, interactive 2D and 3D applications. Mac OS X has supported OpenGL from the outset and in Leopard it adds pixel buffer objects, color managed texture images in the sRGB color space, and improvements in the shader programming API.

Leopard also provides a dramatic increase in OpenGL performance by offloading CPU-based processing onto another thread which can then run on a separate CPU core feeding the GPU. This can increase, or in some cases, approach double the performance of OpenGL-based applications.

QuickTime Improvements

Working with video using QuickTime has long been a strength of Mac OS X. Now, it’s as easy to build video capture into our application as it is to work with pre-recorded video. By using the QTKit Capture API, we can capture video from an iSight, USB (VDC) and FireWire (IIDC) devices, and DV cameras. The API even lets us output to multiple outputs.

QuickTime’s plumbing has received significant upgrades in Leopard. There have been significant enhancements in handling the H.264 encoding. And QuickTime now supports 64-bit. Note that the functionality for 64-bit QuickTime is only provided via the QTKit framework. The C-based QuickTime API is supported in 32-bit mode.

Picture Taker Panel

Mac OS X has used images in many applications for a while. For example, when we first set up Mac OS X, it will take our picture if we have a camera built-in or attached. Leopard exposes the ability to take a user’s picture to any application using the Picture Taker Panel. It will handle all the details of whether a user has a camera built-in or plugged in as well as providing a flash. In short, we can open the Picture Taker Panel and, once the user is satisfied with their picture, our application will be handed a picture return value.

OS Foundations

The heart of Mac OS X, the Open Source Mach- and BSD-based UNIX layer known as Darwin, continues to see improvements in Leopard. The latest Mac hardware brings more processing power, more cores, and more memory than ever before. The kernel in Leopard has been updated to take advantage of this new hardware.

Leopard certainly won’t be UNIX in name only. Apple will submit Leopard and Leopard Server to The Open Group for certification against the UNIX ‘03 product standard.

64-bit Support in Leopard

First implemented at the UNIX level in Tiger, Leopard brings 64-bit support to the Mac OS X application level. We can create 64-bit applications that can address extremely large data sets, up to 128TB using the current Intel-based CPUs. The 64-bit model used in Mac OS X is known as LP64 and is the same model used by other 64-bit UNIX systems from Sun and SGI as well as 64-bit Linux.

And there’s more: the 64-bit support in Mac OS X maintains the ability to run current 32-bit applications. On Intel processors, 64-bit applications have an increased number of CPU registers available in 64-bit mode, and they may run faster than their 32-bit counterparts.

Information Technology

Mac OS X and Mac OS X Server provide a rich set of features and enable a rich set of solutions. Each version of Mac OS X Server has made impressive leaps in functionality while offering easy to use management tools. Leopard provides the most powerful version of Server, built on a host of open source products and tools that make it a server for anyone from sysadmins to IT managers to developers maintaining a lab of test and development Macs.

Best of all, Leopard Server is a superset of Leopard. That means it uses the same kernel and supports the same 64-bit capabilities. Building on top of Leopard’s 64-bit frameworks, all of the services in Leopard Server are 64-bit enabled. This means that they will be able to take advantage of massive amounts of memory as well as enjoy performance increases on Intel-based 64-bit processors.

Open Directory 4

Leopard Server ships with an updated version of Open Directory that supports LDAP proxying, cross-domain authorization, cascading replication, and replica sets. It even supports RADIUS authentication for AirPort base stations deployed across our office or campus.

Calendaring Server

The Darwin Calendar Server allows users to collaboratively share calendars across an organization. It provides a shared location to store their schedules and sync them between machines. It also allows users to send each other invitations to events. Implemented using the CalDAV protocol, it supports multiple calendaring clients, including Leopard’s iCal, Mozilla’s Sunbird, OSAF’s Chandler, and Microsoft Outlook.

Ruby on Rails

Leopard Server features a built-in installation of the powerful and productive Ruby on Rails web application framework. Ruby on Rails is a full stack framework optimized for sustainable productivity. Leopard Server will ship with Mongrel for simplified development and deployment of web-based applications.