Temporal data

The word temporal came into middle English from old French that took the word temporalis from Latin. Temporal data means data constrained by time such as audio or video. This time constraint adds much complexity for web developers.

Unicode character-set is the de-facto choice for how text data should be encoded. This is a straight forward decision as the efficiency and the universality of Unicode supporting all characters in all languages makes it an obvious choice.

To include graphics in documents is slightly more complex. Like text, the data is still static and there are no time constraint but typically each image contains much more data, hence the various compression techniques and formats such as PNG and JPEG to choose from.

With video data things become more complex. For humans to perceive a moving video we typically have to refresh the display with 25 images per second, not to mention keeping a audio channel(s) synchronised at the same time. The audio channel is also temporal data and its quality depends on how many time the audio data is sampled (and stored) over the period of recording. And there could be multiple channels of audio, typically what you hear from a right speaker is different from the left speaker. And just as seeing is only 15% or so by the eye, the rest is what your mind perceives it to be, with audio you only detect multiple audio waves with your ear but its your mind that perceives what it has heard (sometimes incorrectly!). Typically, with 25 fps images for the perceived smooth movement in video as well the audio, the volume of data increases by an order of magnitude to tens or 100s of megabytes.

So, what does this means for developers?

If browser programs included the logic to manipulate the temporal data in the same way that it manipulate text or graphics data then all we need is appropriate elements such as audio or video with appropriate attributes for controlling the playback. This is not a new idea and has finally arrived with html5 and supported by most modern browsers. For non standard or unsupported formats, you still need to install a plug-in player before browser can play the audio data.

Audio

<audio controls="controls">
   <source src="fur_elise.webm" type="audio/webm" />
   <source src="fur_elise.ogg" type="audio/ogg" />
   Your browser does not support the audio element.
</audio>
  1. The audio element has opening and closing tags, the source element is self closing with />
  2. The attribute controls indicates that the browser should use its internal player, other attributes include auto-play to annoy your users! :).
  3. Note, we have kept to XHTML notation with controls="controls" rather than controls
  4. The source element has two attributes, src and type both should be specified.
  5. The src attribute specifies the pathname to the audio file. The filename extension implies the graphics format. If the file does not exist or the correct format the player controls are not displayed or disabled by browsers.
  6. The type attribute also specifies the format of the source file. So, why specify it twice. It is important to include it as the browser can then decide whether it can playback the format as soon as the page is loaded. If the type is not specified the browser would download a large source file and use much bandwidth before it can decide whether it support the format by reading the meta data within the file (we will say more about this in the next section on containers and codecs).
  7. Should the browser not support the music format the source element returns with an error and the last line with "Please use a standards compliant browser!" message will be displayed.

Exercise – Audio

  1. Right-mouse click on the audio player and select “Save Audio As”.
  2. In the portfolio wd folder, create an temporal.html file.
  3. In the temporal.html file, insert a section with an audio element to play the audio file.
  4. Validate the temporal.html file.

Video

<video controls="controls">
   <source src="physics_works.webm" type='video/webm; codecs="vp8, vorbis"' />
   <source src="physics_works.ogg" type="video/ogg; codecs='theora, vorbis'" />
   Your browser does not support the video element.
</video>
  1. Being temporal data, the video element looks very similar to to the audio element.
  2. Despite the file name extension indicating the file type, we still specify the type attribute so that the browser does not have to download megabytes of data only to read the meta data information in the video file and find that it does not support it.
  3. The first part of specifying the type is the open standards containers, video/ogg; and the second part of is to do with how the different video and audio tracks within the container are encoded and decoded; codecs='theora, vorbis'". The video codec in this example is theora and the sound codec is vorbis. See the following sections for an explanation of containers and codecs.

Video containers

The specification of type for video data is more complex as there are normally video and several different audio channels of data all of which has to be synchronised. For example, a video file may contain a video track and two separate audio tracks including marker data for synchronisation to the video images. There will also be meta data such as the video title, length, author and so on. This is akin to a container with several tracks and so when considering the type of the temporal data it is better to think of it as a container of tracks each with its own type rather than a simple type as in images. Open standards for such containers include OGG and WEBM.

A common reason for a correct video not being played in a browser is that most servers don't by default serve Ogg media with the correct MIME types, and it is likely that it needs to be added to the server configuration file. For example, the configuration for the Apache server is:

AddType audio/ogg .oga
AddType video/ogg .ogv
AddType application/ogg .ogg

Encoder/Decoder (codec)

Audio or video codec (short for encoder/decoder) is a specification for how to store temporal data digitally. For example, using your audio recorder, physical audio waves are sampled at particular time intervals, there is a process of quantisation that ends with the audio waves being represented in a digital bit stream which is typically saved as an mp3 or wav format file. A particular codec specification would have been used to encode the sound and this is the second part of the specification in the type definition, so that the browser is better able to decode the encoded sound and play it back through its built in sound player. This is much more complex than representing a character digitally in so many bytes.The rate at which we sample the audio is very sensitive to the volume of data to be stored as well as the quality of the audio. Video codec is a similar idea. You may have heard of how digital TV tries to save on the volume of data by updating only the part of the screen that is changing. Typically these are the consideration that various codec try to optimise. Given the range from High Definition (HD) to low quality video and multi-channel surround audio to mono audio and the variety of devices, it is not surprising that there are plenty of choice in codecs.

The much used MPEG still has copyright issues and for this reason you need an external program (a plug-in) to display it in Firefox. Firefogg plug-in for the Firefox browser will convert most other formats to ogg and webm.

Where do we go from here?

Honolulu sounds a good idea! :) And on return you want to publish your holiday videos (most likely MPEG format) and not to worry which browser on a desktop or mobile is going to view it. HTML5 caters for this by allowing multiple source elements in audio and video elements. The browser can then go through the list until it finds the format it can play. This will require you to convert and save you video into the necessary format supported by various browsers. Three different format copies of the video should cover most modern browsers including Chrome and Firefox. To summarise and with your Honolulu video at hand, You need to decide:

  1. which container (for example: ogv, mp4, webm), which video codec (for example: Theora, vp8) and which audio codec (for example: Vorbis, AAC) to use.
  2. Typical type declartion covering the range of containers and codecs for html5 web browsers are:
    1. type='video/ogg; codecs="theora, vorbis"
    2. type='video/webm; codecs="vp8, vorbis"
    3. type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"

Exercise – Vedio

  1. Right-mouse click on the Video player above and select “Save Video As”.
  2. In the temporal.html file, insert a section with a video element to play the video file just saved.
  3. Validate the temporal.html file.