Episode 08: Timing Is Everything

This podcast episode discusses the topic of synchronization in film music composition. The host explains the concept of timecode, which is a system used to keep time in film and television. He explains how timecode is represented and its importance in composing music that matches the picture. The host also discusses file formats and methods for film scoring, including the approximation method, which involves creating cues with fading tails that can be overlapped and aligned in the digital audio workstation. This method is useful for contemporary-style underscore. Hi, I'm Karl Irwin, and this is Spotting Cues, a podcast for the amateur and hobbyist film score composer. On this episode, we're going to be looking at a very technical bit of information here. This is going to be all about synchronization. I think this is a topic that is very interesting to people that are into film music. Even if they're not composers, it's just a very interesting topic to understand the nature of, you know, the scoring sessions and how everything is synchronized, but even more so the composition end of it. How is the music written so that it will match the picture and everything that's going on within the scene? And that's going to be the topic of discussion. So I'm going to give you my own sort of workflow on this issue. I'll talk about some of the old school tactics in very brief terms, but mostly I'm going to talk about just my own pieces of advice that I can give to you from my experience that I think fit well into the amateur and hobbyist tool chest. So first of all, we need to start the discussion with timecode. So there is a thing called timecode, sometimes called SMPTE timecode or S-M-P-T-E timecode. That refers to the Society of Motion Picture and Television Engineers, and it is a system of keeping time in film. And what it represents is hours followed by minutes, and these are two-digit numbers. So if everything is zeroed out, it's zero, zero for hours, then a colon, then you have minutes, zero, zero for minutes, and then a colon, and then zero, zero for seconds, and then another colon, and then the last element is a double-digit number, zero, zero, whenever it's at zero, and that is frames. That's the frame rate. It is the actual number of frames for every second in the film. So for movies, this is usually 24 frames per second. There are some other frame rates out there. They go up quite high. They can go up to 48, sometimes even 60 frames per second. If you're dealing with, I don't know, a specialty kind of footage that might have high-speed photography, they might even have a higher frame rate than that. But really, for film, it is 24 frames per second. There was a time there when some filmmakers were pushing for 48 frames as the new normal for digital projection, but that seemed to be a fad that came and went. It really is 24 frames per second. There's a slight variation on that for the traditional frame rate, which is like 23.98, but it rounds to 24 frames. For television, it's typically 30 frames per second. That is traditionally what it's been. Some TV shows, I believe, were shot, though, at 24 to get that filmic sort of look. Most films eventually get a conversion to 30 frames per second, and there's some interpolation that they do to accomplish that of frames. But these are common frame rates in dealing with film. It's almost always going to be 24 frames per second. So that last number really, really matters because it's a subdivision of the second, but it's a subdivision in ways that we don't think. It's not milliseconds. It's not any other kind of metrical division, decimal, hexadecimal, or imperial division. It is frames, right? Frame rate. And to understand 24 frames per second as a subdivision of the second is very helpful because from that, you can figure and calculate, as we'll talk here in a moment, a little bit down the road. We'll talk about how you understand beats under this time code. Okay? But you do need to understand time code. Time code is very useful. It's essential, if not useful, for a lot of film score composition. In my experience, I use the hard-coded time code. If I get a film that does not have hard-coded time code on it, I will often render it. I'll drop it into a video editor and render time code to it in a lossless way so that I'm not losing any frames. I'll maintain the same frame rate so that frame zero is the beginning of time code. You will get, if you get time code burned in, usually the way it will be burned in is that it starts at zero, but the hours might be set to one hour. Often time code will begin at hour one rather than zero. I'm not sure what the tradition is behind that, but that is usually how time code is burned in. Regardless, you're going to want to use the hard-burned time code. You're going to want to use the burned-in time code because that is what the filmmaker is looking at. If you're depending on the time code in your digital audio workstation or your notation program, there may be some variance in terms of how it relates to frames, and there might be some nudging that you didn't account for. You always want to make sure that your references that you're making to the music that you're writing is a reference to the burned-in time code. If you are working in a situation where you can nudge the video so that it lines up to your workstation time code, that's fine, that's great, and that's very helpful, especially whenever you're dealing with file formats, which is the next topic here. So file formats, I usually output in WAV format as lossless as I can make it. We can talk about bit rates maybe some other time. I don't think that really matters, but WAV format is pretty typical. There's also another WAV format, a BWF, which is a broadcast WAV format. It's really just a WAV file, and it's read by the codecs that read WAV files, but it actually has additional metadata on it that includes the time code metadata. So if you have a hard-coded video, a film, and you drop it into your digital audio workstation, and you do align the time code to the workstation time code, and you have a workstation that can output broadcast WAV form files, it should include the time code information on the file, and that's useful because in larger productions, you can send those WAV files out, and they'll already have the metadata that tells them exactly where in the project to drop it to synchronize. I'll be honest with you, in amateur and hobbyist work, you'll hardly ever need to use that. It just is not going to be something you need to deal with because the films are going to be short. I mean, they're going to be 15 minutes maximum, usually just a couple minutes long for student-type films, so this is not really much of an issue. I often will just submit a single file after picture lock, and we'll work in that format, so it'll be a one-way file for the entire picture that starts at frame zero and has everything on it, and that just keeps things very, very simple for the filmmaker and simple for myself. But if you're working with a larger project, and you're working on, you know, in a picture lock situation where scenes are locking, but the full picture is not locking altogether, so maybe a scene gets locked down, and you're working scene by scene, you may want to submit this SMPTE timecode information with the cues so that it can be dropped into the timecode that is burned into the film that you've been working with. So that's a quick discussion on timecode. It's very, very important to understand it and to be using it because we are, you know, writing to a technical kind of format. We're writing to picture, and we need to relate our music to the picture, sometimes to the frame, not as much as you think, but we do need to be mindful of timing. So let's talk about methods here. The first method for film scoring I want to talk about is one that a lot of people will use, and I find myself doing this a lot more, and mostly because of the kinds of films that are being made, particularly live-action films that are using – filmmakers are asking for more contemporary types of film score, the use of synthesizers, the use of, you know, atmospheric sort of music, and this is what I call an approximation method. It's the approximation method, we'll call it. It's a very common way to write where you don't have to actually figure out a lot of hard-thinking points. You can do some approximation in time. You can look at the amount of time for a particular moment, and then you can score some music that fits into that approximate time, and you might give kind of a hard beginning to it or a faded beginning, but in general, when you're working in this way, you'll create cues that have a fading tail or a tail out. So different kinds of outs you can put on a cue. You can have a hard out where it ends abruptly, maybe at a hit that's been calculated, or it'll have a tail, which has a fade. It's like it ends on one note or a chord, one kind of final resting moment, or non-rest. It doesn't have to be musically resting at all, but just a final moment that then fades gradually so as not to be noticed. And it's very useful because you can take different cues that have tails on them, and you can actually put them together and overlap them. This only works, however, so long as the tails are musically compatible with the start point of the next cue. But this is a very, very useful way to work on contemporary film using just approximation. You calculate the amount of time that you need for a cue, just approximately. You write the cue approximately in that amount of time, leave yourself a little bit of margin, and then you can align it just in the digital audio workstation. Just slide it around, set it where it needs to be, you have that final tail at the end. The nice thing about using the tail method in approximation is that you can actually truncate that fall-off inside of the digital audio workstation. You can just put a quicker fade on it in the alignment to other cues, and it works very, very well. Again, the only way that this works when you're dovetailing cues is if it is created to be tonally compatible. So you've got to make sure that whatever you're going to next in the next cue is some kind of compatible starting point to the tail that you just had as a shared note or a logical kind of modulation that's going on if you're going to be changing key. This is perfectly adequate for non-thematic or contemporary-style underscore that operates informally. And when I say informally, I mean it doesn't have musical form. So when you're doing atmospheric stuff, this works really well in thriller kind of suspense, maybe with horror movies, or even some very, very understated drama, you can do this kind of composition. This also works well for vamps and ostinatos, or sustained synthesized sounds, or symphonic textures that are sustained. They're not metrical or melodic. You can get away with a lot of this kind of approximation. And even with some drawn-out thematic fragments, so you might take a theme and you're playing back the theme, but maybe it's a piece of it, a fragment of it, and it's drawn out, it's augmented and stretched out over time, and because the time is so broad and lengthy, it can be approximated. It doesn't have to be precise. So you can even get away with the approximation method using thematic material in that way. And as I said, the technique can be used to stitch together hard hits, too, really. So if you're looking at like a suspense picture or a thriller or horror picture where there might be a number of hits, you can compose a short hit that is tonally compatible with other hits, which are in turn tonally compatible with other tones or synthesized or sustained orchestral textures, and then you can kind of mix and match these ideas in the workstation. So you can do quite a bit of that. And I just want to point out there's a lot of films that are made in this way right now, big budget blockbuster films, particularly dramas are done in this way, very serious dramas are done like this using ostinato patterns. I mean, I can point out to Johnny Greenwood, he's a, you know, very successful film composer, rock musician, and his music is often orchestral, if not almost always, really. I mean, it's kind of, he works with chamber type music and 20th century kinds of techniques in his music, quite a bit of that. But a lot of those cues are written apart from picture. They're just written based on a idea, and then that music is dropped into the picture and it is edited to length in post-production. You could write film music like this. There's no reason why you can't do that. So approximation, it gets you a very long way. Don't be afraid to do it. Don't feel like you have to do any one of the technical methods when approximation will do. If approximation will do and it will speed up your time to get turnarounds and get approvals, then do that, do that. That works. There's nothing wrong with that. There's no rule, you know, there's no industry rule that says that you have to map everything out perfectly. You can write informally via approximation if you're really clever about it. There is one issue with this, though, is that the technique does have limitation in use. As I said, like if you're working with an animation with a lot of hard hits, it gets kind of hard to do this kind of thing. You'll need to do some tempo mapping, which we'll talk about in a minute. But there's also an aesthetic problem with it. If you're doing only this kind of patchwork approximation, I think it does suffer aesthetically and maybe the viewer, listener won't be able to pinpoint exactly what's wrong, but they may notice that something is very wrong about it, that things don't seem right, they don't seem settled, or they seem a little too predictable in terms of the formula. That there's, you know, there's always some kind of sound underneath, and then there's some other sound that accents a major moment, and then it dissipates, and then the next time the exact same formula plays out. So it can become, I think, aesthetically formulaic, and that can be an issue. So just some considerations. Next look at, we'll look at the absolute mapping, okay? So this is when we need to do synchronization that has to be very absolutely mapped to the picture. First, I just want to touch very briefly on the old school synchronization method. So this was done just through calculation, and it's a simple calculation. You take your beats per minute of the music that you're going to use for the scene, and then you divide that by 60, and it gives you the beats per second. Very simple mathematical calculation. Once you know the beats per second, you can then figure out, based on the time code, where things happen, how many beats are happening up to that point and in between points. And once you know that, you can create a tempo map. You can say, well, this point here in the time code has so many beats up to that point at this tempo, that gives me how many measures. And if it comes up in the middle of a measure, then you know that you can put some kind of a meter change, a metrical change there in that measure leading up to that hit point, either right at that point or at some other point earlier so that it comes out right. And then you just do your, you create your map from there. It does get a little bit more technical when you're thinking about frames, but I don't really want to belabor this because nobody does this anymore. We don't really need to do it. We have enough technology now to synchronize the picture, but it is worth explaining this. The one thing I will mention about it that does relate to even scoring to picture from digital audio workstation or notation to video, it is, it can be helpful to use some tempos that fit well to timing. Tempos like 60 BPM, 90, 120, 150, 180, 210. Some of these subdivide very, very well to the second and are just a little bit easier to think about whenever you have a hit that can land in the middle of a measure on a subdivision of a beat. It just kind of lines things up a little bit better. So that's a consideration. You don't necessarily need to have absolutely every single tempo at your disposal. You can pick ones that are slightly adjusted to fit better into timing. The one thing I will say is that I find no value, personally, some people do and some people may disagree, but personally, I find no value in finding out where all of your hits are in a video and then doing some kind of calculation that tells you what kind of tempo would fit best. I don't find a lot of value in that simply because associations, as I mentioned on an earlier episode, associations to picture are made within approximately a quarter of a second. So if you're thinking about this in terms of frames, 24 frames per second, you're talking about about six frames. If you have something happening within six frames after the thing happens to the frame, so something happens on a frame and then within five frames after that, hopefully less, but within five frames, whatever the sound is you have within that time will be associated with that moment. And that is kind of how we psychologically associate image to sound. Because there's such leeway there in terms of the frame rate, I don't see a value in doing that kind of calculation. I think you can really just pick a tempo that fits well to the minutes and seconds and then go from there, you know, whatever fits the type of music, the style of music that you're after. You can really make it quite simple. There's nothing gained, I think, by picking a tempo that is just a few clicks faster or slower than one of those that fits well with timing. So that's my quick soapbox opinion on that. With timing, picture to notation, you typically, though, will be using DAW. You'll be using digital audio workstation. You're not going to be doing calculations of this sort. But people have done it. I've done it early on 20 years ago when I started this and I didn't have, you know, workstation technology to work from and notation that would, you know, synchronize the video. There was a lot of playing back in, you know, I don't know what I, I can't remember, QuickTime player, I guess, is what I used back then. So I was using QuickTime with .mov movie files, kind of the rage, and, you know, sitting with a calculator and a watch just trying to figure out where things would happen and making a tempo map. So it can be done. If you ever want to find a place where this is done extremely well, you want to look at Back to the Future. We'll talk about that again. But the Back to the Future movies, particularly the first one, that is just a, I think this is an overused term, but it is a master class in tempo mapping. The way that Sylvester was able to get such exact music to a movie, a live action movie, that had such precise hits in it. You'll see that there. And that was all done old school. That was all done through calculation. So this is what we're talking about. Now, you're going to be on DAW, though. You're not going to be doing it this way. So if you're on Digital Quality Workstation or in a notation program, I write from notation. I actually usually write from notation. So I'm usually syncing video to a notation program. When you're doing that, what you can do is drop markers or you'll put out a meter and then an extensive number of measures. You can drop markers in there onto a track. Once you've placed your markers, then you can negotiate the meter and tempo changes and create a tempo map onto which the music will be composed. And this is a very important step. And it's got essential limiting information in it, okay? So limiting information we're already dealing with is, you know, the filmmaker says they want this kind of music. The filmmaker's made the film and it's got this kind of character. You're going to have to write music that satisfies their vision. But another piece of limiting information is this tempo map. Once you map out the tempo, you're writing your music to that map. There's no getting around it. So this is a very vital step if you need to do a lot of hard calculation. Again, this is mostly for action or animations. Most animations I've scored, I've had the tempo map. I didn't do the approximation method at all. Usually, I mapped out the whole film. And they're usually short. So you're talking about three to four minutes of film and really just one large sequence of music beginning to end with a tempo map. And that's what you would do. And an action sequence in a movie would be about that long, too. It'd be several minutes. And you'd map it all out. So you create your tempo map based on the markers that you drop in at the hit points. And one way to do this, to deal with this in terms of musicality, you want to make sure that if you're dealing with musical phrases, if you're writing music that has phrasing, it's got thematic phrasing in it, you want to be writing in two-measure, four-measure, eight-measure, or 16-measure phrasing. If you don't have enough time to start a phrase, then don't start a phrase. You want to extend into that, okay? Or use a truncated size for that phrasing. So you might be writing four-measure phrases, and you have just enough time to start something new but not enough time to get four measures in. You might do a two-measure truncation of a piece of the theme, or a repeat, or an extension of some kind. But you want to be writing with phrasing that fits kind of the standard musical understanding of phrases, because it will be heard. If it doesn't make sense musically, that will affect the picture. Now, there's some places where you can get away with this, especially if there's a lot of hit points that are happening, and you can put some ambiguous, non-formal pieces of music between the hits. That's a different story. You don't really necessarily need to follow this advice. But if you're dealing with music that fits into phrasing, like thematic music that has a cadence point, it has some kind of a development to it, you want to make sure that you're following proper phrasing so that it feels right to the listener. Then, if you have some time left over, you want to extend complete phrases. So you have a complete phrase, and you've got a little bit more time. Rather than starting something new, you want to extend that, either with a repeat of a fragment or an extra measure on the end of it. This is also the place where you would put in a meter change. You could place it, maybe you're in 4-4, and you've got a little bit of extra space in the end, and you've got a hit, but it would happen on beat 4 of one more measure. You could put a 3-4 measure at the very end of that phrase and extend the last note, like a hold, or you could place a run, an extra run, and then you're arriving at the next strong downbeat on the hit, and this is a very, very useful kind of technique. You can also play odd metrical changes in the middle of something, and you can place a hold in the middle that adds a few more beats for one meter change, and then you come back to the standard meter, and that kind of shifts things over one way or the other to get it aligned more accurately for the next hit. Again, I point back to the Back to the Future movies, particularly that first one, because you can, if you know something about music and you're watching these action sequences and thinking, you can kind of tap it out and you see where he'll create a hold or some kind of run pattern that is operating over top of these metrical changes to get the hits to line up. It's a very, very useful tool, and it's very fun, I find, to have to work to that template, to have to put these measures in and say, well, that's what I've got, and then finding creative ways to make it kind of work and fit. I find that very enjoyable, personally. So that's one of the... I find that to be one of the most enjoyable things about writing film music versus just writing any other kind of listening music, concert music, or what have you. So those are some pieces of advice when it comes to timing. Make sure you're dealing with whole phrases, extend phrases at the end, don't start things you can't finish. You can truncate some ideas. You might have what would be maybe an eight-measure phrase, and if you take the beginning of that standard phrase and the end of that standard phrase, you could truncate those two ideas together and come up with a four-measure or a two-measure statement that might help in some places. So think about cutting things up in logical ways and giving them even measures to make it work out in a logical way. If the music is formally ambiguous, if it's just an ostinato pattern, if it's minimalist, this is very common today, minimalist music is all the rage, right, as if it's a new thing. It's not. I mean, Philip Glass really wrote the book on this, and everyone else is just trying their best to emulate him. But minimalist music doesn't necessarily have to follow all these things unless there's something harmonically about it that determines phrasing. You really don't have to worry too much about this stuff. You have a lot more leeway in terms of what you do. And a lot of that style of music can be done in the approximation method without doing hard calculations at all. You can write your ostinato pattern. You can put a fade on it at some point, and you can have a transitional element that goes to the next idea, whatever the case may be, and then you piece those together and stitch them together in the project. Music made in the box. I do want to talk about this. Music that's made all in the computer, and many of you will be doing this because you're hobbyists and amateurs. You're not going to have anybody performing this stuff live, or if you are, you're going to be the one performing it. But music that's made all kind of in the box in the Digital Audio Workstation or in the notation program, but it's played back using samples. I want to just impress upon you the fact, if nobody's going to be playing it, it doesn't have to be legible. You don't necessarily have to have everything sequenced the way a person would need to have it done if they needed to read the music, because nobody's going to read the music. It's all going to be played in sampled sounds. What that means is that you can have hits happen in the middle of a measure. You can create a hold coming from one measure into the beginning of another measure. Halfway through, you have a hit, an arrival, and then you have another kind of suspended or held sort of tone coming out of that measure into the next one, and then pick up meter at the next available beginning measure without changing meter at all. You can do weird things like that, and you can be as complicated as you want to be because no one's ever going to read it. The goal is to create a file, to create, to produce audio. And if that is the goal, and there's no middleman involved, nobody that's going to be playing it, you don't really need to worry about issues in notation. I use MuseScore quite a bit in the Muse sounds for final mock-up since that's been done within the last couple of years, that project. And I find that I write a lot in notation because that's what I'm used to, and the notation doesn't make a lick of sense if someone was going to read it. But it doesn't have to. I'm adding all kinds of different obscure articulations in places that would otherwise be illegible. It would be very hard for someone performing to read that music and play it accurately in a scoring session, and I don't need to worry about that because no one's ever going to read it. It's only being made in my box. So music in the box, just remember this, that we're dealing with in the amateur hobbyist kind of realm. We're dealing with really music production. We're not really dealing with ensemble music in any real way. Whatever we're doing in ensemble music is a mock-up of ensemble music, and the files that you use to create the mock-up don't have to be logical. Now, one reason why this is true, and it's easy to make this case, is because that's how digital audio workstation MIDI mock-ups work anyway. Those people that are writing music directly to DAW, playing into keyboard part for part, generally that has to be cleaned up considerably and made logical and then transcribed into notation for people to read it for larger productions. We're really never doing that. Just remember, you can treat your notation program like you would treat the digital audio workstation environment. You can be very unorthodox about what you're writing in notation, just so long as it works out in the calculation of timing in the end. So, those are just a few final thoughts on writing to timing. So, hopefully you enjoyed this episode. On later episodes, we'll be getting back more into the theory of film music and talking about types and styles of music and what to use and when and what works well and what I've found to work well in subsequent conversations. Want to remind you again, I've got an account on X, formerly known as Twitter, under Spotting Cues. Find me there, subscribe, and be sure to leave me a comment if you have anything you'd like me to address. I'd be happy to consider it and maybe circle back around and give you an episode on things that are of concern to you. So, with that, good luck and happy composing.