![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
absolutedestinyAs it's been something of an issue of late and as I've been explaining the ins and outs of the problem to ![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
sockkpuppett, I figured I'd try and extend upon the basic words of wisdom and warning that I wrote on this page of the guide way back when and hopefully offer some insight into this often troublesome problem.
What happens to your memory when you put Avisynth files into Premiere...
Every .avs file requires its own instance of Avisynth. Imagine that avisynth is a program like premiere or firefox or any other kind of program you might launch. Now imagine opening that program once for each .avs file. You're looking at a pretty full taskbar.
Every instance of avisynth keeps hold of a little cache of memory, usually a percentage of your total RAM. This memory is used for caching the series of images that get sent to premiere as video clips. As we all know, the images you get from avisynth are virtual - they are generated as they are needed from your script - so they have to be stored somewhere. Your memory is where they are stored, for a short time, until avisynth decides you no longer need those particular frames for now.
As you may also know, avisynth gives Premiere lossless, full-quality images. Usually 24 bit RGB. If your footage is 720x480, that makes each image being stored in memory 1012.5 kilobytes (around a meg). One second of footage, around 30mb... the keep a second or two in cache per avs file.... It can mount up pretty quickly. Premiere also does a lot of caching - it caches a lot of preview frames, audio, timeline thumbnails etc etc. Add all these caches together and they can be pretty big.
Avisynth will, after some time, clear out its cache; freeing space for new stuff. Generally this is fine and allows you to edit merrily away. But the designers of aviysnth weren't expecting people to use 10, 20, 30 or even 50 .avs files at once. Each of those files has its own instance and each instance has its own cache... meaning each cache has to duke it out with each other AND with Premiere for those precious bytes of your ram. If you then access a lot of different avs clips at the same time with fast cutting, fades or just through exporting, something may eventually break.
What happens when there is no memory left when it's needed?
Sometimes, under the stress of many .avs files, avisynth will try and write to some part of the memory that's already in use... This causes it to crash. For most people, this generally happens during export or long renders as that is when the most sustainably intensive use of all those .avs files happens. As a result, the export will be working fine until it's not and then at best you get an error message instead of footage and at worst you get garbage, black video or a total program crash with no discernible explanation.
That's not even the worst part. When avisynth crashes, it returns a 10 second clip containing an error message instead of your (several hour long) original avs file. This is a clever thing to do if you are just dealing with one avs file because instead of spending hours encoding something that is broken, the encode will be finished in 10 seconds and you'll be able to work out that something is wrong. However, this is of no use when you've edited stuff for Premiere. See, the way premiere puts clips on your timeline is that it stores the time codes - this clip here starts at 1:05.34 and ends at 1:05.41. But what happens if the clip is now only 0:00.10 long? It returns the nearest frame it could find... the one at the end of the 10 second error message... and it does this for every cut on your timeline that uses the 'crashed' or 'broken' avs clip - reducing all your wonderful editing work to useless single frame clips that are all at frame 0:00.10. This is a really disastrous situation - if you save the project like this then the edits are lost - and should this happen to anyone they should *not* save but instead close down premiere and load a recent save before the crash happened. If you save, you lose all your edits on that avs file forever, destroying large chunks of your video (and in some circumstances ruining every clip on the timeline).
Is there anything that can be done to make this better/not happen?
There are a number of ways to handle this problem, which I will got through one by one and in detail. Most people will want to do a combination of the solutions.
1) Use the Premiere AVS GUI program to reduce the SetMemoryMax value to something low (like 8mb).
This has saved me on a number of occasions where I started getting problems either during an export or when getting to the end of a video project and has given avisynth and Premiere just enough breathing room that they don't step on each other's toes. It's the reason I wrote that part of the GUI in the first place. What this setting does is limit the amount of memory each avisynth instance can consume by default (though external filters in avisynth can ignore this).
2) Upgrade avisynth.
A lot of people who use avisynth in premiere do so with the assistance of the old AMVapp that I put together a few years ago. Avisynth has been updated a lot since then and has better memory management these days. It's not perfect but it is an improvement. You can get an updated version here (the latest as of writing is 2.5.7 dated Jan 7th 2007) There shouldn't be any real compatibility issues with the other stuff in the AMVapp, so go and get the latest build of avisynth and install it. As long as you have your project files backed up, it shouldn't be a problem - you can always uninstall and go back to the AMVapp version if there is some unforeseen incident.
3) Upgrade the Premiere AVS Plugin
Just like the version of Avisynth, the plugin that is used to import avisynth files into Premiere has also been updated (very recently, by me and my flatmate Tom). You can get it from here. While there aren't that many differences between the old and the new versions, we managed to write some code that discovers when avisynth has crashed and instead of giving Premiere a 10 second clip, it will give Premiere a 10 hour clip!! So, unless your original .avs footage was over 10 hours long you should never have your edits reduced to single frames any more - or at least it will only happen in the most extreme circumstances. This is good news indeed - I lost a whole day of editing on I Wish I Was a Lesbian because of this problem.
4) Regularly close and restart Premiere.
This will forceably flush the cache and is a pretty good idea if things are starting to feel more sluggish. Of course, if the sheer weight of avs files makes things unbearable from the start then it's not going to be that helpful but if you find that it starts out fine and then gets worse as the editing continues, then this is a simple (if obvious) solution.
5) Export in sections.
If this problem only rears its ugly head during an export, then this is usually because at a certain point in the project it is running out of memory and crashing. The solution? Export the video in sections and combine them later on (with virtualdub or even Premiere). This is the most common occurrence of the memory bug so it's worth keeping this trick up your sleeve for those desperate situations where you have a finished video but have no export to show for it.
6) Use something other than Avisynth to get your footage into Premiere.
While this won't be useful for everyone (and it certainly loses a vast amount of flexibility), a lot of people simply want a reliable way to get mpeg2 footage from their vob files into Premiere without any extra bells and/or whistles. There are (not many) other ways to do this that still have the reliability of the indexing from DGIndex. When I last wrote the guide, this method didn't work but since then, Donald Graft has fixed a bug which makes it a viable option. In recent versions of DGIndex (part of DGMpegDec), the d2v project files that you save for use in avisynth can be converted to 'virtual avi files' using a technique called VFAPI frameserving. This requires two things - the VFAPI codec and the VFAPI application that does the conversion. Google these items, install the codec, run the d2v files through the application and you should get some teeny tiny avi files to use in Premiere... and many other programs. I can't really vouch for its stability as it's not a method that I use but as it's not using avisynth it isn't going to have the same cache issues. [perhaps it will have different ones... I've not tested it thoroughly enough to say.] It could be potentially faster too as it doesn't have quite so much processing overhead.
I hope this goes to some way toward explaining what is going on with this issue and how you can ameliorate the situation.
sockkpuppett, I figured I'd try and extend upon the basic words of wisdom and warning that I wrote on this page of the guide way back when and hopefully offer some insight into this often troublesome problem.What happens to your memory when you put Avisynth files into Premiere...
Every .avs file requires its own instance of Avisynth. Imagine that avisynth is a program like premiere or firefox or any other kind of program you might launch. Now imagine opening that program once for each .avs file. You're looking at a pretty full taskbar.
Every instance of avisynth keeps hold of a little cache of memory, usually a percentage of your total RAM. This memory is used for caching the series of images that get sent to premiere as video clips. As we all know, the images you get from avisynth are virtual - they are generated as they are needed from your script - so they have to be stored somewhere. Your memory is where they are stored, for a short time, until avisynth decides you no longer need those particular frames for now.
As you may also know, avisynth gives Premiere lossless, full-quality images. Usually 24 bit RGB. If your footage is 720x480, that makes each image being stored in memory 1012.5 kilobytes (around a meg). One second of footage, around 30mb... the keep a second or two in cache per avs file.... It can mount up pretty quickly. Premiere also does a lot of caching - it caches a lot of preview frames, audio, timeline thumbnails etc etc. Add all these caches together and they can be pretty big.
Avisynth will, after some time, clear out its cache; freeing space for new stuff. Generally this is fine and allows you to edit merrily away. But the designers of aviysnth weren't expecting people to use 10, 20, 30 or even 50 .avs files at once. Each of those files has its own instance and each instance has its own cache... meaning each cache has to duke it out with each other AND with Premiere for those precious bytes of your ram. If you then access a lot of different avs clips at the same time with fast cutting, fades or just through exporting, something may eventually break.
What happens when there is no memory left when it's needed?
Sometimes, under the stress of many .avs files, avisynth will try and write to some part of the memory that's already in use... This causes it to crash. For most people, this generally happens during export or long renders as that is when the most sustainably intensive use of all those .avs files happens. As a result, the export will be working fine until it's not and then at best you get an error message instead of footage and at worst you get garbage, black video or a total program crash with no discernible explanation.
That's not even the worst part. When avisynth crashes, it returns a 10 second clip containing an error message instead of your (several hour long) original avs file. This is a clever thing to do if you are just dealing with one avs file because instead of spending hours encoding something that is broken, the encode will be finished in 10 seconds and you'll be able to work out that something is wrong. However, this is of no use when you've edited stuff for Premiere. See, the way premiere puts clips on your timeline is that it stores the time codes - this clip here starts at 1:05.34 and ends at 1:05.41. But what happens if the clip is now only 0:00.10 long? It returns the nearest frame it could find... the one at the end of the 10 second error message... and it does this for every cut on your timeline that uses the 'crashed' or 'broken' avs clip - reducing all your wonderful editing work to useless single frame clips that are all at frame 0:00.10. This is a really disastrous situation - if you save the project like this then the edits are lost - and should this happen to anyone they should *not* save but instead close down premiere and load a recent save before the crash happened. If you save, you lose all your edits on that avs file forever, destroying large chunks of your video (and in some circumstances ruining every clip on the timeline).
Is there anything that can be done to make this better/not happen?
There are a number of ways to handle this problem, which I will got through one by one and in detail. Most people will want to do a combination of the solutions.
1) Use the Premiere AVS GUI program to reduce the SetMemoryMax value to something low (like 8mb).
This has saved me on a number of occasions where I started getting problems either during an export or when getting to the end of a video project and has given avisynth and Premiere just enough breathing room that they don't step on each other's toes. It's the reason I wrote that part of the GUI in the first place. What this setting does is limit the amount of memory each avisynth instance can consume by default (though external filters in avisynth can ignore this).
2) Upgrade avisynth.
A lot of people who use avisynth in premiere do so with the assistance of the old AMVapp that I put together a few years ago. Avisynth has been updated a lot since then and has better memory management these days. It's not perfect but it is an improvement. You can get an updated version here (the latest as of writing is 2.5.7 dated Jan 7th 2007) There shouldn't be any real compatibility issues with the other stuff in the AMVapp, so go and get the latest build of avisynth and install it. As long as you have your project files backed up, it shouldn't be a problem - you can always uninstall and go back to the AMVapp version if there is some unforeseen incident.
3) Upgrade the Premiere AVS Plugin
Just like the version of Avisynth, the plugin that is used to import avisynth files into Premiere has also been updated (very recently, by me and my flatmate Tom). You can get it from here. While there aren't that many differences between the old and the new versions, we managed to write some code that discovers when avisynth has crashed and instead of giving Premiere a 10 second clip, it will give Premiere a 10 hour clip!! So, unless your original .avs footage was over 10 hours long you should never have your edits reduced to single frames any more - or at least it will only happen in the most extreme circumstances. This is good news indeed - I lost a whole day of editing on I Wish I Was a Lesbian because of this problem.
4) Regularly close and restart Premiere.
This will forceably flush the cache and is a pretty good idea if things are starting to feel more sluggish. Of course, if the sheer weight of avs files makes things unbearable from the start then it's not going to be that helpful but if you find that it starts out fine and then gets worse as the editing continues, then this is a simple (if obvious) solution.
5) Export in sections.
If this problem only rears its ugly head during an export, then this is usually because at a certain point in the project it is running out of memory and crashing. The solution? Export the video in sections and combine them later on (with virtualdub or even Premiere). This is the most common occurrence of the memory bug so it's worth keeping this trick up your sleeve for those desperate situations where you have a finished video but have no export to show for it.
6) Use something other than Avisynth to get your footage into Premiere.
While this won't be useful for everyone (and it certainly loses a vast amount of flexibility), a lot of people simply want a reliable way to get mpeg2 footage from their vob files into Premiere without any extra bells and/or whistles. There are (not many) other ways to do this that still have the reliability of the indexing from DGIndex. When I last wrote the guide, this method didn't work but since then, Donald Graft has fixed a bug which makes it a viable option. In recent versions of DGIndex (part of DGMpegDec), the d2v project files that you save for use in avisynth can be converted to 'virtual avi files' using a technique called VFAPI frameserving. This requires two things - the VFAPI codec and the VFAPI application that does the conversion. Google these items, install the codec, run the d2v files through the application and you should get some teeny tiny avi files to use in Premiere... and many other programs. I can't really vouch for its stability as it's not a method that I use but as it's not using avisynth it isn't going to have the same cache issues. [perhaps it will have different ones... I've not tested it thoroughly enough to say.] It could be potentially faster too as it doesn't have quite so much processing overhead.
I hope this goes to some way toward explaining what is going on with this issue and how you can ameliorate the situation.
