Recently I posted about an AvsPmod macro by developer vdcrim for piping RGB to Imagemagick but didn’t go into detail on how to achieve this, so here goes.
First, why convert to RGB image sequences and why 16bit? It’s a plain and simple work around for applications that require RGB import or in the case of Blender convert to RGB at import regardless. If the NLE or grading application you choose does a color space conversion from YCbCr to RGB at 32bit float precision then you probably don’t need the 8 to 16bit conversion steps.
But more with regard to Blender, the color space conversion is done by ffmpeg at 8bit integer, with cheap interpolation and a typical 16 – 235 luma, 16 – 240 chroma to 0 – 255 RGB. Very few video cameras, particularly ‘prosumer’ use that restricted range, most mpeg2, mpeg4, h264 videos captured by these cameras are 16 – 255 or even 0 – 255 YCbCr, so highlights above YCC 235 are pushed to white RGB 255 and any shadows below YCC 16 are compressed down to RGB 0 in the conversion to 8bit RGB probably giving you a typical high contrast output with a fair amount of artifacts at edges.
If you happen to be using Canon, Nikon and Panasonic GH3 h264 MOV’s then they’re all JFIF ie: JPEG levels, normalized levels over full 0 – 255 and the h264 has the ‘fullrange’ flag set on to signal to scale levels before the conversion to RGB, ffmpeg respects the flag so Blender’s conversion to RGB is fine with regard to clipping but the interpolation is still cheap Fast Gaussian interpolation so edges suffer rather than using bilinear, bicubic or Lanczos. Maybe the reason is for minimum processing or just for speed of import and playback, I don’t know.
Why 16bpc, because even a simple 8bit to 16bit conversion appears to performs much better in ‘grading’ in Blender and other applications like Darktable, than simply importing 8bit into a 32bit processing workflow. Even better if we add a deblock, denoise, add discrete noise/grain processing step before conversion to 16bpc.
First of all a few prerequisites including installing Avisynth, Imagemagick hdri MS Windows build is required for this macro and collecting a few Avisynth plugins and the Pipe To RGB macro:
I then created a Project Folder with subfolders like this:
Within the 16bit folder I added a sub folder called “Scenes”
For this test I created a short video edit in kdenlive http://www.kdenlive.org/ using Canon h264 remuxed to mp4 with a special build of MP4Box, why? http://blendervse.wordpress.com/2012/04/02/waiving-the-fullrange-flag/ and as long as no filters or processing are added to kdenlives timeline it’s possible to avoid any color space conversion to RGB, (which would be at 8bit and not lossless) and then rendered out to a lossless codec like ffv1 in a matroska container.
I added the source.mkv output file from kdenlive to the ‘Source’ Project Folder.
An Avisynth script is required next to process the source.mkv file.
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\mvtools2.dll”)
#LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\ffms2.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\fft3dfilter.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\removegrain.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\mt_masktools-25.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\dfttest.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\smoothadjust.dll”)
source = ffmpegsource2(“Source.mkv”, threads=1)
return source#Gives us 8bit. Hash the line out for 16bit.
Dither_convert_yuv_to_rgb(matrix=”601″, tv_range=false, cplace=”DV”, chromak=”bicubic”, lsb_in=true, output=”rgb48y”)
Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false, curve=”709″)
Dither_convey_rgb48_on_yv12 (SelectEvery (3, 0),SelectEvery (3, 1),SelectEvery (3, 2) )
Save the script as Source.avs into the ‘Source’ folder.
The script is very simple and does very little processing to give us 16bit RGB. It is possible to do many different processing stages dependant on source file such as adding a denoising process but for now, keep it simple.
Just to mention a few of the settings within the ‘Dither…..’ lines of the script:
matrix=”" relates to the luma coefficients, ie: color matrix of the source file.
cplace=”" relates to the chroma placement of the source file. DV, MPEG1 or MPEG2
curve=”" relates to the Transfer Curve applied to the source file. 709 or sRGB
tv_range= relates to whether the source file luma is within the 16 – 235 8bit range (tv) or not. If unsure amend the 8bit line to this: return source.histogram(mode=”classic”)
The “classic” histogram is a luma waveform, if you see any waveform outside of the black 16 – 235 zone of the histogram you’re camera or other video source is ‘full range’ so tv_range=false, make sure of using a high contrast shot to test with.
The Dither_y_gamma line linearizes the output. Hash the line out if you prefer to leave them with gamma applied.
Best to read the Dither.html file included in the plugin download to establish your settings.
The script serves two purposes:
1. With the “return source#….” in place we are able to view the 8bit decompressed frames without any processing overhead in order to shuttle through the video and set bookmarks for ‘Hero’ frames first and then set bookmarks for ‘Scene’ splitting.
2. Hashing out the “#return source#….” line invokes the remainder of the script, the conversion to 16bit per channel RGB.
Stage 1 – Hero Frames
Open the Source.avs script in AvsPmod. You should see the first frame of your video.
Use the forward & back arrow keys on your keyboard to shuttle through the frames & press ctrl+b to set a bookmarks ready to generate ‘hero’ frames for looks development. ‘Hero’ frames may be those that best describe a scene with regard to visual appearance but maybe also worst case of shadows and highlights re clipping.
Each time you add a bookmark a small black triangle appears below the timeline.
Once you are ready to export your hero frames add a hash to the start of the “return source#….” line, the whole line should turn green.
I’ve found that at this point before starting the intensive 16bit conversion that using AvsPmod’s ‘Release all videos from memory’ appears to help memory usage. This is on the RMB menu when in the video frame area of AvsPmod.
Once actioned the video preview will disappear but your bookmarks will remain.
Then choose from the Macros menu “Pipe RGB To Imagemagick”:
The macro works by automating the splitting of our source file into manageable chunks due to memory overhead of processing and batches the conversion.
Specifying ‘Split by frame step’ allows us to set the number of frames to have in memory at one time, 10 is a good start.
If you choose the ‘Split by time step’ alternative then set a Time step instead, if splitting by ‘Number of intervals’ set number of intervals ie: How many divisions of the source file.
For ‘Hero’ frame export tick the box for “Include only bookmarks if any”, this will export only the frames at bookmarks.
The Imagemagick processing arguements are:
-limit memory 500MiB -limit map 1GiB -depth 16 -size 1920×1080
Amend the frame size to suit your source files. Be aware if using native Canon DSLR video the frame size could well be 1088 lines in height depending on camera model.
FFmpeg doesn’t crop the bottom 8 lines where as QT or similar codecs do. The kdenlive output in my test, was based on Canon h264 but kdenlive via MLT crops the 8 off the bottom. Hence size 1920×1080.
Choose an output folder for the 16bit image frames. Following the Project Folder set up above the folder to choose would be ‘Heros’.
Name the file suitably and include the .tif file extension, if you prefer you can export 16bit .png’s amongst other image formats that support 16bpc.
Tick the box ‘Add the padded frame number as suffix’, this will give sequential file numbering onto the end of your filename set earlier, seperated with a – hyphen. Just as an aside if you intend importing the image sequences into an NLE such as kdenlive a hyphen rather than underscore must be used.
Tick ‘Show Progress’. Click OK, after a minute or so your 16bit hero frames should land in the folder chosen and ready for import into whatever application you intend ‘grading’ your video whether that be Darktable, Blender etc.
Stage 2 – 16bit Output & Scene Splitting
Once the ‘Hero’ frames are exported, unhash the 8bit line “return source#Gives us 8bit. Hash the line out for 16bit.”, so that we prevent the 16bit processing for now, if you added the histogram option you could drop a hash in after “return source” to prevent luma waveform being drawn.
In AvsPmod go to the menu and choose Video -> Clear All Bookmarks.
We’re now looking to export our movie to an image sequence or preferably sequences, with the option to put each sequence of images into a named ‘Scene’ sub folders because we will undoubtedly be dealing with hundreds of frames per scene, thousands of frames in total, for manageability, file manager performance sub dividing really helps.
With the movie loaded in AvsPmod once again create new bookmarks at the points in the movie where you’d like to do a scene split for possibly applying a different grade or ‘look’.
This time however your bookmarks need to be in pairs, like an ‘in’ point and an ‘out’ point, so if you don’t want to loose any frames in the 8 to 16bit export then bookmark the first and last frame of each ‘Scene’, the result will be two bookmarks next to each other like frame 235 (Last frame of Scene) & 236 (First frame of next ‘Scene’) and so on. This should also give you a bookmark on the first frame and the last frame of the movie.
If you want your Scene sub folders to be suitably named, (the default will be sequential numbers) then we use AvsPmod’s Bookmark Titling Tool in the menu Video -> Titled Bookmarks -> Title Bookmarks (Manual). A box will appear on the screen showing all the bookmarks ready for naming, but we only need to name the ‘in’ bookmark for each scene pair.
Take care here, the dialogue is a little flaky, make sure before you start to expand it wide enough to view properly, particularly if you’re using AvsPmod with Wine on Linux.
When bookmark titles have been completed we’re nearly ready to Pipe To RGB again, but first add that hash back to the start of the 8bit line “#return source…..” to turn the line green and diable it so the 16bit processing steps after are activated.
RMB in the preview window space and choose, “Release all videos from memory”, preview will become blank.
In the Macro menu choose, “Pipe RGB To Imagemagick” and we need slightly different settings ticked.
Particularly untick “Include only bookmarks, if any”, we no longer just want ‘hero’ frames and tick the “Include only the range between bookmarks, if any”, this option does also include frames on bookmarks, so if bookmarks have been paired correctly as described previously, all frames should export.
The Imagemagick arguements remain the same. But choose a new folder for exporting ‘Scene’ sub folders to, if following the Project Folder setup then this will be the ‘Scenes’ folder.
Name the file suitably, I find Frame.tif ok for this, again the sequential frame number with hyphen will be added to become Frame-001.tif etc.
We keep the boxes ticked for “Add the padded frame number as suffix” and “Show Progress”.
Tick the box for “When using bookmarks save every range to a subdirectory”
When ready, click OK, after a short while the macro’s progress window will appear and show you how many batches based on the ‘step’ value you chose have been calculated and an estimated time will appear based on the time taken processing the first batch, depending on processing power this could take a while, in the meantime we can create the ‘Looks’ ready to apply to the 16bit frame sequences when export is complete.
For example: http://blendervse.wordpress.com/2013/03/21/darktable-for-video/
For final encoding we have numerous options such as:
Export graded 8bit images from DT and encode directly via ffmpeg to a suitable video codec. Or include minor additonal 8bit processing via Avisynth such as conversion to YCbCr, luma sharpening and adding grain.
Export graded 16bit images from DT and import into Blender’s VSE for adding wipes, transitions, titling, tweaking grade, compositing, adding sound tracks including Jack Transport sync with DAW’s such as Ardour and encode out via Blender’s ffmpeg interface or Frameserve RGB from Blender to Avisynth for conversion to YCbCr, luma sharpening, adding grain, dub audio and final encode.