My first attempt at the Magic Lantern HDR feature recently released by the awesome ML team here: http://magiclantern.wikia.com/wiki/Unified>
This first attempt follows the interpolate in between frames approach but since considering it further and the fundamental problems with this method and the fact that all the motion has been captured its exposure and color that need constructing. To make use of the potentially wider dynamic range the output its assumed a bit depth greater than 8bit is required and possible as seen in a recent post.
But the following is a Interframe based method for now.
Really following on from my last post regarding 16bit Scene referred linear .EXR’s and 10bit lossless 4:4:4 h264 from video shooting Canon DSLR’s.
First some files as to where I’m at:
Basic output 1920×1088 8bit 4:2:0 h264 file: [Save As]
http://www.yellowspace.webspace.virginmedia.com/merge.mp4
A 16bit .tif frame:
http://www.yellowspace.webspace.virginmedia.com/IM-hdr_000000.tif>
A 16bit linear light .exr frame:
http://www.yellowspace.webspace.virginmedia.com/IM-hdr_000000.exr>
The process and tools is basically the same as here:
8bit Video to 16bit Scene Referred Linear .EXR’s>
With the addition of a great frame doubling/interpolation tool called Interframe for AVISynth:
http://www.spirton.com/interframe/>
And as always the excellent support from the Doom9 community.
Particularly -Vit- (QTGMC) & cretindesalpes (Dither Tools). The Interframe referencing lines in the script below is courtesy of -Vit- word for word, although not his tool 
. Disclaimer this does not negate the need to read the Interframe documentation.
So using the sample Canon MOV file provided by the RedKiteMedia, log in to download native file:
And this script, which is work in progress and needs work.
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\mvtools2.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\ffms2.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\fft3dfilter.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\removegrain.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\mt_masktools-25.dll”)
LoadPlugin(“c:\Program Files\AviSynth 2.5\plugins\dfttest.dll”)
Input=ffmpegsource2(“77506626.mov”)
Assert( Input.FrameCount() >= 3, “Need at least 3 frames!” )
num = Input.FrameRateNumerator()
den = Input.FrameRateDenominator()
# Interpolate even and odd frames separately to source rate
A = SelectEven(Input).InterFrame(FlowPath=”c:\Program Files\AviSynth 2.5\plugins\”, Preset=”Placebo”, NewNum=num, NewDen=den)
B = SelectOdd(Input).InterFrame(FlowPath=”c:\Program Files\AviSynth 2.5\plugins\”, Preset=”Placebo”, NewNum=num, NewDen=den)
# Trim off end frames for which an interpolated frame cannot be calculated for the other exposure
A = (Input.FrameCount() % 2 == 0) ? A.trim(1,0) : A.trim(1,A.FrameCount()-2)
B = (Input.FrameCount() % 2 == 0) ? B.trim(0,B.FrameCount()-2) : B
src1=A
src2=B
# Blending amount for the first clip
bl = 0.75
bls1 = String ( bl)
bls2 = String (1 – bl)
# 8-bit clips converted to linear 16-bit full range (gamma undone)
ug = ” 16 – 0 max 1.41624 / 2.2 ^ “
# Redo the gamma, result in 16 bits YUV
rg = ” 0.454545 ^ 362.5585 * 4096 +”
# Blend
Dither_lutxy8 (src1, src2, expr =”x ” + bls1 + ” * y ” + bls2 + ” * + 256 *”, yexpr=”x” + ug + bls1 + ” * y” + ug + bls2 + ” * +” + rg, y=3, u=3, v=3)
#Dither_convert_yuv_to_rgb(matrix=”601″, tv_range=false, cplace=”MPEG2″, chromak=”bicubic”, lsb_in=true, output=”rgb48y”)
#Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false, curve=”709″)
#Dither_convey_rgb48_on_yv12 (SelectEvery (3, 0),SelectEvery (3, 1),SelectEvery (3, 2) )
Dither_convey_yuv4xxp16_on_yvxx()
There are things to tidy up in the script, such as a reverse rec709 curve to linearise as per previous .EXR workflow rather than the sRGB 0.45 ‘degamma’ in the script.
Although blending is done linear light and creating a 16bit output, there’s room for a lot of improvement. I question the ‘merge” method compared to enfuse or Tufuse, but kind of feel this is more a LCE / Local Adaptiation process rather than a ‘traditional’ HDR one. HDR from only two exposures?, not really. Compared to a bracketed set. Where as Local Adaptation / Local Contrast Enhancement may provide more natural results, scroll down to the “Concept:Tonal Hierarchy and Image Contrast” here:
http://www.cambridgeincolour.com/tutorials/high-dynamic-range.htm
A way of merging the two luma planes into one 16bit levels range and whether to merge both chroma or just one from decent exposure, haven’t bothered with audio currently either.
Save the script as an .avs and then use AVS2yuv or AVS2pipemod along with a 10bit build of x264 or a 16bit hdri build of Imagemagick depending on output required. All as per previous post:
8bit Video to 16bit Scene Referred Linear .EXR’s>
Although I’ve loaded the latest Magic Lantern Firmware still haven’t had chance to test it yet. Awesome work by the Magic Lantern Team.
Yay, HDR video for the masses! The real issue is not the combining of levels but the combining of frames seperated by 1/fps of a second. Can Blender export 16bit tiffs from h264s then use them in wide colour space internally or is the VSE still crippled?
Depends on how you look at it, the source is a progressive stream regardless of the flicker all the motion is there in the file why interpolate something thats a given, when what is really happening is a screwed up exposure in reality.
Going back to the initial release of the 550D the firmware had a bug that meant every so many frames the exposure would jump sporadically the motion was fine and a relatively simple luma equalisation smoothed the exposure, the ML HDR feature is very similar but easier as the exposure jump is controlled / known.
Optical flow is complex and prone to error, dealing with exposure ‘issues’ less so even though the target is to produce a wider dynamic range from the two exposures, not smooth motion.
Just my opinion though.
With regard to 16bit tiffs, no blender can’t produce 16bit or 10bit levels range from 8bit video sources, the workflows with Avisynth use a denoiser to generate additional data that is then stacked back into the file, not a true 16bit file as if captured in camera but the additional data and levels range gives more to grade against.
Color space is unchanged whether its 8bit, 10bit or 16bit or HDR, the width of the gamut is the same, rec709 / sRGB defined by the color primaries in camera in relation to CIE 1931.
However hopefully using an 8 to 10/16bit workflow and the benefit of avoiding conversion to RGB at 8bit will provide a wider dynamic range from the two exposures by interpolating them into a 10/16bit levels range. Rather than exporting as jpegs, using enfuse and tonemapping.
Early days and who knows what will follow.
About that 550D bug, I never read about it? Is that a stills issue? I have seen this in timelapse but wondered if it was auto iris related. Then on my HDR interior shot I noticed that the white clip bounced up and down regularly. I just thought that it was an issue with the Highlight Tone priority.
hi, video as well I think, I certainly have gigs of 550D video files from before the firmware update with jumping exposure, which may have been caused by the bug. Cheers. Great new posts by the way.
Thanks, my posts are just sweets to your meat and three veg meal posts. I applaude your detailed testing of the color spaces.
I thought that ML only worked on the earlier formware (or is that the 5D II) what firmware are you running? I was pretty sure I was up to date as far as I could be.
Hi yellow,
considering that the latest ffmpeg version (v0.10) supports 10 bit quicktime encoding and decoding, I’m just wondering if is it possible to do all or part of this from Blender?
Btw, I’ve just tried the latest ffmpeg to convert 550D footage to DNxHD but in all fairness, I cannot tell the difference between current and older versions. File sizes are the same and the scopes appear to be almost identical. But there’s a noticeable difference in scopes when comparing original and DNxHD footage:
https://rapidshare.com/files/1577853855/scopes_original.jpeg
https://rapidshare.com/files/2114406545/scopes-DNxHD.jpeg
Any thoughts?
hi, yes your scope images illustrate a squeezing of the luma into the 16 – 235 range going from Canon native h264 to DNxHD or any other codec for that matter. Evident in the horizontal gaps to the waveform and combed histogram where levels have been stretched back out in Blender, shadows crushed and highlights clipped.
The problem with using FFmpeg to handle the complete transcode of Canon h264 source rather than simply decompressing and then do the transcode via another method is that it squeezes the luma.
If you were to do the complete transcode with FFmpeg with other cameras h264 then the same issues do not necessarily occur.
The thing that screws up FFmpeg’s handling of native Canon h264 is that in the header of each Canon file has a flag which marks the source as fullrange ‘on’, because of this FFmpeg interprets the source as yuvj420 rather than yuv420 and in any conversion squeezes the luma, like a typical handling going from RGB to YCbCr. The idea that RGB 0 [Black] goes to 16 in YCbCr and RGB 255 [White] goes to 235 in YCbCr for correct preview at the playback device, DVD, Bluray whatever.
Transcoding ‘native’ Canon h264 via FFmpeg to a lossless codec is not a lossless conversion because of this handling.
As Blender relies on FFmpeg for video handling without any user intervention (something that needs to change in the ‘color management’ improvements slated for Mango), the yuvj420 designated source is imported into Blender and assumed full range therefore there’s no need to mess with the luma levels to preview in Blender correctly, ie: as if it was RGB as Blender handles it all RGB internally, they are already assumed 0 to 255, but try rendering out of Blender full range and it’s impossible even to that dodgy lossless h264 addition in the encoding panel, as FFmpeg once again does the typical full levels, as if RGB to YCbCr restricted levels output.
It could be argured that that’s exactly what should happen because DVD, Bluray etc are 16 – 235 range by specification and so are many web players, some web playback methods handle full levels like flash but that’s a different discussion. If however Blender is not the last tool in the chain then really there should be user control in order to maintain levels through from start to finish.
There is another implication in transcoding to other codecs like DNxHD and that is to do with the Colormatrix declared in the header to the source file. Native Canon h264 files use BT601 not BT709. BT709 is the default choice by majority of video handling tools for HD sources when the colormatrix is not declared in the header, based on pixel count in it’s absense. This only matters to the YCbCr conversion to RGB and that happens for preview in Blender and all the image processing nodes.
When transcoding there is a high chance that important bit of meta data is lost, as a result the wrong colormatrix is assumed and suddenly not only has contrast increased, shadows crushed, highlights clipped from bad levels handling but the wrong colormatrix gives pinks/reds in Canon native files an orange appearance, so the starting point to do color corrections, adjust white balance, contrast and finally grading are from a skewed perspective not representing what was actually shot in camera and encoded to h264.
Finally there are a number of simple ways round FFmpegs initial handling of native Canon files, a tool called MP4Box which allows us to quickly reset the fullrange flag to off and remuxs into an mp4 container. FFmpeg then sees the flag is off treats the source as 4:2:0 and happily transcodes to all the usual codecs including lossless ones leaving the levels where they were shot. The BT601 Colormatrix can be declared as well just in case the remux looses it.
BUT then the problem with import into Blender is that as the source is not treated as full range luma it gets stretched out to RGB levels, so histogram gets combed and waveforms get the horizontal gaps. So what if it’s just the scopes showing a representation of what we’d expect to see at final playback, problem is it’s also encoded that way with squeezed luma levels, same comment applies as above re; playback vs further tools in the chain.
With blender compiled with latest FFmpeg I’d assume so. FFmpeg has had 10bit decoding support for certain codecs like h264 for probably 12 months now and I’ve imported 10bit lossless h264 from the Avisynth route with success and there’s definetly improved scope for color manipulation in the nodes with the 10bit sources from the Avisynth route as it’s not just a 8bit to 10bit / 16bit conversion but additional data is included, where as I’d assume simply transcoding Canon files to 4:4:4 10bit isn’t going to be the same, tests will tell.
As for transcoding to 4:4:4 10bit with FFmpeg, we appear to have the opposite levels handling of the description in the post below, now we’re discussing 4:4:4 10bit. So as Canon h264 is considered full range yuvj420 by FFmpeg the transcode to 4:4:4 leaves levels as the source, I guess treating 4:4:4 as RGB levels, where as setting the full range flag to ‘off’ as described below now causes FFmpeg to squeeze the luma levels transcoding to 4:4:4.
To get encoding options for 4:4:4 will need blender interface updating and once again encoding out from 4:4:4 to 8bit YCbCr squeezes luma levels from full range to restricted range from the quick tests I’ve done.
Ok, I’ve made a quick test to illustrate my previous comments on Canon MOV handling, here’s a zip to a full range round trip test, there’s a text file describing contents within the .zip
http://www.yellowspace.webspace.virginmedia.com/fullrangetest.zip
A couple of gotcha’s in Blender. Encoding to lossless HuffyUV appears to be RGB rather than YUY2 and the ‘lossless’ h264 option only visible when choosing the h264 preset first in Blenders encode panel appears not to be lossless at all with Canon DSLR MOVs.