BYE Histogram question

[Phil Leigh]

Guylain,

I know you use the aforge.net library to compute the histogram. Does that code take into account the difference between 12 and 14 bit cameras when displaying the curve relative to the axis? I guess it must do but I want to be sure.

I'm finding that even curves which appear very far to the right on the BYE display are actually quite low in levels when opened in post-processing software such as PixInsight.

This makes it hard to set the correct exposure for flats.
many thanks
Phil

[jcinpv]

My last set of Flats was taken by letting BYE show me the histogram setting. So, I went out to take another set of Flats. This time doing it manually on the camera (60D) with the histogram near the center. The only difference between the two results in the unlinked stretched output (for PixInsight users, this will make sense) is that the latter appears to be a little brighter but is not significant in any way. Can you see any real difference?

 

Old:

 

 

New:

 

[Phil Leigh]

Nope..,they are both somewhat underexposed. The BYE histogram and the camera histogram are both equally ineffective for flat exposure assessment. You need to get the histogram as far right as you can.

[jcinpv]

Phil: That's very interesting that you say it's underexposed. Here's what the RAW image looks like without any processing (screen capture):

 

 

Would you still call that underexposed?

[jcinpv]

In almost every description I read about taking Flats, there is missing detail. The only thing consistent is to put the camera in AV mode and let the camera figure out the exposure time. In one place, I read that the aperture value should be wide open (lowest value), then adjust the time value to get the histogram centered. In another, it says to ensure that the camera stays on the telescope with the same focal length and orientation as with Lights (same orientation? really?). The method of taking Flats varies from 1) aiming the uncovered telescope at the sky at dusk or dawn when the sky is evenly illuminated (just exactly which shade of blue is best?), 2) aiming at a computer monitor with a white screen (hope you have a light-weight LED monitor for that), and 3) placing a T-shirt over the aperture and shining a white light into it (supposed only an LED light - it would have to be a big one for a 12" aperture).

 

So what's wrong with covering the aperture with a T-shirt and aiming the telescope at the Sun? Can't get any better 'white light' than that!

 

What seems most critical to me in this process is that the aperture should be wide open and the time value adjusted to place the histogram where you want it. So putting the camera mode in Manual, setting the aperture wide open first, and then adjusting the exposure seems to be the best method.

 

Also, the position of the histogram has to be an important part of the process. The person who said to put the histogram in the center was a professional photographer also working the astronomy angle. That lends a great deal of credibility to his statement.

 

Yet another photographer said to get as many Flats as possible, and 100 is better than 50 in his experience.

 

To say that the histogram has to be as far right as possible seems to me to overexpose the image. If that is done, it also seems to me unlikely that any dust motes on the sensor will show up. Something for me to experiment with. If I get the dust motes, as I do when the histogram is centered, then it seems to me I've done the right thing. I'll be back after getting some more Flats using various methods.

[s3igell]

Flats are only Valid if they are taken with the Same Optics Setup as the DSO Images - same Focus, same Filters, same Orientation, same EVERYTHING except the Light Source.

 

"Wide Open" has no meaning in AP Imaging with a Telescope - it is ALWAYS "Wide Open".  However, when Imaging through a Camera Lens, one must keep the same Aperture Setting as the Images were taken else the Lens Distortion and the Vignetting and even the FOV will be change - invalidating the Flats.

 

You will want a decent number of Flats (20-30), but after that the statistical averaging will invoke the Laws of Diminishing Returns.

[jcinpv]

s3igell: You're right. I forgot that when the camera is Prime Focus on a telescope, aperture value is meaningless. Time value is the only factor, aside from where the histogram lies - left, center, or right. I'll have to take another look at the logic given for doing a 100 Flats. I know very little about math where photography is concerned. So I guess only my experiments will convince me. Science was established on duplication and replication. Thanks for the knock on the head. <grin>

[Phil Leigh]

Thanks for the quick reply Guylain... yes it makes sense.
I think what I am seeing is the consequence of the rescaling of the x-axis to 8-bit (from the actual 12 or 14-bit data in the image).

This would be representative if 128 (in the 8-bit histogram display) equalled 8192 in the 14-bit image data... but I'm not convinced that it does. I'm seeing 128 in your display (halfway across) looking more like ~1,000-2,000 in the actual 14-bit data.

This is leading my flats (usually exposed so the BYE histogram is halfway) to be too low in level. If expose them so they are way over to the right on the BYE histogram they are much better.
I wonder what your view on this is?

I think halfway on your histogram should be 8192 for 14-bit cameras and 2048 for 12-bit cameras...
kind regards
Phil

 

[admin]

Keep in mind that I only load the JPG representation of the raw data, which is in 8 bits format.  I can get away with this because I don't do any processing to the image... so the 8 bits jpg representation actually speed things up when loading the image... and keeps memory/cpu consumption down.

 

Guylain

[Phil Leigh]

In almost every description I read about taking Flats, there is missing detail. The only thing consistent is to put the camera in AV mode and let the camera figure out the exposure time. In one place, I read that the aperture value should be wide open (lowest value), then adjust the time value to get the histogram centered. In another, it says to ensure that the camera stays on the telescope with the same focal length and orientation as with Lights (same orientation? really?). The method of taking Flats varies from 1) aiming the uncovered telescope at the sky at dusk or dawn when the sky is evenly illuminated (just exactly which shade of blue is best?), 2) aiming at a computer monitor with a white screen (hope you have a light-weight LED monitor for that), and 3) placing a T-shirt over the aperture and shining a white light into it (supposed only an LED light - it would have to be a big one for a 12" aperture).

 

So what's wrong with covering the aperture with a T-shirt and aiming the telescope at the Sun? Can't get any better 'white light' than that!

 

What seems most critical to me in this process is that the aperture should be wide open and the time value adjusted to place the histogram where you want it. So putting the camera mode in Manual, setting the aperture wide open first, and then adjusting the exposure seems to be the best method.

 

Also, the position of the histogram has to be an important part of the process. The person who said to put the histogram in the center was a professional photographer also working the astronomy angle. That lends a great deal of credibility to his statement.

 

Yet another photographer said to get as many Flats as possible, and 100 is better than 50 in his experience.

 

To say that the histogram has to be as far right as possible seems to me to overexpose the image. If that is done, it also seems to me unlikely that any dust motes on the sensor will show up. Something for me to experiment with. If I get the dust motes, as I do when the histogram is centered, then it seems to me I've done the right thing. I'll be back after getting some more Flats using various methods.

 

 

1) The light from the sun is not white. Far from it. Especially when it passes through layers of atmosphere at twilight or near dawn. However the precise colour balance of the light is not too critical

 

2) You need to read the other posts here about the histogram - carefully. The crucial thing about flats is that the red, green and blue channels all need to have a strong enough (but not identical) level. The Histogram in BYE (which is constrained by the Canon SDK) and on the back of the camera does NOT show you the true levels of the RAW 14-bit data. It shows you whether or not your in-camera jpeg is broadly under or over exposed. For astrophotography  - and especially flats - this is irrelevant. What you need is a strong level in the RAW data, not a pretty jpeg. This means getting the histogram over to the right. To properly analyse the data in an AP context requires proper software tools designed for the job - and most of what you need to know for terrestrial photography at any level of accomplishment is simply not important or downright wrong when it comes to using DSLR's for AP. 

 

3) If you can't be bothered to do that, then just set the camera to AV mode and set exposure compensation to +2EV, With a decent light source you will get flats that are almost certainly usable.

 

4) 100 flats is TOTALLY unnecessary. 25 is fine. 100-200 bias frames are absolutely required if you want the lowest noise - but they can be shot ahead of time.

[Phil Leigh]

Phil: That's very interesting that you say it's underexposed. Here's what the RAW image looks like without any processing (screen capture):

 

 

Would you still call that underexposed?

 

You cannot tell by looking at the image visually - you need to measure the image statistics in PixInsight. In 14-bit mode (I presume you have a 14-bit camera?) you are looking for a mean value in the region of 4000-8000 (although you can probably get away with 2000+)

[Phil Leigh]

Ah.. I see... So the histogram is derived from the JPG data - which is pre-scaled to 8-bit and (presumably) has white balance, sharpening and a non-linear stretch applied too?

The fog is starting to clear in my brain...

I've been doing a lot of reading on this and it seems that this difference in the way histograms look when derived from JPEG vs RAW data is well known about (except by me until now!) and when shooting RAW - which we are - we should ALWAYS expose "to the right" on the jpeg histogram to maximise the use of the RAW 12/14-bit data range...

So flats should be exposed so that the BYE histogram is as far to the right as possible without actually clipping the right hand margin.

CMOS sensors in modern DSLR's have extremely flat linear response so provided there is no clipping, the flats will work very well.

I'm going to prove this next time I go out.
Many Thanks Guylain!

[jcinpv]

Phil: Thanks for the feedback. I understand now. But about Bias frames, in other places I have read that taking Bias with a Canon is unnecessary. Every little bit that comes out here seems to bring up another conflict with what I've read elsewhere.

 

I'm going to wait until my 60Da with clip-in LPS filter arrives this week before continuing to acquire images.

[admin]

Ah.. I see... So the histogram is derived from the JPG data - which is pre-scaled to 8-bit and (presumably) has white balance, sharpening and a non-linear stretch applied too?

 

Yes, this is correct.

 

Guylain

[Phil Leigh]

Actually Bias frames are critical but at least they work properly. Dark frames are the real problem. They are a nightmare with Canon DSLR's thanks to some internal processing that happens before the RAW file is created.

 

If you use PixInsight or DSS, stick to one ISO(800 for the60D) and take 200 bias frames to create a "superbias" master. As you camera is not cooled you will need darks but you must make sure that the ISO, duration and temperature match as closely as possible.

 

 

[jcinpv]

Thanks Phil. I will be using the 60Da starting later this week. Thanks for the suggestion on the ISO. I use PixInsight exclusively for processing (at the bleeding edge of learning how to use it), BYE for imaging, and PHD2 for guiding. When imaging, I take 5 Darks every 25 images. Images are typically between 30s and 120s depending on the target. In one article I read, the sweet spot for ISO for the Canon was given as 1100. I've been tinkering with different levels from 400 to 3200. Lower levels mean longer exposures, of course. My last run was at 1000 for 120s. Kind of noisy.

[Israel G.]

That's good news Guylan...  I'm also having problems taken the correct flats with Ha filter.

Looking forward for this implementation.

Thanks and regards.

Israel.

[Phil Leigh]

Unfortunately for some reason your image attachments are not visible. Flats must be taken at at the same ISO as the lights they are correcting.

You have to use exposure duration to alter the exposure.

 

 

 

[jcinpv]

Argh! Phil, you're right! I didn't notice that the images weren't showing up. Doesn't matter. I've got my 60Da installed now and had to take a whole new set of Flats and Bias. Does the ISO setting apply to Bias as well?

[s3igell]

Flats must be taken at at the same ISO as the lights they are correcting.

 

Not necessarily true...

 

While it is "best" to take Flats at the Same ISO as everything else, there is a lot of general acceptance for "Flats at whatever ISO allows one to get a suitable mid-level Exposure and no evidence of Shutter shadows".  This is especially so for DSLRs (or any modern sensor), where any ISO-related elements of Bias Current is rather well controlled, and where Flats Exposures are so short that little-to-no Dark Current Noise has time to accumulate.  And where higher ISOs have been used for the Image Exposures, even modern Curtain Shutters have issues maintaining "perfect" control at the shortest high-speed Shutter Speeds that would be required of a Flat at such ISOs.

 

It was proper to be concerned about the other factors when Amateur Astronomy CCDs were still struggling with Bias and Dark Currents and non-linear ADCs and other "Issues" that a further decade of Research and Design have worked to tame.

It is much more important to be concerned with sufficient Exposure to properly capture Vignetting and Dust Motes, as the rest is basically Math that trends to Zero under most DSLR Imaging Conditions.

[jcinpv]

Here's my latest attempt at M3. All Flats, Bias, Darks, and Lights were taken at ISO 800. Canon 60Da. Imaged with BYE, guided with PHD2. This is only partially processed with PI.

 

[jcinpv]

Now, regarding the histogram question, here's what I get using PI Statistics on my last set of Flats with the Canon 60Da (LPS filter installed):

 

ISO 800, 1/100s, Av.

 

The setup is with a double layer of T-shirt over the aperture and using an LED panel of lights covered with a diffuser.

 

Acceptable?

[Phil Leigh]

Yes those levels are fine for your flats.

Regarding ISO... I have been doing a lot if work this week on the PixInisght Forum regarding this with several real experts.

The bottom line is this... To correctly calibrate flat frames from DSLR's it is vital to use a good master bias... And the ISO of the bias and flat frames MUST match, as must the ISO of the light and dark frames. In Part this is because of the pedestal based processing that Canon are doing in-camera using the optical black overscaan region of the sensor in conjunction with internal lookup tables that shift the pedestal(bias point) according to ISO and exposure duration. A side effect of this is that Canon darks so not scale properly (somewhat off-topic I know).

Bottom line: IF you want the best results, use the same Iso for everything, use a master bias made from 200 bias frames and if you have a super cooled camera, skip darks entirely, otherwise shoot darks exactly the same duration and temperature as the lights.

.. And ALWAYS dither using a reasonably large dither scale.

[s3igell]

Keep in mind that I only load the JPG representation of the raw data, which is in 8 bits format.  I can get away with this because I don't do any processing to the image... so the 8 bits jpg representation actually speed things up when loading the image... and keeps memory/cpu consumption down.

 

While this JPG Histogram approach makes perfect for "imaging" purposes, where throughput is quite important and the usual use of the Histogram is to confirm one has separation from the Left edge, could you consider (as a future Feature) an additional routine (perhaps managed by a Settings Check Box) that performs a more extensive RAW Histogram Evaluation when the Imaging Frame Type is "Flats" ??

(I'd suggest triggering it only for "Flat Previews", but those are only L-JPGs...)

[dts350z]

And ALWAYS dither using a reasonably large dither scale.

 

Would you or someone please post suggested dither settings?

 

I had problems with DSS aligning when I used the defaults in BYEOS, however that was for an older scope where the focus wasn't near as good as what I'm using now.

 

Thanks,

Z

 

 

[admin]

While this JPG Histogram approach makes perfect for "imaging" purposes, where throughput is quite important and the usual use of the Histogram is to confirm one has separation from the Left edge, could you consider (as a future Feature) an additional routine (perhaps managed by a Settings Check Box) that performs a more extensive RAW Histogram Evaluation when the Imaging Frame Type is "Flats" ??

(I'd suggest triggering it only for "Flat Previews", but those are only L-JPGs...)

 

Yes Phil, I already have that on the new feature backlog.

 

Thank you

 

Guylain

 

 

[admin]

And ALWAYS dither using a reasonably large dither scale.

 

Would you or someone please post suggested dither settings?

 

I had problems with DSS aligning when I used the defaults in BYEOS, however that was for an older scope where the focus wasn't near as good as what I'm using now.

 

Thanks,

Z

 

 

Leave the default in BYE until you are comfortable and fully understand dithering.  The defaults work pretty well as-is

 

 

[dts350z]

Well since I changed them may we know what the defaults are?

 

Thanks (and sorry for the slight thread hijack)

[admin]

Defaults are...

 

 

[s3igell]

Keep in mind that I only load the JPG representation of the raw data, which is in 8 bits format.  I can get away with this because I don't do any processing to the image... so the 8 bits jpg representation actually speed things up when loading the image... and keeps memory/cpu consumption down.

Guylain

 

Guylain, do you know whether the embedded JPG from which you derive the displayed Histogram is internally processed by the Canon DSLR to the extent that any applicable White Balance is applied ??

 

While most of us who correct for heavy tints introduced by Ha-modded Cameras or strong Light Pollution Filters are more likely to shoot RAW and apply the CWB equivalent during Post-Processing via Levels or Stretch Functions, some users might use a Custom White Balance during AP Imaging in order to attain the same in their straight-from-Camera JPGs.  

For those using the RAW workflow, the Histogram is already only a "loose guide" due to the vagarancies discussed in Phil Leigh's posts above.  And other AP Image Tools will be used to evaluate the true RAW Histogram.

But, for those without such tools, one of the biggest uses for the BYE Histogram is during Flats capture.  And while Flats by nature don't require "perfectly aligned R-G-B Histogram Humps", those humps do need to be kept in the appropriate middle-ground as to not indicate Clipping.  It would seem that if the In-Camera CWB is already applied, then those humps would be even further distorted from any representation of the original RAW data.

 

Am I on the right track ??  Is this, too, something that you've already thought of for BYE v3.1/3.2 ??

[admin]

Guylain, do you know whether the embedded JPG from which you derive the displayed Histogram is internally processed by the Canon DSLR to the extent that any applicable White Balance is applied

Yes, I believe so.

 

 

Am I on the right track ??  Is this, too, something that you've already thought of for BYE v3.1/3.2 ??

What I would like to do in future release is actuallt put that thing to rest and load the linear raw histogram.  Not where that fits in the release plans yet.

 

Regards,