Discussion on various topics between R. Gran, M. Yokoyama, and S. Andringa
from December 2005.

I have formatted so that the thread of the discussion is more clear.

There are other substantive discussions from Summer 2005.  I'll post them soon.

MY> For the momentum scale, what I have requested (or try to) is:
MY> "Please demonstrate that you can RELIABLY estimate the UNCERTAINTY of the 
MY> momentum scale using neutrino data."

MY> In the spectrum measurement, we carefully checked that the CENTRAL value of 
MY> the systematic parameters were reasonable (within externally estimated 
MY> uncertainties), but not so careful about their errors, because they were not 
MY> used in the oscillation analysis.  This "measured" momentum uncertainty is under 
MY> assumption of a particular set of neutrino interaction model, and only accounts 
MY> for the "statistical" error.  There should be other "systematic" sources that 
MY> adds uncertainty on the momentum scale. One such example is the correlation 
MY> to M_A itself, as you already studied.  How about effects of other sources, 
MY> for example QE/non-QE ratio, M_A for CC1pi, etc?

MY> For oscillation analysis, these systematics have very little effect 
MY> on the oscillation parameter measurement with statistical precision of K2K.
MY> In coherent pion analysis with SciBar, we were concerned about low-q2 region, 
MY> where angular resolution dominates the q2 uncertainty.  We also refit the q2
MY> distributions with various neutrino interaction model parameters and 
MY> estimated the effect.  However, M_A measurement is very much sensitive 
MY> to this parameter as you write and we need more careful assessment of it.

MY> For convenience, I quote my past comments on this issue below.
MY> On August 8:
MY>>   However, I am not yet convinced we can really constrain this parameter
MY>>   from the neutrino data. Many systematic effects that may affect the 
MY>>   momentum scale may not be considered yet.
MY>>   For example, spectrum shape in higher energy region should have correlation
MY>>   with MRD energy scale.  In the spectrum analysis, this correlation gives 
MY>>   relatively small uncertainty to neutrino energy spectrum, but for M_A, 
MY>>   small change of the momentum scale causes large uncertainty, and I 
MY>>   am not sure if we really understand it with currently quoted precision (1%).
MY>>   (In addition, there may be effects from neutrino interaction uncertainty
MY>>    other than QE.)

Rik>>  The momentum uncertainty is the largest single systematic effect in the 
Rik>>  MA analysis, so the estimate of its uncertainty is important.  We 
Rik>>  actually have three separate estimates for what this might be.
Rik>>  The basic estimate, without any fitting, is 1% from MRD material assay, 
Rik>>  1.7% from GEANT range and dE/dx, and 0.5% from the neutrino interaction 
Rik>>  model.  Taken in quadrature, this gives 1.5% uncertainty.  Traditionally, 
Rik>>  K2K (SciFi and SciBar) has taken the linear sum of the first two and 
Rik>>  reported 2.7%.

Rik>>  The second estimate is the one reported from the SciFi fit.  The reported 
Rik>>  uncertainty in this parameter, when taken from the spectrum fit, already 
Rik>>  includes correlations with nQE/QE and the other standard SciFi systematic 
Rik>>  errors.  CC1pi?  I have not checked this.  For SciFi, I have taken care 
Rik>>  that the estimate includes LG density for K2K-I, and also the correlation 
Rik>>  with QE-MA.  These estimates have been done by hand, and taken in quadrature 
Rik>>  with the reported error from the fit.  The result is equivalent to 
Rik>>  approximately 1.7% error on the central fit value, if the SciFi data had 
Rik>>  been from a single data-set with no Lead Glass.

Rik>>  The final estimate is the pure statistical error in this measurement, 
Rik>>  which is better than 1%, and can be estimated from the width of the MC 
Rik>>  true-rec muon momentum distribution.

Rik>>  In summary, the apparent error in this value already includes most 
Rik>>  correlation with other known systematic parameters, seems to be reasonably 
Rik>>  between the overly conservative 2.7% and the pure statistical error.  The 
Rik>>  fact that the MA fit and SciBar all give a very similar value gives some 
Rik>>  confidence that this error is not overestimated. 

MY>>>  However, we assign additional 20% uncertainty to the error estimated by 
MY>>>  fitting.  So, is the uncertainty in the fit really enough?

Rik>>>> This additional uncertainty (in nQE/QE) is assigned after the spectrum 
Rik>>>> fit for input to the oscillation analysis.  Within the spectrum fit (at 
Rik>>>> least when I run it for SciFi) it is unconstrained, so the resulting 
Rik>>>> error and correlation in pscale includes the full possible range of nQE/QE.

Rik>>>> Incidentally, the spectrum fit is actually binned in pmu, so the 
Rik>>>> correlation with other systematic parameters is quite small, and the 
Rik>>>> resulting constraint on pscale is very nearly the best we can do. 

MY>>>>> Yes, the 20% uncertainty is added AFTER the spectrum fit, because it 
MY>>>>> is NOT addressed by out fitting procedure. Nevertheless, we quote this 
MY>>>>> enlarged error as the "nQE/QE" uncertainty for the final (=oscillation) 
MY>>>>> analysis, so I think it is the final result of the near detector analysis.
MY>>>>> As this additional uncertainty (which comes from discrepancy b/w near 
MY>>>>> detector results) is not taken into account in the fitting,
MY>>>>> I am afraid just quoting the uncertainty from the fitting may not be 
MY>>>>> sufficient. If the effect on the p_mu scale is proved to be small, 
MY>>>>> it is fine.

On another related topic:

MY>>On August 19:
MY>>   - Another question about muon momentum scale.
MY>>   If the momentum scale is changed, Q2_rec changes accordingly,
MY>>   which leads to migration of events near Q2_rec cut threshold.
MY>>   How do you treat this?
MY>>   I remembered this migration prevented us from using Q2_rec as
MY>>   cut variable when we performed spectrum fit in 2004.
MY>>   (theta_mu gave us much smaller effect and we used small angle cut)

MY> Here I meant when you vary the momentum scale, some events moves up/down 
MY> beyond Q2 threshold and number of events over Q2 threshold may change. 
MY> I wanted to know how you treated such events.


Rik>>  In the MA analysis, the muon momentum scale is a fixed input parameter 
Rik>>  (not varied by MINUIT), and the Q2 cut is applied later.  Thus, for every 
Rik>>  choice of pmu scale, events will migrate in or out of the lowest Q2 and 
Rik>>  Enu bins as the data file is read in, and every such fit will include 
Rik>>  slightly different set of events.  When I investigate pmu scale in the 
Rik>>  MA analysis, I rerun the fit for a range of input parameters, and look at 
Rik>>  the chisquare by hand and produce a plot like I have shown at the June 
Rik>>  collaboration meeting.

MY>>>  Then, the error quoted is a sum of change in Q2_rec shape due to momentum 
MY>>>  scale shift  and a fluctuation of data around the threshold.  Is there any 
MY>>>  evidence that the latter is small enough, or not bias the measurement?


Rik>>> You mean one of the following?
Rik>>>
Rik>>> 1.  The error estimate for pscale.  This is done with the spectrum fit, 
Rik>>> and has no Q2 cut, so there is no migration involved.  And it yields the 
Rik>>> same central value as the MA fit with the Q2 cut.  This estimate I claim 
Rik>>> is robust.

Rik>>> 2.  The conversion from pscale error to MA error (given as 1:5). 
Rik>>>
Rik>>> 2a.  Fluctuation at the threshold due to statistics?  This pscale is 
Rik>>> applied to a high statistics MC and is negligible.  And I see linear 
Rik>>> behavior for a very large range of pscale.  Any fluctuation at threshold 
Rik>>> due to data (not MC) statistics is accounted for already by Minuit from 
Rik>>> the defnintion of the likelihood.

Rik>>> 2b. There could indeed be correlation of this 1:5 value with other 
Rik>>> systematic errors at low Q2, such as Pauli Blocking or nQE/QE.  
Rik>>> At different times in the past three years I obtained results between 
Rik>>> 1:4 and 1:6, but never as high as 1:7.  This variation is almost 
Rik>>> certainly due to other systematic biases -- most likely nQE/QE.  Taking 
Rik>>> this range as a starting point, if I took the highest value obtained (1:6), 
Rik>>> this would increase the total error in MA from 0.12 to no more than 0.13. 
Rik>>> Furthermore, I estimate this effect comparing MC to MC with different 
Rik>>> pscale, which also gives 1:5.  This should, by defninition, have no 
Rik>>> other systematic biases, yet I get the same number.  [It actually does 
Rik>>> have some bias, for technical reasons, which does not seem to affect 
Rik>>> these physics result enough for me to take the time to remove it.]

Rik>>> Regardless of which of these (1, 2a, or 2b) you meant, I think 2b is the 
Rik>>> most serious, but it also seems satisfactory.  We are talking about the 
Rik>>> error in an error which will be taken in quadrature with other similar
RIk>>> sized errors, so the effect on the final quoted error is not so large. 

MY>>>> OK, now I think I have given sufficient explanation for this point.


On a different topic:

SA> As I see from the proton rescattering note, the main systematic effect is
SA> a change on the selection eficiency of the 2track-QE selection. I think
SA> mantaining just that makes both the explanation and, mainly, the
SA> interpretation much easier.

SA> As with other systematics, I am not totally sure we expect it to be a
SA> completely linear effect. Given the fit result which prefers a larger
SA> rescatering (and not a smaller one, as simulated for your studies), I
SA> would prefer to have an extra calibration point, with a positive effect.
SA> If the rescattering is much larger, and since it affects more low energy
SA> protons, there can be multiple rescattering which enhances the affect:
SA> the 2trackQE->nonQE would be even more replaced by 2trackQE->1track.
SA> If we parametrize all this information in the fit with just a linear
SA> scaling, we can be trying to introduce more information than the one we
SA> do have.[By the way, in the fit procedure description you say the
SA> rescattering increases the number of 2track-QE. In fact it decreases.]

SA> Redifining the proton rescattering systematics as an effect on the
SA> 2track-QE efficiency only, would further simplify the interpretation by
SA> separating it from the 1/2-track migration. With the advantage that no
SA> literal meaning would be extracted from the result of the fit for these
SA> parameters, which we use only to propagate several kinds of uncertainties
SA> (in the physics models and detector simulation) together.

SA> Again, i think the discussion of nQE/QE has to be re-written. The
SA> uncertainty we quote for this parameter is for the ratio on number of
SA> events seen, or if you want, total cross-sections. Here you change
SA> a lot the QE cross-section by changing MA, and this parameter reflects a
SA> change in the nonQE cross-section alone. Most of it can be absorbed by
SA> redefining the ratio using the new QE cross-section, and only the other
SA> part is expected to be within the uncertainty quoted. This takes out the
SA> correlation with MA and normalization, and leaves to discuss only the
SA> correlation with proton-rescattering and 1/2-track migration.

Rik>> The parameter is applied to scale the nQE up or down.  This parameter 
Rik>> moves with the value of MA because I recalculate QE-MA with the correct 
Rik>> normalization for the new QE-MA.  So the reported parameter correctly 
Rik>> reflects the scaling for the best-fit QE-MA, not for the default 1.1.

Rik>> What I should include in that section is some guidance for the reader 
Rik>> about how much of that parameter can be attributed to the movement of 
Rik>> the absolute QE cross-section in the MC due to the MA parameter.  
Rik>> From memory, I think this is of the order +10%, but I'll do the calculation.

Rik>> About proton rescattering:
Rik>> I'm not sure about whether it is useful to avoid "literal meaning" in 
Rik>> the fit value of proton rescattering.  The alternatives seem to be a 
Rik>> simple 2tkQE->1tk migration as you suggest, or a 2tkQE <-> 2tknonQE 
Rik>> migration, and letting the 2tk<->1tk and nQE/QE normalization parameters 
Rik>> shift also.   I ran the fit with this last method (it is described in the 
Rik>> note) for comparison, and the former method is so mathematically similar 
Rik>> to what iI have done, it would make a negligible difference.  Personally, 
Rik>> I don't feel strongly about these different possibilities.

Rik>> Running the 120% NN cross-section would take a significant amount of 
Rik>> time, if I did it the same way as the 80% NN study, because it involves 
Rik>> a new set of NEUT vectors and the complete scifi simulation.
Rik>> Incidentally, using 80% is already an improvement from before, where the 
Rik>> comparison was with 0% and 100% NN cross-section.  I looked very quickly at 
Rik>> that, and concluded that the interpolation was indeed roughly, suprisingly 
Rik>> linear for most of this range.  I am very comfortable that using only 80% 
Rik>> and 100% NN gives good enough result on a parameter that has a small effect 
Rik>> on the MA fit. 

SA>>> My comment for the proton rescatering came from the fact that i do not
SA>>> know if it should affect also the backgrounds (I know in SciBar, the
SA>>> second track in CC-1pi two track events is usually the proton, for
SA>>> example) and thus applying it only to CC-QE would already be an
SA>>> approximation. Anyway, I have no strong preference for any of the methods,
SA>>> except for the feeling it could ease the understanding of what happens in
SA>>> the fit. Also, if the effect is roughly linear from 0% to 100% we can
SA>>> accept it will be linear to 120% (i didn't know the previous method
SA>>> started from there). I do understand that making extra vectors at 120% 
SA>>> is extra work, and it can be avoided if we have confidence on the linearity.
SA>>> So I am happy with the present result and text.

SA>>> For the nQE/QE there were two questions: one that the correlation with 
SA>>> MA comes mostly from the change in the total QE cross-section while for a
SA>>> distracted reader it can be understood as an absolute change in the amount
SA>>> of nonQE (meaning a large uncertainty in the nonQE cross-sections). On the
SA>>> other hand, the expected uncertainty quoted in the note is for the change
SA>>> in the amount of nonQE, since the QE cross-section is fixed in the spectra
SA>>> fit (no variation in MA). Again i think it is good to avoid, or at least
SA>>> explain explicitely, the obvious correlation and to comment on the
SA>>> expected uncertainty for the value in the fit - which much then be larger
SA>>> than the one in the spectra fit. My comment here is for the text that i
SA>>> feel is misleading more than for the fit procedure or results.

SA>>> The muon momentum scale is obviously a more serious source of systematic
SA>>> uncertainty. I do think that the discussion in the draft is clear and
SA>>> convincing. The main question could be the effect on the thresholds in Q2,
SA>>> which are not used in the spectra fit. Maybe it can be studied in the MA
SA>>> fit with varying Q2 thresholds?


SA> Just a small extra question on the efficiency and purity table: i wonder
SA> if the numbers are very different after the Q2 cut, and if they should be
SA> given also.

Rik>> I had not thought of that suggestion.  I'll check to make sure it is easy 
Rik>> to calculate.  Might be a good idea.  I think the QE purity improves, 
Rik>> because of the way the QE assumption is used to calculate Q2.
[Note:  not included in the draft at this time.]