Contains replies from Rik and the SciFi group
Q1 It would be nice to have a bit more on the data selection, before starting the discussion of systematics. Some of the effects are migration from one sample to other and so, for example, the Dtheta plot could help. Another question is, are cosmic rays and other background (not from neutrino interactions in SciFi) vetoed efficiently, or should a systematic (most probably negligible) be considered?
(Reply) A full discussion of the SciFi detector and data selection was omitted for efficiency, or because it is published elsewhere (in NIM). Obviously some of this will need to be returned for a publication draft. we will add more of it to the K2K note also as it is prepared. But yes, cosmic rays are a negligible background.
Q2 Related to this, it would be interesting to know the relative contributions to the fit from the three sub-samples for each data set. The importance of each can be guessed by the efficiencies, but the precise effect in each of the parameters could give some insight to the systematics too.
(Reply) This is related to a question below regarding nQE/QE. The answers to both are there.
Q3 The Pseudo-Vector contribution is negligible. (pg 3) A reference would be useful, and maybe an order of magnitude (at least when the change in the Vector form-factors is discussed).
(Reply) The pseudo-scalar term contributes only when tau neutrinos are involved, or at lepton production threshold, neither is true for our muon neutrinos. It is not measured, but going back to Llewellyn-Smith's 1971 paper, it is argued on theoretical grounds that it probably not more than a few percent (p.305) of the total cross section for tau interactions, and less for electron or muon interactions because it involves terms that go as (mass lepton)/(mass nucleon).
Q4 The MA 1P is also important (pg 12), naively one would expect this to be part of the nucleus description and MA 1P=MA QE, no? Can they be completely different? It might be worth commenting.
(Reply) It is usual to assume that they can take separate, independent values. The use of a dipole is just a phenomenological approximation. Even in Rein Sehgal (?) the resonance production form factor is not a true dipole, it has already been modified based on bubble chamber results. So, there is no expectation anymore that these MA should be the same.
Q5 We would not refer to the change of Vector form factors as producing a shift in MA (pg. 27). They basically change your definition of MA as an effective parameter: the part of the cross-section which is assigned to the "Vector" changes and leaves different space for the "Axial". Is there a way to represent grafically the dependence of MA with the part which is assigned to the "vector"...
(Reply) We agree that the description of how changes to the Vector component affect the MA fit should be more carefully worded. Another person who read the draft said the same thing. And I think we all agree that the two results, one with dipole vector form factors, and one with the updated ones should be presented together. One represents our current best estimate of this "effective parameter" while the other allows us to compare the result with previous bubble chamber MA analyses.
Q6 Anyway, if those new form factors are the good ones, why should we not use them? To keep a reference for comparison with other measurements, a possibility would be to report always the two of them and assume that MA is an effective parameter, which somehow it is. Is there any compeling reason no to use them ? What is the Xi2 of the fit for the new form factors ?
(Reply) The chisquare is identical for K2K-2a. Both MA values are presented together in the text and in the conclusion.
Q7 You almost always discuss Pauli Blocking and Fermi Motion together (since they depend on the same parameters), but they have different effects: Fermi Motion gives you harder Q2 (related to high angles), Pauli Blocking blocks low Q2 (related to low energies): that is, they affect different parts of the spectra, and different kinematic variables, and it is not obvious that it can all be reduced by the low q2 cut, as you say in several places.
(Reply) I have added a plot to section VI.C.5. which demonstrates the difference between a free nucleon and the Fermi gas model (from a calculation by H. Nakamura). This shows the three regions where a nucleon momentum distribution has an effect on the differential cross-section. As you say, it is not obvious that it can all be reduced, but this section describes with some care how the other (non Pauli-blocking) contributions affect the analaysis.
One interesting result is that this analysis has no statistical sensitivity to the high Q2 tail, I eliminate low Q2 uncertainties due to pauli blocking, and the resulting region to which I have senstivity is affected by fermi motion at the level of at most 3%. A higher statistics data sample might show an effect due to the high Q2 tail.
There is no Q2 cut, but as a systematic check, I tweaked the distribution there to see if I had any sensitivity what happens at high Q2. The answer was no.
Q8 It is a bit worrying because we see a Data/MC disagreement in the angle between the proton and the muon, coming from the proton angle... and which could be due to the Fermi Motion. Do you see a similar effect? (you have no plot of the second track variables)
(Reply) We do see some discrepancy in the observed proton angle. This might be due to a combination of Fermi motion, and proton rescattering. I have prepared the basic plots from SciFi events using the baseline Monte Carlo. They are available separately in the document linked here: protonmonangle.pdf
Q9 On the other hand (pg. 30) you say in reality you change "the amount of Pauli Blocking". Can it be that, by the way you calculate it, you effectively disregard the changes which come from the Fermi motion? For the sigma(QE) and Q2 distributions, the Pauli blocking is the most important effect, but changes in the efficiencies and purities of the sub-samples might be also important (this is the same kind of systematic as could come from the tracking efficiency or proton rescattering)...
Q10 In fact, when you describe (pg. 4) the effect of the Pauli Blocking, you don't explain the method you are using to simulate it. As far as we understand it, in NEUT, if the proton is below the K_F level the interaction is prohibited. Do you use the same model ? This is important also to know how do you do it in the other nuclear models where the nucleon momentum doesn't have an absolute maximum. The same for the Fermi Motion: are they included in the cross-section calculation or in the migration matrix?
(Reply) The implementation in NEUT is as you have described. For the spectral function model, the Pauli Blocking is still implemented in a simple way, as it is for the Fermi Gas, with a kind of K_F. In this sense, it is not correct for the spectral function model, and will need to be improved. However, a more sophisticated analysis is not yet available from the theorists who prepare such calculations, though some preliminary results have recently been completed. I have added references to this in section VI.C.5.
I incorporate the uncertainty due to uncertain Pauli blocking by changing the number of interactions blocked by 10%, relative to the Fermi Gas calculation. I have made this more clear in the text. This is independent of any assumptions about the nucleon momentum distribution.
Q11 Is it that the Pauli Blocking is also one of the main effects on the 1pi production? If you produce a real resonance, the final state - the decay of the Delta - is not connected to the production state. It is instead in the decay that the proton/neutron appears and where the Pauli Blocking should be applied. So the effect should come from the non-resonant part of the cross-section....but our understanding is that it is very poorly known... That is the cross-section unknowns are probably more important than the nuclear effects themselves. The question is: can we assign a reasonable error for those uncertainties?
(Reply) I agree with the first part, it is the final state where P.B. applies. In the paper by Paschos, Yu, Sakuda (PRD 69 (2004) 014013 citation is added to the K2K note), they calculate the amount of Pauli Blocking for the production of three resonances, including the Delta. The total amount of Pauli Blocking in their calculation (something like 10 percent at low Q2) is less than what is calculated for quasi-elastic (something like 50 percent). However, it might be that the Pauli blocking reaches a little higher in Q2 for the resonance production state, at least for higher energy events. A quick test of the MA analysis shows that even this will have a negligible effect on the SciFi MA fit value. It is noted in that paper, and elsewhere, that bubble chamber experiments which can very cleanly separate the single pion production see a deficit at low Q2, though it is not certain from their data alone that it is due cross-section, nuclear effects, or scanning efficiency.
As for the uncertainties in the cross-sections, this analysis accounts only for an overall normization, and does not assume an uncertainty in the shape of the cross-section with Q2 or Enu, except for the MA-1pi study. But since the resonance cross-section is mostly motivated by a first-principles calculation, I am not sure an additional uncertainty can be explicitly added.
Q12 The gaussians you present do not really fit the Q2 distribution (and the Theta_mu). In the Q2 one there even seems to be two different contributions...
(Reply) There is no expectation that a gaussian is the correct function, but it does give the simplest, and not unrealistic estimate for the resolution. The shape of the q2 resolution curve tracks the shape of the angle resolution curve, that is where the non-gaussian tail is coming from.
Q13 Both E_nu and q2 are highly correlated, but you are using them independently in your migration matrices...wouldn't it be better to use the muon momentum and angle as parameters to migrate ? It is also true that the muon momentum and angular resolution are correlated, but we would expect somehow smaller correlation values.
(Reply) I am confident that my method (discribed in the migration matrix section) is actually independent of whether there is a correlation, because it is already accounted for by the MC when building the migration. A couple systematic checks have been done to confirm that there is no significant problem. The first is that the MC reproduces itself: I get MA = 1.1 when I use the MC as the input data. A second check of the method is to apply a simple one-parameter acceptance calculation for Q2 only (and not for the correlated Enu), which also give consistent results. The resolution in Q2 is by far the most important to this analysis.
Q14 The Theta resolutions are for long tracks in SciFi, right? It is relevant to know if they are very deteriorated for proton tracks (this will tell you the resolution in DeltaTheta), and might be interesting to know the one in MRD (do you have some specific treatment for the cases in which the two tracks leave the SciFi and could be matched to the MRD?).
(Reply) They are deteriorated for short tracks, but still quite good. The width of the true - reconstructed distribution is about 1.9 degrees, twice as wide as the 0.8 degrees for muon tracks.
Q15 You say the Pmu is shifted and this will be arranged later (pg 6), presumely by momentum scale, but this shift is not refered again. It would be good to give also the final numbers (to separate what part of the shift in Enu is due to this, what part to Fermi Motion?).
(Reply) The shift is applied simply, as the ntuple is being read in for analysis. The shift I am referring to here is actually just a tuning of the reconstruction algorithm (because the comparison here is not made with data), and is distinct from any other physics effect, such as Fermi motion. We intend to add tuning directly to the ntuple, which will make it easier to produce the correct plots. This regeneration of the ntuple will happen in early 2005.
Q16 In fact, the discussion on the problem of the muon momentum scale is very interesting (and somehow worrisome). We understand your point that the scale, energy spectrum and MA are higly correlated and it is very difficult to resolve the problem with neutrino data alone. But we also see that, in TABLE VI, the results with the spectrum fit are systematically lower than one while the ones with material assay or test beam (not beam related) are above one. Shouldn't we believe the test beam results ? We understand that if we set the scale to 1.0 or 1.03 the value of MA jumps to values of 1.4 or 1.6, that are not very physical. But, it seems, this is showing a serious problem somewhere in the montecarlo, or the neutrino spectrum or detector simulation.
(Reply) The neutrino data and the spectrum fit represent our best experimental measurement of this parameter. However this parameter may include systematic effects not completely related to the MRD energy scale calculated from the material assay. Since the correlation with MA is taken into account, this is the best available measurement to use. It also consistent with the method the conveners agreed should be used for the spectrum fit.
There are some further comparisons of SciFi and SciBar MRD data that we would like to do, in order to clarify this.
Q17 Also, given the justification for Pscale, and its definition, one would expect that it would be equal for the two data-sets, and it is not. Actually, the fitted flux should also be compatible between the two data sets and that might deserve a comment.
(Reply) Given the uncertainties, the SciFi K2K-IIa and K2K-I data agree with each other. There is a further uncertainty not listed in that table due to the uncertainty in the Lead Glass migration which is correlated with the MRD energy scale with about the same +/- 0.005 error.
(Reply) The second question: is the fitted flux compatible between the two data sets? The SciFi K2K-1 and K2K-2a data sets are consistent, only one of the flux values is beyond the one-sigma error. If the MRD escale is artificially forced to be closer to its nominal value of 1.00, the chisquare fit to all data samples becomes poor and many of the values for the flux are beyond the nominal 1 sigma errors.
For the SciFi fluxfit, we do not include events with angle less than 10 degrees, because of the low angle, low q2 deficit. This is the same criteria used for SciBar and similar to the 20 degree cut of 1 kT. This cut is known to have a significant effect on the flux and the escale results.
Q18 In the text, it is not clear (pg. 38 or before) how the Pscale enters the calculation (affecting Data or MC, correcting the shift in pg.6 or not, ...). Also no justification is given for choosing the values between MA=0.9 and MA=1.4 to define the maximum change in Pscale, is there a reason?
(Reply) The reference on page 6 is in reconstruction only (MC vs. MC). The discussion on page 38 refers to disagreement between Data and MC, presumably because the MC is not perfect. It is applied simply by scaling the muon momentum of the data following the method of the spectrum fit.
(Reply) The two justifications that this range is reasonable are the chisquare in the spectrum fit remains reasonable here, but worsen significantly outside this range, and also this is not so different than the previous experimental results on MA given in many of the references.
Q19 The QE/nonQE ratio comes from the fitting of the three different samples (pg 8). Are they consistent if you fit them separately? If the samples are dominated by different types of background then maybe this could give some insight also to the modelling of the different backgrounds... (it is hard to see why we should have one number for a relative normalization between one cross-section and all the others).
Q20 You are fitting the NQE/QE ratio, but!, why not also the 1pi/Npi? You will be very sensitive to that in your 2track-NQE sample and it could improve the xi2 at the end.
(Reply) The SciFi analysis is not particularly sensitive to 1pi/Npi. Only the delta-theta separation of nonqe/qe is used. If we had a large and well-reconstructed three-track or multi-track sample, then this could meaningfully constrain 1pi/Npi.
Q21 In the fit tables (pgs. 17/21) what does it mean QE/nonQE is not exact?
(Reply) There are parameters for five energy regions, nonQE/QE, and an overall normalization. In this analysis, the last one is not an independent parameter, and it is fixed by hand from a calculation. The value of this overall normalization parameter does not affect the value for MA at all, but does affect the fitted values for the flux and nonQE/QE.
The value used now is only approximate. The final value will be calculated for the first draft of the paper. It is a technical point that does not affect the physics of the analysis. The warning is there so that the fit value of nqe/qe is not compared to the spectrum fit until the calculation is finalized. Again, it has no effect on the fit value for MA or the chisquare of the fit.
Q22 We come now to our biggest worry. We are a bit puzzle with your likelihood. You are saying all the time of the note that you are interested in the q2 shape. But, you use the E_nu and the actual spectrum fit to do your fit. This is (it seems) because of your binned likelihood, where a good fraction of the Xi2 comes from the fitting of the energy distributions. This way you can also, as you mention in the note, migrate the sensitivity to MA to a sensitivity to the spectrum and viceversa. This is reducing your sensitivity and introducing some conceptual problems. We also understand that the information is in fact in the q2 distribution for each Energy value. If this is the case, why not performing an unbinned likelihood where only the actual shape is taken into consideracion. It is not 100% correct because it will come from the substraction of the NQE background shape but it is possible that the overall situation gets better this way. You just simply do:
lodlklh = - (dsigma(q2)/dq2)|E_nu * (dsigma(q2)/dq2)|E_nu
This way the actual number of events in the bin E_nu does not affect your measurement but only the shape of the q2 for each E_nu value (dsigma(q2)/dq2)|E_nu .
This way you can also remove this circular problem of the fixing the spectrum for MA and MA for the spectrum fit. It could be that the sensitivity is lower but also the systematics, and it looks like a more solid measurement... Are we missing something in the arguments ? In fact, you do something similar in FiG.12 although you never quote the result of combining these numbers.
(Reply) For the primary MA analysis, the spectrum fit flux parameters are NOT used, and the normalization of each energy region is completely free, making this the shape fit, equivalent to what you describe. There is a secondary consistency check that separately includes this flux information.
We plan to do a non-binned likelihood also.
Q23 Since you have a bad Xi2 (C.L. lt 4% ) in the fit (although this could be accounted for in the systematics ), it would be nice to confirm that/if there is a piece of the data that fits well. For that it would be nice to have the Xi2 for each energy beam and sample for q2.gt.0.2. This can be considered another consistency check.
(Reply) In the SciFi spectrum fit (not the MA analysis), the poor chisquare always comes from the one-track sample (> 1.5 per dof). When the same data is binned for the K2K-1 MA analysis, only the lowest energy bin stands out, and the three event samples are farily similar. In the K2K-2a MA analysis, the two-track QE subsample shows a poor chisquare, as does the lowest energy bin. I have added much more information on the chisquare values for different subsamples in the updated K2K note.
Q24 And due to the relation between spectrum and MA, it would be nice to show what are these values:
CORR (MA,SPECTRUM_1) ....
even graphically. This way people would have a better filling of how the variables are related and it could help in your discussions.
(Reply) You mean the correlation between MA and each spectrum variable. This is studied twice, once in the spectrum_fit results, once in the MA analysis. I think you mean a demonstration of this in the MA analysis. I am working to add this to the K2K note, but it is not there yet.
Q25 How do you include it in the fit? In the migration matrix, right? But what variables does it depend on? And should it be varied together in the background, creating a common migration of QE+nQE from the 1track to 2 track, etc?
(Reply) We have calculated a reweighted MC with the scattering cross-section reduced by 20% and compared it with proton scattering data. This provides a central value and separate migration for this systematic effect.
Q26 In pg. 25 you mention a bias to low values because of the low statistics. Presumably the shift is still compatible with the systematic error. Anyway, we don't understand why you are so confident that this is a problem of the lack of statistics.
(Reply) This refers to low statistics in the K2K-IIa data set. The shift is small compared to the overall systematic errors. As a systematic test, I have further divided the data in half, and that shows a significant change, in the same direction for the first half and the second half of the data.
Q27 Also, in pg. 35, you mention effects at high q2. Do you have an upper limit in the q2 fit ?
(Reply) There is no upper limit, though the statistics are lower at high q2, so the analysis is not as sensitive to what is happening there. This was tested by artificially changing the MC expectation at the high q2 end, but still without a cut.
Q28 As you know, MA is a phenomenological parameter and thus, if you change the G_M and G_E, the meaning is different. Therefore, I think the change of MA value due to the change of G_M and G_E should not be in the systematic error term and each value should be presented in parallel and separately. Basically, it is better to use latest G_M & G_E. On the other hand, we can not compare the existing values obtained by the other experiments. This is the reason why I suggest you to present these values separately.
(Reply)Yes, we agree. This is the same comment made above in Q5. It is not included in the systematic error terms (and has not been since August 2004).
Q29 Also, it is not only for your analysis but I little bit worry about the energy scale and energy shift uncertainties. As far as I remember, there existed some inconsistency between K2K-I and K2K-II. (Also, consistency of energy scale in MRD obtained from SciFi and SciBar was slightly different.) This affects your result directly and it is better to be checked again with SciBar group.
(Reply) We also have significant concern about this. As of this moment, a study of the SciFi data alone and fit procedures does not indicate the reason for the range of energy scale values obtained by the different detector subgroups.
Q30 One general comment concerns the "low" energy bin between 1-1.5 GeV, which your report takes as evidence for a possible systematic at lower energy. I have to say that I don't think the data warrants such a conclusion. If I look at Figure 12, I see 10 different data points. The low point in K2K-I seems to be just 2sigma low, and the K2K-IIa point is within 1 sigma. Given 10 data points, on average we would expect to see one 2 sigma deviation. What is the chi2 for flat, anyway, and with what confidence would we reject the flat hypothesis? Overall I am under the impression that this point, while low, is entirely consistent with a mild statistical fluctuation. There's no reason not to check carefully for missed sysematics, but no reason to conclude that there is such a thing either. Indeed, if you spend too much time trying to come up with reasons why that data point might be low, you risk biasing the analysis. (It might be useful in the future to see if there isn't some way to do this as a blind analysis.)
(Reply) This problem has disappeared with the corrected LG migration code. My intention is that by itself I would not -- did not -- think it is systematic. Later, I realized the apparent bias due to the 1L events seen in a separate test would have this same effect on this fit.
Q31 On p. 25, in the second paragraph there is a comment that the fits could be biased by a small number of events per bin, which can be avoided by rebinning. I have a hard time reconciling this with the likelihood equation on p. 11. In a likelihood analysis, if you use the full Poisson statistics formula in the likelihood, then you should be utterly insensitive to the the statistics in any bin. Indeed, you could choose to do an unbinned maximum likelihood analysis to no ill effect. I would understand why low bin statistics would bias a chi2 fit, but not a likelihood.
(Reply) I am actually referring to the approximation of the chisquare from eq. 31.12 of the PDG-Review of Particle Physics from 2002/3. The likelihood calculation that lies underneath it is the usual binned likelihood (and, I think can give biased estimators in the binned case), but the value for the goodness of fit is sensitive to the statistics in each bin. The RPP gives a warning that some care should be taken in this case.
And in any case, the exercise of dividing the data in half (or in quarters) or using different binning map out where statistics is the limiting factor in the result.
Q32 There are a number of unevaluated systematics that you mention. I am unclear on what you propose to do with these. Examples are the form factors for single pion production and resonant production (p. 29), and the aluminum-oxygen difference (p. 43). I am not certain what other experiments have done about these, but strictly speaking these should only be ignored if they can be justified to be negligible. If they are negligible, then we ought to be able to justify that. If we can't show that they're negligible, then they need to be estimated somehow. Leaving them hanging is rather unsatisfactory. What are your plans for these?
The aluminum-oxygen difference has been estimated to be small, based on a combination of Pauli Blocking and binding energy. We plan to also test this with the K2K-IIc data. The MA-1pi parameter and the overall normalization (nQE/QE) represent the study of the 1-pi form factor study, further refinement does not seem to be justified. They are listed this way because they will become increasingly important for future, higher precision studies.
Q33 I am curious about how the systematics are quoted. I understand that some systematics are included as free parameters in the fit (ex. nQE/QE ratio). So really the uncertainty on the fitted parameters is a combined statistical/systematic error. I agree that you can determine the uncertainties of individual components by fixing all the other components to zero, as was done to get the statistical error. But because of the fit procedure, the different uncertainties will in general be correlated with each other, and cannot be simply added in quadrature. Basically what I'd like to see is a discussion of how the erorrs in Table IV are combined to give the final quoted systematics. The way I would do this is to list the statistical error on M_A (with all other parameters fixed to zero), then list the total systematic uncertainty on M_A all fitted parameters, then list the total of the additional systematics errors that are external to the fit.
(Reply) The revised draft has a better description of this, including a table.
Q40 p. 41, second paragraph from bottom: I am mystified by this statement "The actual situation is more complicated, because the relative amounts of 1L events and 3D events at each energy are similar, but the relative amounts of both are different at low energy and high energy." To my ear the tail end of this sentence contradicts the first part, so I'm not certain what you're trying to say. Are you saying that (1L/3D)_lowE is different from (1L/3D)_hiE? Or are you saying (hiE/lowE)_1L is different from (hiE/lowE)_3D?
(Reply) This description was poor, however I recently concluded a study of the effect of the uncertain lead glass density which eliminates the problem I tried to described here.
Created 12 December 2004 by Rik
Last updated 9 January 2005 by Rik