Listening tests are very difficult to administer. Small errors in the setup can lead to erroneous results.
Vinyl places many restrictions on the mix and mastering engineers. High-frequency content must be limited due to the mechanical limitations of cutters and playback cartridges. Low-frequency energy must also be limited for similar reasons. Stereo content must be limited and stereo separation of the system is also limited. Vinyl cannot be aggressively mastered for increased loudness, the way CDs can.
CDs can handle a wider variety of mixing styles, and a wider variety of spectral content. However, CDs do not always deliver better results. The loudness wars have degraded the quality of many newer CD releases. Early CD releases were degraded by poor quality conversion and poor quality digital processing. The conversion and processing problems are behind us, but the loudness wars are alive and well.
Bottom line- vinyl is mastered very differently than CDs. It is possible to record any vinyl master on a CD, but many CD masters cannot be recorded on vinyl. Unfortunately (for those who like the vinyl sound), most recordings that were originally released on vinyl, are re-mastered when they are re-released on CD. The result is that vinyl and CD releases of the same album never sound the same.
The restrictions of the vinyl format dictate the tonal balance of the final master. This gives a certain character to all vinyl recordings. These restrictions dictated the mixing style of the 60s, 70s, and early 80s.
Your ADC1 and DAC1 can capture and reproduce everything recorded on the vinyl and can do so with near-perfect transparency. We have produced 16th generation dubs using the ADC1 and DAC1. A group of mix and mastering engineers were unable to tell the difference between the 1st generation and the 16th generation. These converters are very transparent and will not impact the sound of the vinyl recording.
Set your ADC1 so that the highest peak reaches -3 to -10 dB. Use the peak-hold function on the ADC1 to see if you have the input gain set properly. If you go over -3 dB you may want to reduce the input gain on the ADC1 and start over. Ideally, we want to leave 3.5 dB of headroom for digital processing. If we record at too low a level, noise will increase (although the ADC1 and DAC1 are more than 30 dB quieter than a vinyl record).
On playback, adjust the DAC1 volume control to match the level produced by the turntable playback path. Ideally the DAC1 should be operated in the upper half of its volume control range (to optimize the signal to noise ratio). If you are using the XLR outputs, you can set the passive attenuators to keep the volume control in the ideal range.
To run your test, you will have the challenge of starting the vinyl and digital recordings in sync. To avoid this issue, you may want to just pass the live vinyl feed through the ADC1 and DAC1 without recording. The only problem is that the converters add a small delay to the signal path and some listeners may be able to detect this delay if the switching is instantaneous. If the signal mutes for a fraction of a second while switching, then you should be able to get away with live A/D and D/A conversion. Make sure that switching transients sound the same when switching back and forth between the two sources. At Benchmark, we have a relay-controlled ABX switcher that does this very well. Listeners will pick up on the slightest differences in switching transient even if they cannot hear a difference in the two sources. Tests should be at least be blind (listener does not know which source is which). Ideally the test should be double-blind (neither the administrator nor the listener know which source is which). Double-blind tests usually require a computer-controlled switcher.
If you do decide to record the digital signal, I would not adjust the gain in the digital system unless you are sure that it is properly dithered. If you do adjust it, I would keep any gain adjustments small ( a few dB at most). Playback levels must be matched to an accuracy of 0.1 dB. A test record with a steady tone is useful for calibration. A good volt meter with a fast-responding peak hold function is useful if you do not have a test record and SPL meter.
The 22 kHz limitation of 44.1 should not be a problem. You should be able to run the test at 96/24 if you wish. One advantage of 96 kHz is that the delay through the ADC1 and DAC1 will be shorter. The live vinyl --> A/D --> D/A comparison to the direct vinyl should work better at 96 kHz because the delay through the converters is shorter.
We have only skimmed the surface. Search double-blind listening tests or ABX listening tests for more information.
In some cases, upsampling will improve the output of your D/A converter. The low-pass filters incorporated into the upsampling process will essentially replace the filters in your D/A converter. If the upsampling software has better filters than those built into the D/A, then you may see an improvement.