Reference sample for batch correction

[AlexBouz]

Dear @CUHIMSR ,

Thank you for sharing this important tool for batch effect correction. I had a question regarding the reference sample named "anchor", i downloaded the fcs files from the vaccination study uploaded on flow repository, and tried to run the R script using the examples terminologies:

[1] 2019-10-28 14:52:28 BatchAdjust.R
[1] basedir:C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction
[1] outdir:C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction/debarcoded
[1] channelsFile:ChannelsToAdjust_example.txt
[1] batchKeyword:Barcode_
[1] anchorKeyword:anchor stim
[1] transformation:FALSE
[1] method:95p
[1] Reading channels file: ChannelsToAdjust_example.txt
[1] Adjusting 39 channels.
[1] Batch 1 anchor not found.
[1] Check file naming requirements?
Error in BatchAdjust(basedir = "C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction", :
The reference anchor must exist and have the batch number 1.
In addition: Warning message:
In system(ls_cmd, intern = TRUE, ignore.stdout = FALSE, ignore.stderr = TRUE, :
running command 'ls -1 C:/Users/Alexandre.Bouzekri/Desktop/Clinical\ files\ mass\ cytometry/Batch\ correction/anchor\ stim.fcs' had status 2
Called from: BatchAdjust(basedir = "C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction",
outdir = "C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction/debarcoded",
channelsFile = "ChannelsToAdjust_example.txt", batchKeyword = "Barcode_",
anchorKeyword = "anchor stim", method = "95p",
transformation = FALSE, addExt = NULL, plotDiagnostics = TRUE)

I had this error about the reference sample, is it coming from some misslabeling requirements?

Thanks for the clarification.

[RonSchuyler]

What operating system are you using?

…

On Mon, Oct 28, 2019 at 1:01 PM AlexBouz ***@***.***> wrote: Dear @CUHIMSR <https://github.com/CUHIMSR> , Thank you for sharing this important tool for batch effect correction. I had a question regarding the reference sample named "anchor", i downloaded the fcs files from the vaccination study uploaded on flow repository, and tried to run the R script using the examples terminologies: [1] 2019-10-28 14:52:28 BatchAdjust.R [1] basedir:C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction [1] outdir:C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction/debarcoded [1] channelsFile:ChannelsToAdjust_example.txt [1] *batchKeyword:Barcode_* [1] *anchorKeyword:anchor stim* [1] transformation:FALSE [1] method:95p [1] Reading channels file: ChannelsToAdjust_example.txt [1] Adjusting 39 channels. [1] Batch 1 anchor not found. [1] Check file naming requirements? Error in BatchAdjust(basedir = "C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction", : *The reference anchor must exist and have the batch number 1. In addition: Warning message: In system(ls_cmd, intern = TRUE, ignore.stdout = FALSE, ignore.stderr = TRUE, : running command 'ls -1 C:/Users/Alexandre.Bouzekri/Desktop/Clinical\ files\ mass\ cytometry/Batch\ correction/anchor\ stim.fcs' had status 2* Called from: BatchAdjust(basedir = "C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction", outdir = "C:/Users/Alexandre.Bouzekri/Desktop/Clinical files mass cytometry/Batch correction/debarcoded", channelsFile = "ChannelsToAdjust_example.txt", batchKeyword = "Barcode_", anchorKeyword = "anchor stim", method = "95p", transformation = FALSE, addExt = NULL, plotDiagnostics = TRUE) I had this error about the reference sample, is it coming from some misslabeling requirements? Thanks for the clarification. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=ABFVYFP3MKHOZ656ESCMLD3QQ4ZJRA5CNFSM4JF6TLJ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HU3LGOA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFVYFINGVJ5PE3RUYLG4RDQQ4ZJRANCNFSM4JF6TLJQ> .

[AlexBouz]

Dear @RonSchuyler thank you for replying, I'm running the script you wrote on Windows 10 Pro, my version of R is 3.6.1 and R studio Version 1.2.5001, if i'm correct the fcs files from the flow repository were already debarcoded and correspond to the anchor samples from the study. I'm not sure if this is related to the error i have though.

[RonSchuyler]

The code currently will run only on linux or mac OS. This is due to path naming, and would be straight forward to extend for Windows, but I don't have access to a system I could use for testing. I suggest running on linux or mac, or digging into the code to handle parsing of directory paths for Windows.

…

On Mon, Oct 28, 2019 at 2:24 PM AlexBouz ***@***.***> wrote: Dear @RonSchuyler <https://github.com/RonSchuyler> thank you for replying, I'm running the script you wrote on Windows 10 Pro, my version of R is 3.6.1 and R studio Version 1.2.5001, if i'm correct the fcs files from the flow repository were already debarcoded and correspond to the anchor samples from the study. I'm not sure if this is related to the error i have though. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=ABFVYFJDXJOBC7G5AVEETXDQQ5DBXA5CNFSM4JF6TLJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECOIWNY#issuecomment-547130167>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFVYFLD6U3T2CVCMNVSXA3QQ5DBXANCNFSM4JF6TLJQ> .

[AlexBouz]

Dear @RonSchuyler , yes i missed the line where is the code is linux or Mac compatible only. I tried to run one example on a remote machine, the path naming looked fined however this batch reference 1 still came up when running the script. I named 4 files from the flow repository using the first example you described

[AlexBouz]

After renaming of the example files as Barcode set 1 there was some progress:

[RonSchuyler]

Alex, The line "batchestToAdjust: 1" indicates that you only have batch 1, and no other batches to adjust relative to #1, so there is nothing for the software to do. All batches must be named according to your pattern. Hope that helps.

…

On Tue, Oct 29, 2019 at 11:48 AM AlexBouz ***@***.***> wrote: After renaming of the example files as Barcode set 1 there was some progress: [image: File list Screenshot] <https://user-images.githubusercontent.com/37419030/67794205-aa200c00-fa52-11e9-98ae-0d3bfa34d95d.png> [image: Mac screenshot] <https://user-images.githubusercontent.com/37419030/67794220-b0ae8380-fa52-11e9-86e0-abb277cc868b.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=ABFVYFPPDPPAIJ6QRB2ZSD3QRBZPTA5CNFSM4JF6TLJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECRO3SY#issuecomment-547548619>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFVYFLS7O3632MYU7WM3PDQRBZPTANCNFSM4JF6TLJQ> .

[AlexBouz]

Hi @RonSchuyler,
I changed the file naming to list a bunch of anchor samples belonging to different batches, they load sequentially. However the TRUE/FALSE argument error still occurs for the Median Threshold.

[SamGG]

Hi @AlexBouz
We ran recently cytofBatchAdjust in one shot on 27 batch of 20 samples each (7 GB of disk IIRC). The process stops during the permutation plot, but all FCS were normalized within less than 10 minutes. We did the plots separately afterwards.
Could you show the files list and the batchAdjustCommand?
Best

[AlexBouz]

Hi @SamGG , @RonSchuyler i run the BatchAdjust.R with the changes you did on
two batches of two files each as a small sampling size, the code generated the normalized files, a folder of distribution plots, i attached the log file for info. I will give a shot wit bigger datasets, this time the code was run on a Windows 10 OS.
LOG_BatchAdjust.2020.04.05.103107.txt

Thanks

[Chelysheva]

Hi @AlexBouz
I have seen your issue from 31 Oct 2019. Obviously, you've managed to overcome it. How did you solve your TRUE/FALSE argument error for the Median Threshold? I am attaching my screenshot, where I try to run a small test subset of 2 batches with several samples in each.
Many thanks in advance!

@CUHIMSR and @RonSchuyler
Maybe you could help me with that issue? (I am using Linux, R version 3.6.2) Would be grateful.

Best

[SamGG]

Hi,
I think your Bi209Di channel is always zero. I would start with a few channels with clear signal (CD4, CD3, CD8, CD19, CD20...) to get confident in the use of cytofBatchAdjust.
Best.

Hi @RonSchuyler @AlexBouz ,
How to solve the TRUE/FALSE argument error? It says : Error in if (FractionZerosPooled[[acol]] > maxFrac0ForMedianThreshold) { : missing value where TRUE/FALSE needed
I have attached a screenshot of the same

[SamGG]

Hi,
You didn't mention me, but here are some checks.
Did you set nz_only argument to TRUE?
Did you careful select channels with signal? Sometimes some channels are opened on CyTOF but no labelling matched them. In such case, there are too many zeroes.
Did this error occurs at the first channel?
You should uncomment line 407, the one just before the error, so you could report some interesting diagnostic values.
I think Ron will answer you.
Best.

[SamGG]

nz_only should be set to FALSE, as in the default value. There is no reason to remove zero from the quantile computation.
"Browse[1]>" means that when you got an error the code browser is opened instead of stopping. It allows to browse the environment, ie the variables. It could help understanding.
Locate line 407 using the original

CytofBatchAdjust/BatchAdjust.R

Lines 406 to 408 in f66c532

	FractionZerosPooled[[acol]] <- fzeros_ref;
	#print(sprintf("acol: %s zsd: %.2f zmin: %2.f%% zmax: %.2f%% pooled0s: %.2f%% %i of %i", acol, zsd, zmin, zmax, pzeros_ref, zeros_ref, length_ref));
	if( FractionZerosPooled[[acol]] > maxFrac0ForMedianThreshold ){

 Hello, Kindly ignore my previous email.I uncommented line 407. I set nz_only argument to TRUE.It shows the same error.I also checked channel names. This is what I got.I do not understand it. Thank you

…

On Tue, Jul 27, 2021 at 8:27 PM Rashmi Rao ***@***.***> wrote: Hello, Thank you so much for your response. I set nz_only argument to TRUE.It shows the same error.I also checked channel names. How do I find out if it is at the first channel? What is "Browse[1]> "? I did not understand what you meant by uncomment line 407. Thank you once again On Tue, Jul 27, 2021 at 8:02 PM Samuel Granjeaud ***@***.***> wrote: > Hi, > You didn't mention me, but here are some checks. > Did you set nz_only argument to TRUE? > Did you careful select channels with signal? Sometimes some channels are > opened on CyTOF but no labelling matched them. In such case, there are too > many zeroes. > Did this error occurs at the first channel? > You should uncomment line 407, the one just before the error, so you > could report some interesting diagnostic values. > I think Ron will answer you. > Best. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#1 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AON3B57GLZDH3BXXOZROIZTTZ37D3ANCNFSM4JF6TLJQ> > . >