LinAlgError: SVD did not converge

[jingc128]

Hi,
we met the problem when we ran code:
fs.tl.learn_intercellular_flows(adata,
condition_key = condition_key,
control_key = 'D1',
flowsig_key = 'flowsig_network',
flow_expr_key = 'X_flow',
use_spatial = False,
n_jobs = 32,
n_bootstraps = 500)
This is the error report:

_RemoteTraceback Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 463, in _process_worker
r = call_item()
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 291, in call
return self.fn(*self.args, **self.kwargs)
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/parallel.py", line 598, in call
return [func(*args, **kwargs)
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/parallel.py", line 598, in
return [func(*args, **kwargs)
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/flowsig/tools/_network.py", line 185, in run_utigsp
obs_suffstat = partial_correlation_suffstat(control_resampled, invert=True)
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/conditional_independence/suffstats/ci_suffstats/partial_correlation_suffstat.py", line 35, in partial_correlation_suffstat
K = pinv(C)
File "<array_function internals>", line 200, in pinv
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/numpy/linalg/linalg.py", line 1983, in pinv
u, s, vt = svd(a, full_matrices=False, hermitian=hermitian)
File "<array_function internals>", line 200, in svd
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/numpy/linalg/linalg.py", line 1642, in svd
u, s, vh = gufunc(a, signature=signature, extobj=extobj)
File "/home/cj123/miniconda3/envs/flowsig/lib/python3.8/site-packages/numpy/linalg/linalg.py", line 98, in _raise_linalgerror_svd_nonconvergence
raise LinAlgError("SVD did not converge")
numpy.linalg.LinAlgError: SVD did not converge
"""

The above exception was the direct cause of the following exception:

LinAlgError Traceback (most recent call last)
Cell In[85], line 1
----> 1 fs.tl.learn_intercellular_flows(adata,
2 condition_key = condition_key,
3 control_key = 'D1',
4 flowsig_key = 'flowsig_network',
5 flow_expr_key = 'X_flow',
6 use_spatial = False,
7 n_jobs = 32,
8 n_bootstraps = 500)

File ~/miniconda3/envs/flowsig/lib/python3.8/site-packages/flowsig/tools/_network.py:344, in learn_intercellular_flows(adata, condition_key, control_key, flowsig_key, flow_expr_key, use_spatial, block_key, n_jobs, n_bootstraps, alpha_ci, alpha_inv)
330 print(f'starting computations on {n_jobs} cores')
333 args = [(adata,
334 condition_key,
335 control_key,
(...)
341 alpha_inv,
342 boot) for boot in range(n_bootstraps)]
--> 344 bootstrap_results = Parallel(n_jobs=n_jobs)(delayed(run_utigsp)(*arg) for arg in args)
346 end = timer()
348 print(f'elapsed time: {end - start}')

File ~/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/parallel.py:2007, in Parallel.call(self, iterable)
2001 # The first item from the output is blank, but it makes the interpreter
2002 # progress until it enters the Try/Except block of the generator and
2003 # reaches the first yield statement. This starts the asynchronous
2004 # dispatch of the tasks to the workers.
2005 next(output)
-> 2007 return output if self.return_generator else list(output)

File ~/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/parallel.py:1650, in Parallel._get_outputs(self, iterator, pre_dispatch)
1647 yield
1649 with self._backend.retrieval_context():
-> 1650 yield from self._retrieve()
1652 except GeneratorExit:
1653 # The generator has been garbage collected before being fully
1654 # consumed. This aborts the remaining tasks if possible and warn
1655 # the user if necessary.
1656 self._exception = True

File ~/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/parallel.py:1754, in Parallel._retrieve(self)
1747 while self._wait_retrieval():
1748
1749 # If the callback thread of a worker has signaled that its task
1750 # triggered an exception, or if the retrieval loop has raised an
1751 # exception (e.g. GeneratorExit), exit the loop and surface the
1752 # worker traceback.
1753 if self._aborting:
-> 1754 self._raise_error_fast()
1755 break
1757 # If the next job is not ready for retrieval yet, we just wait for
1758 # async callbacks to progress.

File ~/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/parallel.py:1789, in Parallel._raise_error_fast(self)
1785 # If this error job exists, immediately raise the error by
1786 # calling get_result. This job might not exists if abort has been
1787 # called directly or if the generator is gc'ed.
1788 if error_job is not None:
-> 1789 error_job.get_result(self.timeout)

File ~/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/parallel.py:745, in BatchCompletionCallBack.get_result(self, timeout)
739 backend = self.parallel._backend
741 if backend.supports_retrieve_callback:
742 # We assume that the result has already been retrieved by the
743 # callback thread, and is stored internally. It's just waiting to
744 # be returned.
--> 745 return self._return_or_raise()
747 # For other backends, the main thread needs to run the retrieval step.
748 try:

File ~/miniconda3/envs/flowsig/lib/python3.8/site-packages/joblib/parallel.py:763, in BatchCompletionCallBack._return_or_raise(self)
761 try:
762 if self.status == TASK_ERROR:
--> 763 raise self._result
764 return self._result
765 finally:

LinAlgError: SVD did not converge

Thank you in advance for your reply.

[YuNong-Qiu]

@jingc128 Hi, Have you solved this problem? I also got this error when run flowsig. Can I ask for the solution?

[fmulenge]

@axelalmet Hi, I also experienced some error with fs.tl.learn_intercellular_flows ( ) function as @jingc128. How can I fix the problem.
Thanks

[YuNong-Qiu]

I seem to have solved the problem, I changed the gem count from 20 to 30 and it worked.

Increase the number of decomposition of the previous build gem：
fs.pp.construct_gems_using_pyliger(adata,n_gems = 20, layer_key = 'counts', condition_key = 'batch')
↓
fs.pp.construct_gems_using_pyliger(adata,n_gems = 30, layer_key = 'counts', condition_key = 'batch')

Then try to run the following code without error.

[fmulenge]

@YuNong-Qiu Thank you for your suggestions. Actually the error is due to mismatch of condition_key variable in anndata and cellchat CSV file. I resolved it by changing all columns in anndata to categorical. It works

[YZY19980107]

I got same problem as you guys mentioned before, hope received suggestions and solved it ASAP!

[fmulenge]

@YZY19980107 Try the suggestions provided above. I hope these will resolve your issue

[YZY19980107]

@YZY19980107 Try the suggestions provided above. I hope these will resolve your issue

Oops.. it still hasn't been solved yet

[fmulenge]

@YZY19980107 was your data generated with scanpy or seurat?. The latter seems to give hardtime especially when converting seurat object to h5ad. Nevertheless, checkout that the metadata variables are all categoricals.

[YZY19980107]

@YZY19980107 was your data generated with scanpy or seurat?. The latter seems to give hardtime especially when converting seurat object to h5ad. Nevertheless, checkout that the metadata variables are all categoricals.

It works! Thanks a lot!

[jagger233r]

Perhaps you could check if your h5ad file has been normalized and log-transformed, and make sure that your layer has been processed. This might resolve your issue.

[axelalmet]

Hi everyone,

I'm very sorry for my late reply and I appreciate all of your comments and discussion points!

I have to say, jinc128, it's still strange that you get an SVD error because there's a step in FlowSig's run_utisgp function to explicitly remove any variables that have zero standard deviation in either the control or perturbed datasets, which I found was the most common reason for this occurring. The implication to me is that, as fmulenge alluded to, the condition key and control label may be misspecified.

YuNong-Qiu, it's very interesting that adding more GEMs solved the issue for you. The reason I say this is that, intuitively, for some datasets, I'd expect you to get more degenerate zero-standard-deviation GEMs if you set the number to be higher.

fmulenge, do you mind explaining the point, '... the error is due to mismatch of condition_key variable in anndata and cellchat CSV file' a bit more? I'd like to know for future trouble shooting of FlowSig.

And to your point about converting seurat to H5AD, I have personally found that it's easier to convert from Seurat -> SingleCellExperiment, and then from SingleCellExperiment -> Scanpy using zellkonverter, but maybe you've already done that. But yes, in general, the condition key should be categorical.

[fmulenge]

@axelalmet Thanks for sharing your insights and info. I meant the SVD error might be due to condition_key variable not been a categorical variable. This often happens when converting seurat to H5AD with SeuratDisk. The characters are converted to factors!. Nevertheless, to circumvent this I always use easySCF which ensures write operations are straightforward.