Can't install ipfx library - python 3.9.13 Jupyter Notebook

Technical
1

Hi there,

I’m trying to load the ipfx library into python version 3.9.13 (jupyter notebook) but I keep getting a red-block errors “error: subprocess exited with error” I’m not sure why it’s not working. I’m using pip install. At the end of the large red-block error I get a message saying “error: metadata-generation-failed.” After reaching out over Twitter, I’ve attached the images of the red block error. The entire error message couldn’t fit into one image, but here is the end of the message - describing an error while generating metadata. Any help would be greatly appreciated thank you!

ipfx error 5
ipfx error 51286×568 20.9 KB

1 Like
2

It looks like you need to compile a library for your platform (Windows), but you don’t have the C++ build tools installed. You can get them from the link in the middle of the error message (in blue text).

Alternatively, if you’re using Anaconda, you could try to install NumPy using that first (conda install numpy), which might help you avoid needing to install the build tools to satisfy that dependency.

3

Thanks for the prompt help! I tried doing a pip install of numpy, it’s version 1.21.0, and I’m still getting the red-block error re: metadata generation. I don’t think I’m understanding correctly. Am I to use the link in the error message to access C++ to satisfy the dependency?

4

Yes, pip install numpy will have the same problem (since that’s basically what is being called when you are pip installing ipfx).

The error you’re seeing is because you need to compile things for NumPy to work, but you haven’t installed a C++ compiler. That’s what following the link in the error message would get you.

If you use Anaconda, though, you can get precompiled binaries of some packages. That means someone else has already compiled the package for a system similar to yours and made it available. So if you have installed Anaconda and use “conda” instead of “pip” to install NumPy, you can get around the need to install a C++ compiler (at least for that package).

5

I tried a conda install for numpy, however I’m still getting the red-block error. To be perfectly honest, I’m not sure how to go about installing a C++ compiler, or whether there’s a specific one I’d need among many, (I’ve never worked with C++, so I’m not sure about how it interfaces with python). If I were to install a compiler, admittedly, I’d be out of my depth as to what to do next. I really appreciate all your help thus far!

Envoyé de mon iPhone

6

I also tried installing the C++ compiler from the visual studio installer. I’m still getting the metadata generation failed error.

Envoyé de mon iPhone

7

It’s a little hard to troubleshoot your specific issue on my end since I don’t have access to a Windows machine. You say you are getting a metadata generation failed error every time - it would also be helpful to know if you’re getting the exact same error after trying those things (i.e., it is still failing on building library "npymath" sources and asking you to install Visual C++ 14.0 or greater), or if it’s failing in the same stage but with a different specific issue.

From what you’ve shown, this doesn’t look like an IPFX issue per se; it seems more like a problem with getting NumPy installed. However, you said you were working in Jupyter notebook - if that’s installed, it seems likely you might have some version of NumPy already installed (and maybe one that is newer than what IPFX or one of its dependencies expects). In Python, can you run import numpy, and if that works, see what the output of numpy.__version__ is?

If the issue is with trying to install an older NumPy (or other packages) on your system, you could potentially work around that by installing IPFX without dependencies (pip install --no-deps ipfx) and install other needed dependencies you don’t already have on your own.

8

When I tried installing ipfx, I would get the same metadata failure error after trying each of the proposed workarounds. I also believe the version of numpy is what’s causing the issue. I have numpy version 1.21.5. Could it also be the version of python? I have version 3.9.13 installed.

Envoyé de mon iPhone

9

Okay, it is confusing that you would still get a message asking you to install a C++ compiler after you have already installed a C++ compiler. But it seems likely that overall what is happening is that you have a newer version of NumPy and pip wants to downgrade you based on IPFX’s stated dependency requirements.

I would try to install without dependencies (pip install --no-deps ipfx) and see if that is successful. You may need to install other things afterwards to actually get IPFX to run, but you can at least see if that operation will complete successfully.

10

I’ve completed the pip install with no dependencies - it loads successfully. Dandi also loads successfully. I tried following the ephys analysis step in the model jupyter notebook on my machine, and I notice I’m still getting some errors.

  1. when trying to retrieve the nwb file, I’m getting a “FileExistsError”
  2. ipfx modules are not found despite the installation. Having installed ipfx without dependencies, what packages do I need to install next?
    I’ve attached a screenshot where the errors arise. I really appreciate all your help throughout this process :slight_smile:

image.png1480×578 55 KB

11

The submodule ipfx.dataset doesn’t have an underscore - the line that’s giving an error should be from ipfx.dataset.create import create_ephys_data_set.

12

Thank you for catching that! I’ve corrected the structures, but now I’m getting the following red-block error. I’m assuming these are the dependencies I need to load to get ipfx to work, and because some versions of these packages installed on my machine are “incompatible” I would need to downgrade them?

image.png1258×400 21.6 KB

13

Sometimes a dependency is specified but is more restrictive than what would actually work. For example, I have numpy 1.20.3 in my current working environment, but IPFX is still functional.

You will need some of those packages for IPFX to run, but you can also install them with the --no-deps option to avoid having to do things like downgrade NumPy.

What lines of code are you running that is generating that error message? It looks like a pip error message, but if you’ve already installed IPFX, I’m not sure under what circumstances you’re running pip again.

14

The pip error message appeared after I tried running the lines in the earlier photo - running the ipfx dataset libraries. I’m not sure why that is, considering, that that is separate from running pip again.

15

Yeah, I don’t know how that would happen - is there more of the message missing?

16

I restarted the Kernel, and then closed and reopened everything. Sometimes I find that clears things up. I did get the same error message when trying to run ipfx. Again, it was after I tried running the import lines - with punctuation corrected. As I was following the model Jupyter notebook. I noticed there was a file “select_markers” that I can’t seem to find anywhere. Would you happen to know where I would be able to find that file. I’d like to try recreating the umap to explore the transcriptomic data. Thank you again for all the help!

17

You could try to simplify the situation to try to figure out what’s going on. I’d try running something minimal outside of Jupyter to see if that clarifies the issue. You can open a Python prompt from a terminal/console, and just try to import something from ipfx.

I created a new environment and installed just the packages that your error message suggests you have installed (h5py, numpy, pandas, pynwb, pyyaml), as well as installing ipfx (with the --no-deps) option. When I run just the from ipfx.dataset.create import create_ephys_data_set line, this is what I get:

Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ipfx.dataset.create import create_ephys_data_set
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/local1/anaconda3/envs/test_ipfx_empty/lib/python3.9/site-packages/ipfx/dataset/create.py", line 8, in <module>
    import allensdk.core.json_utilities as ju
ModuleNotFoundError: No module named 'allensdk'
>>>

So this would mean that I’d need to install allensdk (likely using the --no-deps option in order to not deal with downgrading issues), try it again, and then see if there are additional packages I need to install. But I’m not seeing anything from pip when I run it in this minimal way.

Here is a thread about creating a select_markers file: Question on PatchSeq jupyter example

You could also take something like this file of marker genes and turn it into a text file with a single gene name on each line:

That probably would get you fewer genes than you’d find by following the procedure in the linked thread, but it still would likely make a reasonable UMAP.