IPFS is likely not installed - error on Linux

[ProximaNova]

$ TZ=UTC ipwb replay Qm...
Using custom port 2016 for replay.
2023-10-05 03:08:57,999 [CRITICAL] ipwb: IPFS is likely not installed. See https://ipfs.io/docs/install/
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.3 LTS
Release:	22.04
Codename:	jammy
$

But I do have ipfs installed. I was thinking that ipwb was not seeing the right $IPFS_PATH. After a while I figured out why $IPFS_PATH was different in TTY or after SSHing in, compared to normal usage: because ~/.bash_profile and ~/.bashrc didn't have the same export IPFS_PATH="/pathto/.ipfs" line. I fixed that, but I still have this "IPFS is likely not installed" error. I don't know how to fix it.

[machawk1]

This error is invoked when a Python OSError occurs when checking if the IPFS daemon is alive/accessible. The default assumption is that the IPFS API will be available at /dns/localhost/tcp/5001/http.

@ProximaNova Is your IPFS daemon running when you get this error? If so, can you restart the daemon and paste the output here for debugging?

[ProximaNova]

4 days ago I ran that command with and without the ipfs daemon running in the background: both methods showed that error. Today:

$ ipfs daemon &
[1] 3932437
$ Initializing daemon...
Kubo version: 0.18.0
Repo version: 13
System version: amd64/linux
Golang version: go1.19.1
[...]
API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready[\]
$ TZ=UTC ipwb replay /bin/ab
Using custom port 2016 for replay.
2023-10-09 02:45:16,694 [CRITICAL] ipwb: IPFS is likely not installed. See https://ipfs.io/docs/install/
$ ipfs stats bw
Bandwidth
TotalIn: 5.7 MB
TotalOut: 3.3 MB
RateIn: 176 kB/s
RateOut: 29 kB/s
$

(I don't think it matters that I specified "/bin/ab".) I should have thought of this earlier: maybe updating to a newer version of ipfs or Kubo will fix it.

[machawk1]

I tried to partially replicate this issue without success just yet.

machawk1@Mat-mini ipwb % ipfs daemon &    
[1] 51559
Initializing daemon...
Kubo version: 0.18.0
Repo version: 13
System version: amd64/darwin
Golang version: go1.19.1

Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "4.3 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 30720

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1
Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiBTinDOaTtceU8Xlzta7YxxrUMtL4bk-DavANiNyoK8Ig/certhash/uEiBE8TpLm9JI2RjHkkjs4QzjbI1-XkxkZVLXYcgrmmQGsA
Swarm listening on /ip4/192.168.1.173/tcp/4001
Swarm listening on /ip4/192.168.1.173/udp/4001/quic
Swarm listening on /ip4/192.168.1.173/udp/4001/quic-v1
Swarm listening on /ip4/192.168.1.173/udp/4001/quic-v1/webtransport/certhash/uEiBTinDOaTtceU8Xlzta7YxxrUMtL4bk-DavANiNyoK8Ig/certhash/uEiBE8TpLm9JI2RjHkkjs4QzjbI1-XkxkZVLXYcgrmmQGsA
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /ip6/::1/udp/4001/quic
Swarm listening on /ip6/::1/udp/4001/quic-v1
Swarm listening on /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiBTinDOaTtceU8Xlzta7YxxrUMtL4bk-DavANiNyoK8Ig/certhash/uEiBE8TpLm9JI2RjHkkjs4QzjbI1-XkxkZVLXYcgrmmQGsA
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/127.0.0.1/udp/4001/quic
Swarm announcing /ip4/127.0.0.1/udp/4001/quic-v1
Swarm announcing /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiBTinDOaTtceU8Xlzta7YxxrUMtL4bk-DavANiNyoK8Ig/certhash/uEiBE8TpLm9JI2RjHkkjs4QzjbI1-XkxkZVLXYcgrmmQGsA
Swarm announcing /ip4/192.168.1.173/tcp/4001
Swarm announcing /ip4/192.168.1.173/udp/4001/quic
Swarm announcing /ip4/192.168.1.173/udp/4001/quic-v1
Swarm announcing /ip4/192.168.1.173/udp/4001/quic-v1/webtransport/certhash/uEiBTinDOaTtceU8Xlzta7YxxrUMtL4bk-DavANiNyoK8Ig/certhash/uEiBE8TpLm9JI2RjHkkjs4QzjbI1-XkxkZVLXYcgrmmQGsA
Swarm announcing /ip6/::1/tcp/4001
Swarm announcing /ip6/::1/udp/4001/quic
Swarm announcing /ip6/::1/udp/4001/quic-v1
Swarm announcing /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiBTinDOaTtceU8Xlzta7YxxrUMtL4bk-DavANiNyoK8Ig/certhash/uEiBE8TpLm9JI2RjHkkjs4QzjbI1-XkxkZVLXYcgrmmQGsA
API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready

then with a multihash for a file I recently added to IPFS:

% TZ=UTC ipwb replay QmT8...1TpxW
Using custom port 2016 for replay.
IPWB replay started on http://localhost:2016
 * Serving Flask app 'ipwb.replay'
 * Debug mode: off

[machawk1]

@ProximaNova With the IPFS daemon running, in a web browser can you first visit IPFS WebUI at http://127.0.0.1:5001/webui then try to access it at http://localhost:5001/webui and let me know if it is accessible at either or both locations?

The hypothesis here is that 127.0.0.1 is not mapped to localhost on your system. If ipwb is expecting the IPFS API to be available at an address that involves localhost, that might be a false assumption that we have with ipwb and needs to be corrected.

[ProximaNova]

Both URLs are reachable in under 30sec (ran the first command a couple times, quickest on the newest run):

$ ipfs daemon &
[1] 4127305
$ Initializing daemon...
Kubo version: 0.18.0
[...]
$ utc; TZ=UTC wget --spider http://127.0.0.1:5001/webui; utc
Tue, 10 Oct 2023 10:36:59 +0000
Spider mode enabled. Check if remote file exists.
--2023-10-10 10:36:59--  http://127.0.0.1:5001/webui
Connecting to 127.0.0.1:5001... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /webui/ [following]
Spider mode enabled. Check if remote file exists.
--2023-10-10 10:36:59--  http://127.0.0.1:5001/webui/
Connecting to 127.0.0.1:5001... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ipfs/bafybeiequgo72mrvuml56j4gk7crewig5bavumrrzhkqbim6b3s2yqi7ty [following]
Spider mode enabled. Check if remote file exists.
--2023-10-10 10:36:59--  http://127.0.0.1:5001/ipfs/bafybeiequgo72mrvuml56j4gk7crewig5bavumrrzhkqbim6b3s2yqi7ty
Connecting to 127.0.0.1:5001... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /ipfs/bafybeiequgo72mrvuml56j4gk7crewig5bavumrrzhkqbim6b3s2yqi7ty/ [following]
Spider mode enabled. Check if remote file exists.
--2023-10-10 10:36:59--  http://127.0.0.1:5001/ipfs/bafybeiequgo72mrvuml56j4gk7crewig5bavumrrzhkqbim6b3s2yqi7ty/
Connecting to 127.0.0.1:5001... connected.
HTTP request sent, awaiting response... 200 OK
Length: 725 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

Tue, 10 Oct 2023 10:37:25 +0000
$ utc; TZ=UTC wget --spider http://localhost:5001/webui; utc
Tue, 10 Oct 2023 10:37:36 +0000
Spider mode enabled. Check if remote file exists.
--2023-10-10 10:37:36--  http://localhost:5001/webui
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:5001... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /webui/ [following]
Spider mode enabled. Check if remote file exists.
--2023-10-10 10:37:36--  http://localhost:5001/webui/
Connecting to localhost (localhost)|127.0.0.1|:5001... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ipfs/bafybeiequgo72mrvuml56j4gk7crewig5bavumrrzhkqbim6b3s2yqi7ty [following]
Spider mode enabled. Check if remote file exists.
--2023-10-10 10:37:36--  http://localhost:5001/ipfs/bafybeiequgo72mrvuml56j4gk7crewig5bavumrrzhkqbim6b3s2yqi7ty
Connecting to localhost (localhost)|127.0.0.1|:5001... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /ipfs/bafybeiequgo72mrvuml56j4gk7crewig5bavumrrzhkqbim6b3s2yqi7ty/ [following]
Spider mode enabled. Check if remote file exists.
--2023-10-10 10:37:36--  http://localhost:5001/ipfs/bafybeiequgo72mrvuml56j4gk7crewig5bavumrrzhkqbim6b3s2yqi7ty/
Connecting to localhost (localhost)|127.0.0.1|:5001... connected.
HTTP request sent, awaiting response... 200 OK
Length: 725 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

Tue, 10 Oct 2023 10:37:38 +0000
$ ipwb replay /bin/anytopnm
Using custom port 2016 for replay.
[...] [CRITICAL] ipwb: IPFS is likely not installed. See https://ipfs.io/docs/install/
$ ipfs stats bw
Bandwidth
TotalIn: 3.8 MB
TotalOut: 3.8 MB
RateIn: 14 kB/s
RateOut: 56 kB/s
$ ipfs shutdown

[machawk1]

Thanks again for the report, @ProximaNova. Can you run ipwb --version and comment on what version of ipwb you are using? e.g.,

% ipwb --version
InterPlanetary Wayback 0.2023.07.20.2112

I will try to replicate this on an Ubuntu system but also want to ping @ibnesayeed here, as he is knowledgeable about detecting network resolution issues.

[ProximaNova]

$ $HOME/.local/bin/ipwb --version
InterPlanetary Wayback 0.2023.08.16.1719
$

[ProximaNova]

Still getting this error, after...

• updated ipfs/kubo: "ipfs version 0.23.0".
• setting IPFS_PATH to a repo with only like 1GB of data

Possible fix or maybe related:

• update ipwb
• I use NextDNS, so maybe localhost settings or ports or something is conflicting. Really don't know, but just putting this out there. Basically everything else works regarding localhost. And maybe this is not related to the problem at all: ipwb not detecting that I have software installed.

[ProximaNova]

Where's this error coming from? Probably "./ipwb/build/lib/ipwb/util.py", which has that text. I was doing a bit of debugging on this by editing "util.py" then running "pip install ./" to reinstall. Added ipfs_config_path = os.path.join(os.environ.get('IPFS_PATH'), 'config') to def check_daemon_is_alive() then changed a line to be 'IPFS is likely not installed. See https://ipfs.io/docs/install/ ' + ipfs_config_path,. I then ran "ipwb index whatever" and it said ... [CRITICAL] ipwb: IPFS is likely not installed. See https://ipfs.io/docs/install/ $HOME/.config/BraveSoftware/Brave-Browser/brave_ipfs/config. So it correctly got $IPFS_PATH, just a problem with whatever else. I commented out these 4 lines: except OSError as err:[...]See https://ipfs.io/docs/install/',) from err then ran "ipwb index whatever" which said [today] [CRITICAL] ipwb: Unknown error in retrieving IPFS daemon status. from text below the "IPFS is likely not installed." part in util.py. So the next idea would be searching for whatever connects to "except OSError as err" and "except Exception as err". The code is

def check_daemon_is_alive():
    """Ensure that the IPFS daemon is running via HTTP before proceeding"""
    client = ipfs_client()
    daemonMultiaddr = settings.App.config("ipfsapi")
    ipfs_config_path = os.path.join(os.environ.get('IPFS_PATH'), 'config')

    try:
        # ConnectionError/AttributeError if IPFS daemon not running
        client.id()
        return True

    except ConnectionError as err:
        raise IPFSDaemonNotAvailable(
            f'Daemon is not running at: {daemonMultiaddr}',
        ) from err

    except OSError as err:
        raise IPFSDaemonNotAvailable(
            'IPFS is likely not installed. See https://ipfs.io/docs/install/',
        ) from err

    except Exception as err:
        raise IPFSDaemonNotAvailable(
            'Unknown error in retrieving IPFS daemon status.',
        ) from err

those are 3 errors that could happen if "client.id()" and whatever fails.

I maybe cannot install ipwb on a computer without Internet access:

$ git clone https://github.com/oduwsdl/ipwb
$ # ...I then sftp'd the files to a computer with zero Internet access...
$ cd ipwb; pip install ./
[...]
Defaulting to user installation because normal site-packages is not writeable
Processing $HOME/ipwb
  Preparing metadata (setup.py) ... done
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6eb55aa920>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/flask/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f6eb55a9300>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/flask/
[same warning 3 more times]
ERROR: Could not find a version that satisfies the requirement Flask==2.3.2 (from ipwb) (from versions: none)
ERROR: No matching distribution found for Flask==2.3.2
$ # or it would work if I had all of the dependencies.

Tried installing ipwb on a 32-bit computer:

$ sudo apt install python3-pip
[...]After this operation, 114 MB of additional disk space will be used.[...]
[taking a long time or stuck at "Unpacking libstdc++-4.8-dev:i386 (4.8.2-19ubuntu1) ..."]
[Update: it got past that. ...]
Error: Timeout was reached
$

I ran "sudo apt install python3-pip" again, but it seemingly got stuck (no output).

[ProximaNova]

Hmm, maybe it is a DNS issue? File "util.py" / running "ipwb index whatever":
settings.App.config("ipfsapi") -> f'{daemonMultiaddr}' -> /dns/localhost/tcp/5001/http

Part of "ipfs config show":

  "DNS": {
    "Resolvers": {
      ".": "https://chromium.dns.nextdns.io"
    }
  },

Don't know if any text in this post helps at all. Also commented out the two parts "except OSError as err..." and "except Exception as err..." then ran "ipwb index whatever" after reinstalling and it said [today] [time] [CRITICAL] ipwb: Failed to parse: http://localhost:5001/api/v0/id.

no output

It ended up saying this: fatal error: runtime: out of memory \\ runtime stack: [...]. Some things changed then I ran "sudo apt install python3-pip" again and it said it was already installed. Does "pip install ipwb" work? Will see.

[ProximaNova]

With $IPFS_PATH being set to "$HOME/.config/BraveSoftware/Brave-Browser/brave_ipfs" I got "failed: Connection refused" from "wget --spider http://127.0.0.1:5001/webui" and "wget --spider http://localhost:5001/webui". Even if it wasn't "Connection refused", that wouldn't matter a lot, for as seen above it still didn't work. If Brave's IPFS API isn't at /dns/localhost/tcp/5001/http, then where is it? http://localhost:45005/ipfs/bafybeiamycmd52xvg6k3nzr6z3n33de6a2teyhquhj4kspdtnvetnkrfim/#/settings says "API ADDRESS"="http://localhost:45005/", so run "ipwb --daemon /dns/localhost/tcp/45005/http index whatever" I think.

Will see.

To install pip I had to run "sudo apt-get install python-pip".

[ProximaNova]

This mysteriously is no longer a problem. It now works or works better than before. Only change that happened on that Ubuntu computer: I uninstalled python and/or python3 for an unrelated reason by running "sudo apt remove python3". I don't recommend that anyone tries to uninstall python/python3 because it likely will break things; only uninstall it if you really know what you are doing. (And you can't really remove python as I have heard that it is hard-coded into the kernel/OS, so "uninstalling python" in Linux will just delete important programs and stuff.) One of the big things that doing that broke for me was GNOME, so I installed i3 as a dm/wm instead. Anyways, after reinstalling stuff such as pip, I "reinstalled" ipwb by running "pip install ipwb". It can now successfully index: ipwb -d /ip4/127.0.0.1/tcp/45005 index a02-00000.warc.gz >> f1.cdxj and if I don't post any update then assume it can also successfully replay.

Update: replay does work. (Saw this "nonissue" error: /home/ubuntu/.local/lib/python3.10/site-packages/ipfshttpclient/client/__init__.py:75: VersionMismatch: Unsupported daemon version '0.23.0' (not in range: 0.5.0 ≤ … < 0.9.0))

I suspect that ipwb dev(s) have got this suggestion before, but could you make a thing that does the following? An option for "ipwb index" that adds everything added from .warc under one pin instead of many pins. Use: for those who want to have as few CIDs in their pinset as possible. It wouldn't matter to those who don't care about having hundreds/thousands of pins. For now, I did this:

1. edit .cdxj with vim to just get CIDs, on per line, save to a different file (don't overwrite the CDXJ)
2. run cat f1.cdxj.cid | xargs -d "\n" sh -c 'for args do ipfs files cp /ipfs/$args /a/f1cid/$args && ipfs pin rm $args; done' _
3. run ipfs files ls --long /a then pin the CID you see for that

[machawk1]

Hi @ProximaNova, since you were able to overcome the original problem on your system, I am going to close this issue.

I have created a separate ticket for your suggestions at #830. Please feel free to expound on the idea there.