Getting an error when using ObjectStorage with DVC

Hey there,

So that we can get a better idea of where the error is occurring, can I have you run some commands and post some additional output for us that might help? The commands are listed below.

Please sanitize any sensitive information that may be included in the output.

cat ~/.dvc/config
cat ~/.dvc/config.local ##You may not have one of these
dvc push -v

If I find anything else that might help, I'll be sure to reach back out.

Ryan L.
Linode Support Staff

dbeck@merry /proj/piper (develop) $ cat .dvc/config
[core]
remote = myremote
['remote "myremote"']
url = s3://kmlus-sld-dvc/piper
endpointurl = https://us-east-1.linodeobjects.com/kmlus-sld-dvc

dbeck@merry /proj/piper (develop) $ dvc push -v | tee log
2020-06-19 10:57:26,717 DEBUG: PRAGMA user_version;
2020-06-19 10:57:26,718 DEBUG: fetched: [(3,)]
2020-06-19 10:57:26,718 DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
2020-06-19 10:57:26,719 DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
2020-06-19 10:57:26,719 DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
2020-06-19 10:57:26,719 DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
2020-06-19 10:57:26,719 DEBUG: PRAGMA user_version = 3;
2020-06-19 10:57:27,074 DEBUG: Preparing to upload data to 's3://kmlus-sld-dvc/piper'
2020-06-19 10:57:27,074 DEBUG: Preparing to collect status from s3://kmlus-sld-dvc/piper
2020-06-19 10:57:27,075 DEBUG: Collecting information from local cache…
2020-06-19 10:57:27,077 DEBUG: Assuming '/proj/piper/.dvc/cache/a3/04afb96060aad90176268345e10355' is unchanged since it is read-only
2020-06-19 10:57:27,078 DEBUG: Assuming '/proj/piper/.dvc/cache/e0/499b0a047d8aecfcb725885837951e' is unchanged since it is read-only
2020-06-19 10:57:27,079 DEBUG: Collecting information from remote cache…
2020-06-19 10:57:27,080 DEBUG: Matched '0' indexed checksums
2020-06-19 10:57:27,407 DEBUG: SELECT count from state_info WHERE rowid=?
2020-06-19 10:57:27,408 DEBUG: fetched: [(3611,)]
2020-06-19 10:57:27,408 DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?

2020-06-19 10:57:27,465 ERROR: unexpected error - An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: Unknown

Traceback (most recent call last):
File "/snap/dvc/388/lib/python3.6/site-packages/dvc/main.py", line 49, in main
ret = cmd.run()
File "/snap/dvc/388/lib/python3.6/site-packages/dvc/command/data_sync.py", line 55, in run
recursive=self.args.recursive,
File "/snap/dvc/388/lib/python3.6/site-packages/dvc/repo/init__.py", line 30, in wrapper ret = f(repo, *args, **kwargs) File "/snap/dvc/388/lib/python3.6/site-packages/dvc/repo/push.py", line 27, in push return self.cloud.push(used, jobs, remote=remote) File "/snap/dvc/388/lib/python3.6/site-packages/dvc/data_cloud.py", line 66, in push show_checksums=show_checksums, File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/base.py", line 80, in wrapper return f(remote_obj, *args, **kwargs) File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/local.py", line 567, in push download=False, File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/local.py", line 475, in _process download=download, File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/local.py", line 334, in _status md5s, jobs=jobs, name=str(remote.path_info) File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/base.py", line 919, in cache_exists checksums, name File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/base.py", line 1008, in _estimate_cache_size remote_checksums = set(checksums) File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/base.py", line 962, in _checksums_with_limit for checksum in self.cache_checksums(prefix, progress_callback): File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/base.py", line 735, in cache_checksums for path in self.list_cache_paths(prefix, progress_callback): File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/s3.py", line 219, in <genexpr> item["Key"] File "/snap/dvc/388/lib/python3.6/site-packages/dvc/remote/s3.py", line 206, in _list_objects for page in paginator.paginate(kwargs): File "/snap/dvc/388/lib/python3.6/site-packages/botocore/paginate.py", line 255, in __iter
response = self._make_request(current_kwargs)
File "/snap/dvc/388/lib/python3.6/site-packages/botocore/paginate.py", line 332, in _make_request
return self._method(</genexpr>current_kwargs)
File "/snap/dvc/388/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/snap/dvc/388/lib/python3.6/site-packages/botocore/client.py", line 626, in _make_api_call
raise error_class(parsed_response, operation_name)

botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjectsV2 operation: Unknown

endpointurl = https://us-east-1.linodeobjects.com/kmlus-sld-dvc

Not sure if this is relevant, but usually in the endpoint URL, you don't need the bucket name (I'm assuming "kmlus-sld-dvc" is your bucket name.)

The endpoint URL is the API/authentication endpoint, which is the cluster instance.

What about:

endpointurl = https://us-east-1.linodeobjects.com

That does seem to make things work, I thought I had tried that before also but obviously not.