How do I upload a file to Object Storage using python?
Hello,
I try to access the object storage using python. However, I always receive the following error message:
Traceback (most recent call last):
File "C:/dev/python-playground/s3sync/Test.py", line 16, in <module>
print(s3.bucket_list('my-bucket'))
File "C:\Python38\lib\site-packages\S3\S3.py", line 321, in bucket_list
for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit):
File "C:\Python38\lib\site-packages\S3\S3.py", line 355, in bucket_list_streaming
response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params, max_keys)
File "C:\Python38\lib\site-packages\S3\S3.py", line 386, in bucket_list_noparse
response = self.send_request(request)
File "C:\Python38\lib\site-packages\S3\S3.py", line 1334, in send_request
handler_fn = self._http_400_handler(request, response, self.send_request, request)
File "C:\Python38\lib\site-packages\S3\S3.py", line 1215, in _http_400_handler
return fn(*args, **kwargs)
File "C:\Python38\lib\site-packages\S3\S3.py", line 1344, in send_request
raise err
S3.Exceptions.S3Error: 400 (InvalidArgument)
I use this script:
from S3.Config import Config
from S3.S3 import S3
access_key = 'abc'
secret_key = 'abcde'
cfg = Config(None, access_key, secret_key)
cfg.host_base = 'eu-central-1.linodeobjects.com'
cfg.bucket_location = 'EU'
cfg.host_bucket = 'my-bucket.eu-central-1.linodeobjects.com'
cfg.website_endpoint = 'http://my-bucket.eu-central-1.linodeobjects.com'
cfg.use_https = False
s3 = S3(cfg)
print(s3.bucket_list('my-bucket'))
Can you please tell me what's wrong?
4 Replies
rgerke
Linode Staff
Hey there -
I did some consulting with our resident Object Storage experts, and they recommend that you try doing this with boto/boto3, which are the official Python libraries.
Here is a doc that shows you how to do this. If you follow this, you should be able to get this working:
This also doesn’t quite look right:
cfg.bucket_location = 'EU'
Should this be
cfg.bucket_location = 'eu-central-1'
?
Do you get any more information in your error message, a HTTP 400 error means something in the request was wrong. Most AWS libraries give detailed reasons what’s wrong.
Hello and thank your for the answers.
Unfortunately, I do not have any more error messages than that one above.
I got it to work by first running "python3 s3cmd --configure" in the "Scripts" folder of the python installation and filled in the data according to this documentation: https://www.linode.com/docs/platform/object-storage/how-to-use-object-storage/#install-and-configure-s3cmd
After that, I could connect using the above approach (of course without the need to configure anything before connecting).
Since s3cmd is just one file with a lot of python functions, I take this as reference now. However, I did not dig deep into it for now.
I was able to get boto3 to link directly as follows:
AWS_S3_ENDPOINT_URL = f'https://{LINODE_BUCKET_REGION}.linodeobjects.com'
import boto3
s3 = boto3.client('s3',
region_name = LINODE_BUCKET_REGION,
endpoint_url = AWS_S3_ENDPOINT_URL,
aws_access_key_id = LINODE_BUCKET_ACCESS_KEY,
aws_secret_access_key = LINODE_BUCKET_SECRET_KEY)
response = s3.list_buckets()
The trick was to put in the endpoint_url. It took ages to figure this out but had seen Wasabi had some boto3 guidance and turns out it works here too.