Copy files from blob container to another container using python

Question

I am trying to copy 'specific files' from one folder to another. when I am trying to use Wild card operator (*) at the end, the copy does not happen.

But if I provide just the folder name, then all the files from this source folder are copied to target folder without any issues.

Problem: File copy does not happen when Wild card operator is used. Can you please help me to fix the problem?

def copy_blob_files(account_name, account_key, copy_from_container, copy_to_container, copy_from_prefix):
  try:
    blob_service = BlockBlobService(account_name=account_name, account_key=account_key)
    files = blob_service.list_blobs(copy_from_container, prefix=copy_from_prefix)

    for f in files:
      #print(f.name)
      blob_service.copy_blob(copy_to_container, f.name.replace(copy_from_prefix,""), f"https://{account_name}.blob.core.windows.net/{copy_from_container}/{f.name}")
  except:
    print('Could not copy files from source to target')

copy_from_prefix = 'Folder1/FileName_20191104*.csv'
copy_blob_files (accountName, accesskey, copy_fromcontainer, copy_to_container, copy_from_prefix)

Ivan Glasenberg · Accepted Answer · 2019-11-05 05:47:13Z

The copy_blob method does not support wildcard.

1.If you want to copy specified pattern of blobs, you can filter the blobs in list_blobs() method with prefix(it also does not support wildcard). In your case, the prefix looks like copy_from_prefix = 'Folder1/FileName_20191104', note that there is no wildcard.

The code below works at my side, and all the specified pattern files are copies and blob name replaced:

from azure.storage.blob import BlockBlobService

account_name ="xxx"
account_key ="xxx"

copy_from_container="test7"
copy_to_container ="test4"

#remove the wildcard
copy_from_prefix = 'Folder1/FileName_20191104'

def copy_blob_files(account_name, account_key, copy_from_container, copy_to_container, copy_from_prefix):
    try:
        block_blob_service = BlockBlobService(account_name,account_key)
        files = block_blob_service.list_blobs(copy_from_container,copy_from_prefix)
        for file in files:
            block_blob_service.copy_blob(copy_to_container,file.name.replace(copy_from_prefix,""),f"https://{account_name}.blob.core.windows.net/{copy_from_container}/{file.name}")

    except:
        print('could not copy files')

copy_blob_files(account_name,account_key,copy_from_container,copy_to_container,copy_from_prefix)

2.Another way as others mentioned, you can use python to call azcopy(you can use azcopy v10, which is just a .exe file). And for using wildcard in azcopy, you can follow this doc. Then you write you own azcopy command, at last, write your python code as below:

import subprocess

#the path of azcopy.exe, v10 version
exepath = "D:\\azcopy\\v10\\azcopy.exe"

myscript= "your azcopy command"

#call the azcopy command
subprocess.call(myscript)

Thank you so much for your inputs. I like the first option and I tested with my files. It worked well :)

Shiraz Bhaiji · Accepted Answer · 2019-11-04 19:15:48Z

0

AzCopy supports wildcards, you could excute AzCopy from your Python code.

An example of how to do this can be found here: How to run Azure CLI commands using python?

answered Nov 4, 2019 at 19:15

Shiraz Bhaiji

65.7k37 gold badges149 silver badges260 bronze badges

1 Comment

Idleguys Over a year ago

I doubt if this is available in azure sdk..github.com/Azure/azure-sdk-for-python/issues/6306

Collectives™ on Stack Overflow

Copy files from blob container to another container using python

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related