The osfclient is a Python module and a command-line program for implementing simple actions in OSF, such as uploading files and downloading projects.
This notebook demonstrates the use of osfclient's command-line interface* and Python library by using an OSF Project created specifically to be used in examples here.
Make sure to check out the documentation for osfclient.
Extensive documentation for OSF is available through the OSF Guides. For a quick introduction, check out the OSF page of the Brain Circuits Cluster Data Management White Paper.
osfclient
is no longer an active project (see this comment from a former maintainer) so I created a fork of this repository to fix bugs and make improvements.
To install my version of osfclient (which is used in this notebook), run the following command:
pip install git+https://github.com/BrainUBC/osfclient
This particular command can be executed in a Jupyter Notebook (run the cell below!) by adding !
.
! osf -h
usage: osf [-h] [-u USERNAME] [-p PROJECT] [-v] {clone,init,fetch,list,ls,upload,remove,rm} ... osf is a command-line program to up and download files from osf.io. These are common osf commands: init Set up a .osfcli.config file clone Copy all files from all storages of a project fetch Fetch an individual file from a project list List all files from all storages for a project upload Upload a new file to an existing project remove Remove a file from a project's storage See 'osf <command> -h' to read about a specific command. positional arguments: {clone,init,fetch,list,ls,upload,remove,rm} optional arguments: -h, --help show this help message and exit -u USERNAME, --username USERNAME OSF username. Provide your password via OSF_PASSWORD environment variable -p PROJECT, --project PROJECT OSF project ID -v, --version show program's version number and exit
Note: The OSF project ID is contained within the URL of the project. The project ID of the example is qf9n8
as given by its URL, https://osf.io/qf9n8/.
list
¶! osf list -h
usage: osf list [-h] list() -> new empty list list(iterable) -> new list initialized from iterable's items optional arguments: -h, --help show this help message and exit
The following command provides a list of files of a project:
osf -p qf9n8 list
Notice the -p
argument followed by the project ID.
Output:
github\osf_addon.txt
osfstorage\osf/osf_3.txt
osfstorage\osf/osf_2.txt
osfstorage\osf/osf_1.txt
osfstorage\osf.txt
osfstorage\folder/osf_folder.txt
Note that list
only provides files within add-ons and within the OSF Storage of the project. It does not list files within components.
init
¶! osf init -h
usage: osf init [-h] Initialize or edit an existing .osfcli.config file. optional arguments: -h, --help show this help message and exit
The init
command initializes a configutation file .osfcli.config
in the current directory.
osf init
This gives the following prompts (I included my inputs for clarity):
Provide a username for the config gile [current username: ]:
g.alejo@alumni.ubc.ca
Provide a project for the config file [current project: ]:
qf9n8
The configuration file sets default values for username and project so you won't need to specify these arguments when you run commands. For example, the list
command from a few cells above becomes:
osf list
clone
¶! osf clone -h
usage: osf clone [-h] [-U] [output] Copy all files from all storages of a project. The output directory defaults to the current directory. If the project is private you need to specify a username. If args.update is True, overwrite any existing local files only if local and remote files differ. positional arguments: output Write files to this directory optional arguments: -h, --help show this help message and exit -U, --update Overwrite only if local and remote files differ
This is the download command - it creates a copy of a project or component.
osf clone
If you've already created a configuration file, the command above will clone the project you've specified in that file. To clone other projects/components, use the same -p
argument as before.
osf -p q5mt7 clone \Users\User\
This command creates a copy of component
(component ID = q5mt7
). A folder named after the storage type osfstorage
appears within the specified directory containing a single file osf_component.txt
.
If an output
argument is not specified, a folder named after the project or component ID is created in the working directory with storages as subfolders.
A couple of things to note:
upload
¶! osf upload -h
usage: osf upload [-h] [-f] [-U] [-r] source destination Upload a new file to an existing project. The first part of the remote path is interpreted as the name of the storage provider. If there is no match the default (osfstorage) is used. If the project is private you need to specify a username. To upload a whole directory (and all its sub-directories) use the `-r` command-line option. If your source directory name ends in a / then files will be created directly in the remote directory. If it does not end in a slash an extra sub-directory with the name of the local directory will be created. To place contents of local directory `foo` in remote directory `bar/foo`: $ osf upload -r foo bar To place contents of local directory `foo` in remote directory `bar`: $ osf upload -r foo/ bar positional arguments: source Local file destination Remote file path optional arguments: -h, --help show this help message and exit -f, --force Force overwriting of remote file -U, --update Overwrite only if local and remote files differ -r, --recursive Recursively upload entire directories
The command below uploads a text file called test.txt
(source
argument) from my current directory into the OSF Storage of the project and renames it as osf.txt
(destination
argument).
osf -p qf9n8 -u g.alejo@alumni.ubc.ca upload test.txt osf.txt
Note that this is followed by the prompt: Please input your password:
.
The command below uploads the test file as osf_folder.txt
into a new folder called folder
within OSF Storage.
osf -p qf9n8 -u g.alejo@alumni.ubc.ca upload test.txt folder\osf_folder.txt
This command uploads the test file into the connected Github repository as osf_addon.txt
.
osf -p qf9n8 -u g.alejo@alumni.ubc.ca upload test.txt github\osf_addon.txt
Note that the only add-ons accessible through osfclient are github
, figshare
, and googledrive
.
Use the recursive argument, -r
.
This command uploads the contents of the local folder osf
(osf_1.txt
, osf_2.txt
, osf_3.txt
) into OSF Storage.
osf -p qf9n8 -u g.alejo@alumni.ubc.ca upload -r osf\ \
The ouput of the list
command will include:
osfstorage\osf_1.txt
osfstorage\osf_2.txt
osfstorage\osf_3.txt
To create a folder with the name of the local directory, remove the slash in the source
argument.
osf -p qf9n8 -u g.alejo@alumni.ubc.ca upload -r osf \
In this case, the list
command will output:
osfstorage\osf/osf_3.txt
osfstorage\osf/osf_2.txt
osfstorage\osf/osf_1.txt
fetch
¶! osf fetch -h
usage: osf fetch [-h] [-f] [-U] remote [local] Fetch an individual file from a project. The first part of the remote path is interpreted as the name of the storage provider. If there is no match the default (osfstorage) is used. The local path defaults to the name of the remote file. If the project is private you need to specify a username. If args.force is True, write local file even if that file already exists. If args.force is False but args.update is True, overwrite an existing local file only if local and remote files differ. positional arguments: remote Remote path local Local path optional arguments: -h, --help show this help message and exit -f, --force Force overwriting of local file -U, --update Overwrite only if local and remote files differ
To download a single file, use fetch
.
osf -p qf9n8 fetch \osf.txt test.txt
Here, we downloaded osf.txt
into the working directory as test.txt
.
Note that the remote
argument must be a file path. In this case, the file is stored in the default storage (osfstorage) therefore a slash before the file name is sufficient.
remove
¶! osf remove -h
usage: osf remove [-h] target Remove a file from the project's storage. The first part of the remote path is interpreted as the name of the storage provider. If there is no match the default (osfstorage) is used. positional arguments: target Remote file path optional arguments: -h, --help show this help message and exit
The remove
command deletes files.
To delete osf.txt
which was uploaded earlier, use:
osf -p qf9n8 -u g.alejo@alumni.ubc.ca remove \osf.txt
The target
argument must be a path. Like upload, a password is required for this command.
The examples from above are recreated using the equivalent function to each command. Click on the headers to jump back to the corresponding section under Command-line.
import osfclient
from osfclient import cli
import os
Note: cli
contains the upload, clone, etc. functions (source code). It is imported separately because otherwise we don't have access to it.
osf = osfclient.OSF()
proj = osf.project('qf9n8')
proj.title
'osfclient'
proj.description
'A dummy project created to demonstrate the use of osfclient, a command-line program and Python library for uploading and downloading files to and from OSF.'
os.environ["OSF_PASSWORD"] = "" # insert your password as a string
If the password is not set as an environment variable, a password prompt will appear every time upload
and remove
are used (for public projects).
class args:
def __init__(self, project, username=None, update=False, force=False, destination=None, source=None, recursive=False, target=None, output=None, remote=None, local=None):
self.project = project
self.username = username
self.update = update # applies to upload, clone, and fetch
self.force = force # applies to fetch and upload
# upload arguments:
self.destination = destination
self.source = source
self.recursive = recursive
# remove argument:
self.target = target
# clone argument:
self.output = output
# fetch arguments:
self.remote = remote
self.local = local
Let's initialize the arguments. We can change them later as needed.
arguments = args(
username='g.alejo@alumni.ubc.ca',
project='qf9n8',
# upload arguments:
destination='',
source='',
# remove argument:
target='',
# clone argument:
output='',
# fetch arguments:
remote='',
local=''
)
list_
¶cli.list_(arguments)
github\osf_addon.txt osfstorage\osf/osf_3.txt osfstorage\osf/osf_2.txt osfstorage\osf/osf_1.txt osfstorage\osf.txt osfstorage\folder/osf_folder.txt
This project has one component which list_
does not show. Set the project
argument equal to the component ID to see its contents.
arguments.project = 'q5mt7'
cli.list_(arguments)
osfstorage\osf_component.txt
init
¶You can pass any string to init
but it will not accept no arguments.
cli.init()
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-11-9f65c6c12d3d> in <module>() ----> 1 cli.init() TypeError: init() missing 1 required positional argument: 'args'
cli.init('blah')
Provide a username for the config file [current username: ]: g.alejo@alumni.ubc.ca Provide a project for the config file [current project: ]: qf9n8
upload
¶arguments.project = 'qf9n8'
arguments.source = '\\Users\\User\\osf.txt'
arguments.destination = '' # leave as as an empty string to maintain file name
cli.upload(arguments)
arguments.source = '\\Users\\User\\test.txt'
arguments.destination = 'folder\\osf_folder.txt' # indicate the name of the folder as the first part of the path
cli.upload(arguments)
arguments.source = '\\Users\\User\\test.txt'
arguments.destination = 'github\\osf_addon.txt' # specify the add-on as the first part of the path
Note: If uploading to an add-on and the file name is to be maintained, make sure that the destination
argument is a path! (i.e., "github\\"
)
cli.upload(arguments)
Notes:
github
, figshare
, and googledrive
.destination
argument, the local file is uploaded with its original name.arguments.recursive = True
To upload files from a directory directly into OSF Storage (or other add-on), the path must end with backslashes (or a forward slash for Mac OS):
arguments.source = "\\Users\\User\\osf\\"
To create a folder with the name of the local directory, do not end the path with slashes.
arguments.source = "\\Users\\User\\osf"
arguments.destination = '' # an empty string indicates OSF Storage; can also be a storage add-on
cli.upload(arguments)
Reminder: To upload a file into a component, set the project argument equal to the component ID, i.e. arguments.project = <component_ID>
.
clone
¶arguments.project = 'q5mt7' # component ID
arguments.output = '\\Users\\User'
cli.clone(arguments)
0files [00:00, ?files/s] 0%| | 0.00/5.20k [00:00<?, ?bytes/s] 1files [00:05, 5.03s/files]██████████████████████████████████████████████████| 5.20k/5.20k [00:00<00:00, 1.30Mbytes/s]
fetch
¶arguments.project = 'qf9n8' # project ID
arguments.remote = 'osf.txt'
arguments.local = 'test.txt'
Set the force
argument to
True
to overwrite the local file if it already exists.
# arguments.force = True
Set the update
argument to
True
to overwrite the existing local file only if the local and remote files differ.
# arguments.update = True
cli.fetch(arguments)
100%|█████████████████████████████████████████████████████████████████████████| 5.20k/5.20k [00:00<00:00, 2.60Mbytes/s]
Note: Set the local
argument equal to
None
to download the file into the working directory without changing its name.
remove
¶cli.list_(arguments)
github\osf_addon.txt osfstorage\osf/osf_3.txt osfstorage\osf/osf_2.txt osfstorage\osf/osf_1.txt osfstorage\osf.txt osfstorage\folder/osf_folder.txt
arguments.target = "osf.txt"
cli.remove(arguments)
Confirm that the file has been deleted.
cli.list_(arguments)
github\osf_addon.txt osfstorage\osf/osf_3.txt osfstorage\osf/osf_2.txt osfstorage\osf/osf_1.txt osfstorage\folder/osf_folder.txt
This notebook was created by Glaynel Alejo for the UBC Brain Circuits Research CLuster and other users independently of the developers and maintainers of osfclient.