baskerville.helpers package

Submodules

baskerville.helpers.gcs_utils module

baskerville.helpers.gcs_utils.download_folder_from_gcs(gcs_dir: str, local_dir: str, bytes=True) → None

Downloads a whole folder from GCS :param gcs_dir: string path to GCS folder to download :param local_dir: string path to download to :param bytes: boolean flag indicating if gcs file contains bytes

Returns: None

baskerville.helpers.gcs_utils.download_from_gcs(gcs_path: str, local_path: str, bytes=True) → None

Downloads a file from GCS :param gcs_path: string path to GCS file to download :param local_path: string path to download to :param bytes: boolean flag indicating if gcs file contains bytes

Returns: None

baskerville.helpers.gcs_utils.download_rename_inputs(filepath: str, temp_dir: str, is_dir: bool = False) → str

Download file from gcs to local dir :param filepath: GCS Uri follows the format gs://$BUCKET_NAME/OBJECT_NAME :param temp_dir: local dir to download to :param is_dir: boolean flag indicating if the filepath is a directory

Returns: new filepath in the local machine

baskerville.helpers.gcs_utils.gcs_file_exist(gcs_path: str) → bool

check if a file exist in gcs params: gcs_path returns: true/false

baskerville.helpers.gcs_utils.gcs_join(*args)

baskerville.helpers.gcs_utils.get_filename_in_dir(files_dir: str, recursive: bool = False) → List[str]

Returns list of filenames inside a directory.

baskerville.helpers.gcs_utils.is_gcs_path(gcs_path: str) → bool

Returns True if the string passed starts with gs:// :param gcs_path: string path to check

Returns: Boolean flag indicating the gcs_path starts with gs://

baskerville.helpers.gcs_utils.split_gcs_uri(gcs_uri: str) → tuple

Splits a GCS bucket and object_name from a GCS URI :param gcs_uri: string GCS Uri follows the format gs://$BUCKET_NAME/OBJECT_NAME

Returns: bucket_name, object_name

baskerville.helpers.gcs_utils.sync_dir_to_gcs(local_dir: str, gcs_dir: str, verbose=False, recursive=False) → None

Copies all files in a local directory to the gcs directory :param local_dir: string local directory path to upload from :param gcs_dir: string GCS destination path. Will create folders that do not exist. :param verbose: boolean flag to print logging statements :param recursive: boolean flag to recursively upload files in subdirectories

Returns: None

baskerville.helpers.gcs_utils.upload_file_gcs(local_path: str, gcs_path: str, bytes=True) → None

Upload a file to gcs :param local_path: local path to file :param gcs_path: string GCS Uri follows the format gs://$BUCKET_NAME/OBJECT_NAME

Returns: None

baskerville.helpers.gcs_utils.upload_folder_gcs(local_dir: str, gcs_dir: str) → None

Copies all files in a local directory to the gcs directory :param local_dir: string local directory path to upload from :param gcs_dir: string GCS destination path. Will create folders that do not exist.

Returns: None

baskerville.helpers.h5_baskerville_utils module

baskerville.helpers.h5_baskerville_utils.collect_h5(file_name, out_dir, num_procs) → None

Concatenate all output files together :param file_name: filename containing output (sad.h5) :param out_dir: directory containing output files :param num_procs: number of processes

Returns: None

baskerville.helpers.h5_baskerville_utils.main()

baskerville.helpers.h5_utils module

baskerville.helpers.h5_utils.collect_h5(file_name, out_dir, num_procs)

baskerville.helpers.h5_utils.collect_h5_borzoi(out_dir, num_procs, sad_stat) → None

baskerville.helpers.tensorrt_helpers module

class baskerville.helpers.tensorrt_helpers.ModelOptimizer(input_saved_model_dir, calibration_data=None)

Bases: object

Class of converter for tensorrt :param input_saved_model_dir: Folder with saved model of the input model

build(seq_length)

build_func(seq_length)

convert(precision='FP32')

save(output_dir)

set_calibration_data(calibration_data)

baskerville.helpers.tensorrt_helpers.main()

baskerville.helpers.trt_optimized_model module

class baskerville.helpers.trt_optimized_model.OptimizedModel(saved_model_dir=None, strand_pair=[])

Bases: object

Class of model optimized with tensorrt :param saved_model_dir: Folder with saved model

load_model(saved_model_dir)

predict(input_data)

baskerville.helpers.utils module

baskerville.helpers.utils.exec_par(cmds, max_proc=None, verbose=False)

Execute the commands in the list ‘cmds’ in parallel, but only running ‘max_proc’ at a time. :param cmds: list of commands to execute :param max_proc: maximum number of processes to run in parallel :param verbose: print command to stderr

baskerville.helpers.utils.load_extra_options(options_pkl_file, options)

Parameters:

• options_pkl_file – option file

• options – existing options from command line

Returns:

updated options

Return type:

options

Module contents