Integrate with syft #28
Labels
Priority: 2 - High 😰
Should be fixed as quickly as possible, ideally within the current or following sprint
Type: Epic 🤙
Describes a large amount of functionality that will likely be broken down into smaller issues
Milestone
What?
The current implementation of data loaders and datasets is quite hacky.
We should integrate existing syft functionality and extend it to make Vertically-partitioned dataset
a robust class, making it easy for anyone to apply PyVertical to any dataset
Breakdown
syft.fl.BaseDataset
to create a dataset which holds partitions and may hold either data or targets. This should extendPyVertical
'sVerticalDataset
to includesyft
functionality of ownership Extend syft federated datasets #47dataset_partition
which partitions a dataset, sends the partitioned datasets to the correct worker, and returns asyft.fl.FederatedDataset
of partitioned datssets. This builds on the currentpartition_dataset
function inPyVertical
, and is similar tosyft.fl.dataset_federate
Create partition function for federated datasets #48PartitionDistributingDataLoader
with a dataloader which takes asyft.fl.FederatedDataset
. This should extendsyft.fl.FederatedDataLoader
to account for datasets which may not contain data or targets Create syft-like federated dataloader #49Additional Context
This will developed simultaneously with the extended PyVertical demonstration (#25), so to avoid breaking changes existing dataloaders/data splitters should be kept until this issue is complete
The text was updated successfully, but these errors were encountered: