Skip to content

Connectors for AI-on-Demand

To start connector services that automatically index data from external platforms into the metadata catalogue, you must specify their docker-compose profiles (as defined in the docker-compose.yaml file). Their configuration, if any, is through environment variables which can be set in the override.env file as explained in "Configuring the Metadata Catalogue". For example, you can use the following commands when starting the connectors for OpenML and Zenodo.

./scripts/up.sh openml zenodo-datasets

docker compose --profile openml --profile zenodo-datasets --env-file=.env --env-file=override.env up -d

Connectors and Syncing Nodes

If you are configuring your metadata catalogue as part of a set of root nodes, only one of the root nodes should be running the connectors. Running the same connector on multiple root nodes may introduce conflicts.

The full list of connector profiles are:

AI Builder

Profile: aibuilder

Indexes models in the AI Builder library. When running the AI Builder connector, you need to provide a valid API token through the AIBUILDER_API_TOKEN environment variable.

AI Builder's models are only accessible with authentication, and for this the API key is required to be part of the query in the URL. Because we do not want to expose the API key, we obfuscate it and use AIBUILDER_API_TOKEN in URLs. This means that for using the url of the same_as field of the AIBuilder models, you will need to substitute AIBUILDER_API_TOKEN on the url for your actual API token value.

HuggingFace

Profile: huggingface-datasets

Indexes datasets in the Hugging Face repository.

OpenML

Profile: openml

Indexes datasets and models (OpenML flows) in the OpenML repository.

Zenodo

Profile: zenodo-datasets

Indexes datasets in the Zenodo repository.

Examples

Profile: examples

Adds example assets in many different categories to the database. This connector is for development purposes only and should not be used in production.