Import from the source Mongo database

This page describes the first stage of the migration process where you use a Datastream stream to capture the contents of your MongoDB-compatible source database and transfer them into a Cloud Storage bucket.

Create YAML configuration files for the stream

In addition to usual command-line flags, creating a stream requires two configuration files in the YAML format:

• The mongo_source_config.yaml file configures the selection of specific resources for migration, such as the database name. Mongo connectivity parameters such as the hostname, username, and password are all properties of the connection profile. However, the database (and any specific collections within that database) are a property of the stream.

• The gcs_dst_config.yaml file configures the data placement within Cloud Storage. The Cloud Storage bucket and the root path within the bucket are properties of the connection profile. However, the data format and the data placement within the Cloud Storage bucket structure are a property of the stream.

The following command examples create these files and populate them with values from the environment variables that you've set earlier. As an alternative, you can create these files in any text editor and substitute the values manually.

echo "$(cat <<EOF
includeObjects:
  databases:
  - database: ${MONGODB_DATABASE_NAME}
EOF
)" > mongo_source_config.yaml

echo "$(cat <<EOF
path: "/${GCS_BUCKET_SUB_PATH}"
avroFileFormat: {}
EOF
)" > gcs_dst_config.yaml

The previous example configures the full contents of $MONGODB_DATABASE_NAME for migration. It is also possible to limit the migration to specific collections within the database. For example, to migrate only the collections users and chats use the following:

includeObjects:
  databases:
  - database: ${MONGODB_DATABASE_NAME}
    collections:
      - collection: users
      - collection: chats

Create a Datastream stream

Next, create a stream that connects the source and the destination:

gcloud datastream streams create "$DATASTREAM_NAME" \
--display-name="$DATASTREAM_NAME" \
--location="$LOCATION" \
--source="$SRC_CONNECTION_PROFILE_NAME" \
--destination="$DST_CONNECTION_PROFILE_NAME" \
--mongodb-source-config=./mongo_source_config.yaml \
--gcs-destination-config=./gcs_dst_config.yaml \
--backfill-all

For more information about monitoring the Datastream stream creation, see Troubleshooting.

Activate the Datastream stream

Finally, activate the new stream.

As the stream begins pulling data and streaming changes from the Mongo source, you can observe new directories and files created in the Cloud Storage bucket, under the path configured in the connection profile and the stream.

To activate the stream, run the following command:

gcloud datastream streams update "$DATASTREAM_NAME" \
--location="$LOCATION" \
--state=RUNNING \
--update-mask=state

What's next

Proceed to Write data into the destination database.