Skip to main content

Marquez 0.30.0 offers a big performance bump, with some assembly required

· 3 min read
Michael Robinson

This is a guide to making a smooth transition to the latest version of Marquez.

Marquez 0.30.0

Marquez 0.30.0, which we released on 2023-01-31, contains some exciting new features, including column-level lineage and a soft delete option in the UI.

Along with these long-awaited UI upgrades come some significant performance improvements to the API. In prior versions of Marquez, accessing OpenLineage facets required direct queries against the OpenLineage events table, which could be quite costly depending on the size of this table. Some users have events tables with more than 100K rows -- millions of rows, in fact. To address this performance issue, Marquez 0.30.0 has new tables for dataset, job and run facets.

So far, so simple, right? Protecting against data loss and ensuring a smooth experience for users was not so simple, in fact. We needed a way to migrate the data while not only avoiding loss but also keeping downtime to a minimum. The solution that Willy and Pawel found (with input from Mike Collado) was to keep the data in the events table until the migration process is complete and to create views on top of the table for accessing the facets. After migration is complete, the views are replaced to point to the new tables. Using views provides safeguards and minimizes downtime during the migration.

Migration Procedure

As the migration plan says, users whose lineage_events tables have up to 100K rows can set it and forget it -- no extra work is required beyond upgrading and restarting Marquez, but be prepared for a brief downtime while a standard Flyway migration fills the newly created tables automatically.

For heavy users whose lineage_events tables have more than 100K rows, a different process will be used. Instead of a standard migration, the process will happen asynchronously so that the user can start the API and consume new OpenLineage events as usual. Note that some API calls might return results slowly during the migration, especially if the output is based on facets.

To perform a migration, these users must run a new db-migrate command:

java -jar api/build/libs/marquez-api-0.30.0.jar db-migrate --version v57 ./marquez.yml

Available command arguments, version (required) and chunkSize (optional), allow the user to specify the migration version to apply and the number of lineage events to process (the default being 10,000). The command will process the data not in its entirety but in chunks, storing a state with information about the chunks being processed. Consequently,

  • it can be stopped any time
  • it will continue automatically and process the remaining chunks.

A sample command specifying version 57 and 50,000 events:

java -jar api/build/libs/marquez-api-0.30.0.jar db-migrate --version v57 --chunkSize 50000 ./marquez.yml

More Information

For more details about what, if anything, you will need to do to ensure a smooth transition to Marquez 0.30.0, check out the helpful migration plan put together by Willy and Pawel.

For more information about the improvements and additions in Marquez 0.30.0, check out the release on GitHub.