DBT Labs handles for power selector with new Fusion engine

DBT has become one of the most widely used data tools due to its extensibility and simplicity. People behind DBT are now a tool for the next level by launching a new fusion engine with a rusty lying, which DBT Labs Tristan Handy says, among other things, to bring 30x power over DBT Core. The company also launched three new and updated tools to increase data analysts.

The new DBT Fusion Engine is originally with the acquisition of SDF Labs DBT Labs at the beginning of the year. SDF Labs described its technology “Multi-Kmetek SQL compiler, transformation frame and analytical database engine wrapped in a single cli”. It turns out to be much more than that.

For starters, DBT Fusion is fast. According to Handa, the Fusion Engine DBT, which is based on SDF Labs technology, outside the box, 30 times quickly in DBT analysis than the aging Python engine in the heart of the free core product DBT and all other commercial products from DBT Labs.

But 30x is just an estimate, says Handy. “This is like a code we sprint to get out of the door and not fully optimized,” Bigdatawire. “I think this number will appear.”

The entire basic speed will lead to a better user experience for analytical engineers, which DBT Labs focuses on its products. The better UX-drew through the new plug-in for the VS code that DBT Labs will also be decisive because DBT Labs brings more conversational capacity AI to the product.

DBT Fusion also understands a wide range of SQL dialects, says Handy. So L’Azarless about the dialect you write – Ulick it’s a snowflake SQL, databricks SQL, Google Cloud BigQuery SQL or Amazon Redshift SQL – DBT Fusion will immediately find errors.

“It will show you the red line Squiggly. The mouse above her, shows you an error message. It tells you exactly what is going on and how to fix it,” Handy says. “This is a kind of developmental experience that developers are used to in all modern programming languages, but SQL never had.”

The benefits of future merger

Over the next six weeks, DBT Labs will support all four SQL data warehouses with DBT Fusion, with more coming in the future. DBT Fusion should work largely outside the box with an existing code written with DBT Core, Handy says. More, all the products that DBT Labs built on its core DBT core product will improve as a result of the merger.

“Each of them gets quickly. This means our editor will be much more interactive, better error messages, etc.” However, there are a lot of neat things that we have never been able to build in the past, that we will have to build. ”

For example, Fusion users will be able to run the emulated version of their data warehouse directly on their laptop in the future. This will allow them to develop better transformations on the subgroup of their data and at the same time save money, says Handy.

“Fusion can do this, it is fully imitating the basic data platform and allowing you to actually have a cohesive local development where notolly is not talking to the underlying plafform at that time. You have a reduced copy of data you can try.” This will be a great benefit for the developer. And it also saves a lot of money. ”

According to DBT Labs estimates, this function itself saves customers 10% at the cost of their data warehouse.

New DBT tools

DBT Labs has also launched a number of new and updated tools for DBT users, DBT Insights, Question A-Power tool for quick analysis and sharing; Including DBT Canvas, lightweight tool for Drag-and-Drop used for data model development and data pipeline; and the new edition of the DBT catalog, its tool for discovering data assets.

While DBT is primary about data pipes and metadata associated with data pipes, sometimes the user has to explore his data a little to create these pipes, Handy says. This is what led to the development of DBT knowledge.

“We wanted people to explore their data inside the DBT interface,” he says. “Snowflake and databricks have both of the light, reconnaissance experiences that users love, and now they have ours.”

Interesting is on Canvas DBT, Handy says that it is building on user interfaces for building gas pipeline, which developers have been using for decades by overlapping the AI ​​conversational interface.

“It turns out that we are able to experience this shot that effectively because of the whole context that DBT has about your data,” says Handy. “So it’s a kind of magic. It appears on the screen and there’s an empty screen, and then you just enter the challenge and say,” Here’s what I want. “You hit Enter and then you just greeted the visual pipes, which you can then tune yourself.

These two new tools and upgraded data catalog are loved by bringing more users to DBT experience that contains native DEVOPS integration for data management that lacks other analytical and development tools, says Handy.

“These are all things that make the DBT workflow accessible to data analyzes that traditionally work on work on these villains,” he says. “Central data teams that love DBT so much can actually go to them and say,” Hey, look, I have a better tool for you. Why don’t you join the party? “

End of line for dbt core

DBT Fusion launches a high -performance new start for DBT, but also spells the beginning of the end for DBT Core. The old product will continue to be supported, but will not be seen so many new features and improvements, because DBT Fusion is a new strategic future for DBT Labs.

Handy was retrospective about change.

“We originally built DBT Core in 2016. We have added a lot of functions to it. But the basic technology is the same. It is based on Python. It has some basic design principles that continue to apply this day. And it turned out to say everything.”

“We have moved Core Forwards for nine years and technology has nothing to go there,” he says. “There are a lot of things we wanted for the DBT community that could be with this code base.”

Internally, DBT Labs began to work on the way forward. ASY examined possible paths, homeless from secrecy with the data transformation engine that has already supported DBT. This paid attention to Handy.

“Basically, it builds the engine we wanted to build. It has already been focused on the DBT community. It already knew how to speak DBT. And so it is a kind of how to get,” he says. “Over the past five and a half months, we sprint as hell to integrate this thing into DBT technology.”

From that moment on, the DBT Fusion engine, which also has private commercial DBT Labs products. DBT Fusion, like DBT Core, will have a source and Aviet license, which means it is free for anyone who could use ASY, but DBT Labs retains full control over development.

Handy is particularly amazed by the work of SDF Labs CTO and co -founder of Wolfram Schulte, who is the brother of the CEO and co -founder of Lukas Schulte. Wolfram Schulte has been a long -time main software engineer of Microsoft Research. After the Cambridge Analytica scandal, tungsten were hired to find out where all personal data (PII) traveled around Facebook.

“When you literally create more than a million tables, it is very difficult to watch where PII is about to go. It is too much to go and the hand to classify everything,” says Handy. “So what tungsten did was the position of a multi-diadeel analyzer that could understand all these pipelines. Then they took this technology and created the basis of what had happened.”

Related items:

DBT Labs report reveals how AI increases data budgets and team growth

QLIK and DBT laboratories make acquisitions of large data integration

Still too much adhesive tape in data transformation, says DBT Labs’ Handy