From the course: Implementing Data Engineering Solutions Using Microsoft Fabric (DP-700) Cert Prep by Microsoft Press

Unlock this course with a free trial

Join today to access over 25,500 courses taught by industry experts.

Optimizing a data factory pipeline

Optimizing a data factory pipeline - Microsoft Fabric Tutorial

From the course: Implementing Data Engineering Solutions Using Microsoft Fabric (DP-700) Cert Prep by Microsoft Press

Start my 1-month free trial Buy for my team

Optimizing a data factory pipeline

“

Now when we're looking at the copy data performance within a pipeline, in which we're using to load from a source into a destination, the performance can depend on several factors. Network bandwidth. And this is very simple. If the source system is hugely constrained by network bandwidth, your copy data activity is also going to be affected. So look at the bandwidth stats to understand how much data is being loaded and in what time frame. And of course, we've got IOPS. So this is input and output operations. Essentially, with IOPS, this could be constrained by the data store itself. This could simply be access of the disk that that data store is using. Now, we touched on this briefly before, but we have the intelligent throughput optimization. And what this is doing is it's balancing the CPU, the memory, and the network resources. And we can set that in the pipeline settings itself. We can change that intelligent throughput optimization to auto, standard, balanced, and maximum. Now…

Contents