Incremental Refresh with Power BI Premium

[@AutomationBIAI]

Excellent stuff. I just wonder what values were input for the StartRange and EndRange and how that is going to impact the result ? must the EndRange be a future day ?

[@fspinacz]

For fellas already knowing how much blissful the incremental refresh is -> 3:17

[@khudammunir3060]

What will be the values in Range Start and Range End Parameters ? The Starting or Ending Window of our Data Set or the Time frame other then the history

[@samikpal3583]

I have done an experimentation to apply incrementalRefresh to a dataFlow and I have realized that if the df has future dateTime values in the dateTime column (e.g. dateTime value as 2022-01-01 12:00:00 AM in the dateTime column) and if I set up Refresh Rows from the past 1 year as of today (2021-30-06) the incremental will altogether ignore the rows in the dataset refresh with the dateTime value > 2021-12-31 11:59:59 PM. I followed this video but I got stuck with this.

[@MR-sw4re]

When I create my date/time parameters (RangeStart/RangeEnd), Power BI insists that I specify a date. So I picked any old dates. In turn, these dates get applied to my data set when I put the filer on. Long stir short, my published dashboard is using the bogus date range I was forced to put. I noticed on this video when you created the params, it didnt as you to put dates. Why is that? What did I do wrong ?

[@jacheto]

This video did not specified it enough...there are a lot of possibilities, for example what the parameters has to do with that?

[@tjvillanueva396]

In the video 5:58, where does the LastUpdateDate came from? is it coming from the data or effect of the incremental refresh?

[@GuyInACube]

It would be a date field in your table. Think of it like change data capture. Something is signaling that the row was updated. So, we key off of that to actually update the model with that data. otherwise, it is ignored because it is out of scope of the refresh policy.

[@voxjowel]

what date range should i indicate in my sql query to capture 5 years worth of data but avoid it from loading in my desktop while setting up the parameters and incremental refresh?

[@saieswar672]

Hello Jowel, Did you get this answe, if yes Please share. Thanks in advance

[@akankshaism]

i have the same qs. the video shows it as super easy. but when i set it up, it shows 'Apply changes' and then on applying it starts to download all history on desktop

[@veera8853]

Can we implement incremental refresh in ssas cube

[@kiariesamuel2691]

Hello, Guy in Cube, Will incremental refresh notices when historical data has been reprocessed in the DWH or DB. How can I avoid missing reprocessed data?

[@aaronwatt8590]

is there a way to get these refreshes to go through even when my computer, PBI owner, is offline? Other employees want to access my dataset but because I am offline are unable to refresh it until I come into work and they are simply SOL

[@dmitryvakhrushev81]

Why does the incremental refresh issue TWO queries for each day (e.g. refresh rows in the last 10 days)?

[@syedasadkhan]

Second Question i have is, is the data being pulled from source ie being filtered at query time, or after it is called, POwer Query is limiting results to Service?

[@paradix847]

Hey, I'm wondering... does it matter at all what values I set for the RangeStart and RangeEnd parameters when creating them?

[@alicedwonderland7733]

I'm confused about the RangeStart and RangeEnd parameters. Should we cast a wide net, say from 10 years ago to 10 years from now?

[@ezequielbolwahnn6900]

I noticed in the video that the parameters (RangeStart and RangeEnd) did not have a current value (it was blank). However, for me, when I go into create a new parameter, it requires that the field be populated before I can move forward. Is this a new requirement? Can I just populate dummy data and the set up of Incremental Refresh overwrite this dummy data?

[@shyrlo]

up

[@JeseNavar]

Great video. thank you. Quick question - how does the incremental refresh know which datetime column to use in the 'Last 5 years', 'last 1 day'. Is this the same column specified on the parameters?

[@shyrlo]

up

[@ShivdevSingh]

This was probably answered somewhere here but the parameters you enter in Desktop, get automatically overridden once the report is published to service to use the date ranges defined in the policy.

[@jamesnyoike1475]

Since one cannot download the PBI file, are there any limitations (other than publishing to a premium workspace) on the ability to publish changes using the original Power BI desktop file?

[@DoubtlessLiving]

I'm trying to use incremental refresh with a Dataflow bu my Sales fact table uses ISO dates (YYYYMMDD) in string format for the transaction date field so I don't have a "datetime" field in the file that Power BI dataflow will recognize. Any ideas on how to solve this problem?

[@КоляАрт-ф3ч]

Hi, but if my partition key is not the date column? Can I manage partitions throught SSMS like for SSAS? Thanks

[@hello-u2w]

what happens if the incremental parameters (RangeStart and RangeEnd) are not present? Can I create them manually?

[@vyazici]

Even though I think it isn't but to make sure. Is it necessary to have query folding for dimension tables to set up incremental refresh? does it matter the source of the dimension tables?

[@muralikrishna8392]

The refresh operation failed because it took more than 120 minutes to complete. Consider reducing the size of your dataset or breaking it up into smaller datasets. How can I overcome this error?

[@nishantvarshney2676]

Hi guys......i tries it a lot but still facing a query.....i dont have the predefined date parameters as used....and even the type of the date is not same.....can you please help me

[@vipinmohan1191]

This is great! I am trying to implement this feature on few of my reports. But, when creating parameters do we have to enter any values in the 'Current Value' section? I am not able to create a parameter otherwise.

[@shyrlo]

up for this question and do we have to add lastupdated column?

[@alvaroandressuarezalfonso7842]

Thanks for the vlog, btw i wonder what happens if one has the option to refresh rows in the last 1 day, would the next resfresh done automatically or by demand detect that it has to load last n days of failed refreshes?

[@rameshradipudi]

Will this work with ServiceNow database? I was reading that incremental refresh will work with databases that support Query folding. I am not sure if ServiceNow supports query folding. Can you please advise?

[@rickridgway3225]

Sorry, I am late to the game, but we are jumping in with both feet. One question I have is what if you have a table that holds all the previous entries and you want to incrementally refresh while keeping all previous data? The information in the table will be used in perpetuity, and can be added to at any time.

[@culpritdesign]

There should be an option to export the PBIT file from the cloud service. It seems strange to publish the pbix file and then it's locked into the cloud service with no way to get it back.

[@hello-u2w]

I cannot set the parameters in the filter rows. Yes I have the premium version of Power BI. what do I do?

[@rkacom]

Is there a way to Pass RangeStart and RangeEnd to Blank Query or Azure Data Explorer?

[@AHMEDALDAFAAE1]

Many thanks for this video! I have a question: Can I implement the incremental refresha technique in the computed dataflow which is subsequent to other dataflows?

[@RohitKumar-xm4uk]

Does it take into account the RangeStart while performing the initial refresh which was set in Power BI desktop while deveopment or override that and load the entire data from the source?

[@akankshaism]

yes really this is a big problem n they dont show it here in the video!!

[@anmolganju1864]

Great features, but only limited to some of the services, when this will be available for SAP-BW and SAP HANA connector for powerbi pro and premium workspace???

[@musafirq]

Hey Adam, Is this correct that while wring IF or Switch condition in measure I cannot use columns for filtering and can use only measures.

[@sabkuchdhela191]

I want to connect Google sheets and needs to refresh data continuously on those sheets, which is then further connected to power bi.. watched many videos but seems no trick is in working condition.. can someone help here ?

[@VeeruBhai1410]

Hello ppl, I have one doubt. Say for example, I setup incremental refresh for my data set in my power bi desktop with the name of pbix file as "Test Report" and publish them report to the service. After sometime I republish the report all the stored data is erased?

[@satishtunuguntla1936]

is it possible to change from incremental load to full load and then back again to incremental load... just to cover any manual updates at the source

[@avivzach4926]

Amazing feature! can it be used for updating data? lets say I have 5 years of data, records being updated randomly. can I use it to update just the records that were changed in the last day fir example? and refresh once a day?

[@LandscapeInMotion]

Setting up incremental refresh like this requires import mode? Or DirectQuery mode to the data model? So if I need access to live data, will incremental refresh “detect changes” be responsive enough for minute to minute changes?

[@GuyInACube]

Incremental Refresh is specific to imported data. For DirectQuery, it doesn't make sense as the data stays at the data source.

[@martinnieves6873]

The screen grabs in this video are very outdated. Do you have a newer walkthrough?

[@Bharath_PBI]

Hi, I've been using incremental refresh on import tables.. but started working on a composite model.. Problem is incremental refresh can not be set up on a dual storage table and requires import mode. But having a fact agg table in import mode and base fact in direct query, will injection happen from import to DQ query which is sent to db? If yes how's the performance on large models?

[@tomarshubham18]

"The paramters in desktop will have no effect in the service", what does this mean? I won't be able to filter my data in service using parameters if i use incremental refresh?

[@GuyInACube]

Meaning that they are used in the desktop. Once published to the service, the parameters are not available and will be taken over by the incremental refresh and used for its purposes. So, while they are filtering within Power BI Desktop for the given date range you provided, those values won't be used once it gets to the service.

[@tomarshubham18]

Guy in a Cube Ohh, but the parameters used other than the date column should work. Right? In case I have more than one parameters

[@GuyInACube]

Incremental refresh only works with the RangeStart and RangeEnd parameters. those are reserved parameters. Other parameters you define will still be using the normal way.

[@johnberg9131]

Great video, but one concern about not being able to download the .pbix file. If the report needs to be updated and the original desktop file is corrupted or lost (employee leaves, etc.), then there is no way to download from the cloud. Any plans to provide a way to download the .pbix without the data? Of course companies should have a good backup policy to avoid this, but if the pbix exists on a local drive this is a possibility.

[@victorandarcia9941]

Hi John, you would have to disable the Large Dataset option on the Power BI dataset settings and then download it.

[@RB26_Aces]

Currently the column within the query appear to be required to be Date/Time for this to all work.Would you happen to know if the development team plans to change or add the ability to do this with just Date formats?

[@scottpowell5205]

I'm curious about if there is a way to tell the service to either rerun a specific incremental, or to reload the entire thing. For instance, if I've got the service set to refresh data that changes in the last day...and something happens in the DW loads so the whole thing doesn't update for multiple days until we get the issue resolved. Would there be a way to "catch up" on the missed days?And/or I'm concerned that backdated changes can happen to records - without a "last update timestamp" catching those. I might want to run daily refreshes during the week, but perform a full truncate / reload over the weekends.Just trying to kick the tires and see what is possible!Thanks,Scott

[@manojalapati]

Hi Adam, can we do this in report server version desktops? From which version is it available?

[@syedasadkhan]

This was great. Question, will incremental refresh work with On Demand Refresh using REST APIs

[@LandscapeInMotion]

RangeEnd and RangeStart parameters are special names used by the data refresh service?

[@GuyInACube]

Yes they are reserved parameters used for the incremental refresh feature. You can't modify them within powerbi.com

[@raghavendrapilli7036]

Great information 👍, I have a question if my data source is SQL, I did changes in the SQL server, so will it incremental refresh works? Or else it's works only Direct query?

[@L76videos]

I hope there will be in Pro version too very soon

[@GuyInACube]

The May 2018 Power BI Desktop blog indicated it would be coming for Pro/Shared Capacity. It doesn't indicate a timeline though. It is also not listed in the Spring '18 release notes. powerbi.microsoft.com/en-us/blog/power-bi-desktop-may-2018-feature-summary/#incrementalRefresh

[@chitwinder]

Wonderful Video.....Please try to make a video that include incremental refresh from the very starting point. Thanks for your help!

[@samcarter9111]

Thanks for the Video! I would like to know, does the incremental Refresh refreshes the other tables that has no date column, but are in relation with the table which we filtered?

[@gagandhaliwal40]

Hi Adam, very useful video. I am facing one issue as I set up incremental refresh but after every scheduled or on demand refresh it is not overridden the parameters. Hence, It is not able to load recent data instead getting same(OLD) data from RangeStart till RangeEnd. Any Idea why?

[@diannegillam8788]

Hi. Where do I find the pre defined parameter names. The pre defined names are not showing in manage parameters.

[@GuyInACube]

You have to manually create the parameter. They are reserved names. So, when you create the parameters, they have to have those specific names in order to work.

[@vijaybodkhe8379]

Hey Adam you are always energetic which makes learning power bi more interesting

[@rapatel1642]

Great video, what I was looking for. Thanks

[@robdumont9141]

nice feature, any recommendations when to use Azure Analysis Services vs this Power BI capability?

[@GuyInACube]

It depends. You can do partitioning within Azure AS. Azure AS also supports read replicas for scale out. Honestly, if you are looking for concurrency, Azure AS with scale-out is probably the better option. Also currently Power BI only supports up to 10GB from the dataset size. Azure AS can handle more than that. Over time, that will change.

[@robdumont9141]

One of the reasons AS may be preferred is if you want to report using something other than PBI. Unless other applications, besides excel, can access a PBI data source?

[@robdumont9141]

I found another important consideration and that is $$$$, to refresh my PBI Premium data source (35 million rows) it required A4 capacity ($6K per month) versus my AS model loading the same source which required S1 capacity ($1.5K per month), both approaches used 12Gb cache, which necessitated the tiers, and both took about 20 minutes to refresh. Anyone else experiencing similar results?

[@Pinkpalmpufffyslay]

Hi Adam, would this incremental refresh work for blob storage data as well? Also, after uploading it to the service, would the incremental refresh work there as well? Or, I have to perform refresh every time on the PBI desktop and replace the updated file on PBI service?

[@hemanthkumar5166]

Hi Adam / Christian, Great stuff. But its not working even I have premium workspaces. Please let me know where i am going wrong. My data source is Teradata and i pulling calendar table for testing. I have followed same steps you have given and apply changes after filtering the parameters. Now when I right click on the object and click on incremental refresh i have a error which says "Before you can set up incremental refresh on this table you need to set up parameters" I am not sure I have done that and now its throwing this error where i m not able to turn on incremental refresh. Thanks in advance

[@Parostache]

Hi, thanks for the good video. What would you consider as a best practice when it comes to deployment cycle with Power BI Premium/Pro when the incremental refresh is being used? Now I can see as an issue, when end users might create power bi reports on the top of dataset and admins need to modify the same dataset? That means that all links to dataset disappear, am I right?

[@RahulYadav-my9tm]

A very helpful video. I have a question, If we have a merged table (with Fact & Dim Table) with original source tables not included in the refresh. How we can setup the incremental so that the changes in both tables are considered in the incremental load

[@GuyInACube]

The incremental refresh is based on the Power Query query. So, you did the merge within Power Query? it would take the data for the change off of that merged query. It pulls whatever the result is from the Power Query result to update the model. That's why the Parameters were applied within Power Query. Does that help?

[@RahulYadav-my9tm]

Yes, we have merged the tables in Power Query. Now we want to set the incremental based on changes that may happen on both the tables. So, ideally the incremental that we setup should look for the changes in both tables and not just one. Also, one more question, you mentioned that we will have to create two parameters RangeStart & RangeEnd date, can we use these parameters for setting up incremental on multiple tables? or we will have to create separate parameters for each table.

[@tarikghazali1394]

Very Good and Simple Clarification

[@sifisongobese3113]

I seem to be stuck, I have done the necessary steps and published to the service, my only problem comes up when refreshing the dataset, it says "refresh failed due to gateway configuration issues. the dataset set requires a properly configured gateway in order to refresh. If your using enterprise mode ensure that you have added the following datasources in the Gateway Mangement Portal. > http: URL >Http:URL I have set the authentication level to : Anonymous I have set the privacy level setting to : blank this is on both desktop and service. Please advise on how to properly set the gateway.

[@ABMAndrewMwithi]

This is a great feature addition. Keep up the incremental improvements.

[@GuyInACube]

Thanks Andrew! 👊

[@CurtisMcLee116]

Also if I'm going to load my data in chunks as in the video it was said that initial load might be too large for your desktop (running 16 GBs locally). How would I do that? If you could manage the parameters in the service that would make sense I would imagine (we can't). Unless I change the desktop file, what control do I have over the partition loads? And even if I did change them in the desktop file how would I make sure the partial dataset in the service was appended to with the next chunk of data and repeat this until my entire dataset was incrementally appended?

[@GuyInACube]

Within Desktop, you want to filter the dataset down to a small enough chunk to work within the desktop. The first time you upload to the service, it will backfill to get the range you specified. After that initial load, it will then work off of the window you picked for "Refresh rows in last". The kicker is that if you modify the desktop file, and republish, it will need to do the initial load again.

[@CurtisMcLee116]

Guy in a Cube Thank you your videos are great. Really appreciate them.

[@GHRichardson]

Here is a question about a really common scenario. My tables are calculated from Power Query that works on and joins other tables together - often with their own date fields from disparate data sources. To make Incremental Refresh work here, can I just filter the date field in the final (loadable) table, or do I apply it to each of the underlying 'working' tables that are eventually joined together to make the final table?

[@GuyInACube]

That's a great question. I'd have to try it to know for sure, but I would expect it to work on the final table. Incremental refresh is defined on the table in the model and that is pulling data from a filtered Power Query query. It should work fine.

[@avivzach4926]

Thanks for an awesome video!! Is it possible to configure Incremental Refresh for 2 tables in the same model/dataset? using the same parameters?

[@ajaysharma2061]

how to make available parameters for the user on Power BI service?

[@indianrose1984]

Great Video! I've learned quite a lot. I've set up my first incremental refresh and so far it's working quite well. One thing is still unclear though. You mentioned in a comment that the parameter filters should only work in the desktop app and when uploaded, the incremental refresh setup will be applied to the dashboard. When I do this, the published dashboard still shows the data range from RangeStart & RangeEnd instead of the incremental refresh setup. any suggestions?

[@kapil7746]

@indianrose1984, can you please tell why we are using parameters, if we provide any date to end date in tata case data till that date will only be loaded

[@CurtisMcLee116]

If I choose month for "Refresh rows in last" and do 1 month and it's June 15th does that only refresh what's in the Calendar month? So in this case June 1st-June 15th or does it use Month like 30 days so May 16th-June 15th?

[@GuyInACube]

My understanding is it is a sliding window. I'll verify though.

[@CurtisMcLee116]

Guy in a Cube I would hope it's calendar as that would be way more useful because once I have days I can easily do a 30 day interval but if Month is just defined as 30 days then that's kind of redundant.

[@oscarmqz]

Hi, I've been trying to find info about why does incremental refresh run several SELECT TOP 1000 queries against my data source. I'm not able to find an answer.

[@uzmarat]

My guess is that it is trying to get preview to find out the most optimal way to compress data: www.microsoftpressstore.com/articles/article.aspx?p=2449192&seqNum=3

[@chinmayapatra6450]

How do i implement incremental refresh in ADLS source.

[@supersteen]

Your “New York Taxi Report” object seems to be overlapping the canvas edge. How do you do that?

[@GuyInACube]

It's an illusion :) the edge of the page is actually the dark grey on the outer edge. not the lighter grey on the inside.

[@supersteen]

Neat! That is a pretty cool trick.

[@thanosd79]

Hi guys! When we say PowerBI Premium, do we also include Azure PowerBI Embedded capacities?

[@GuyInACube]

Yes. Dedicated capacity. This will work with either dedicated capacity - whether that is from Office (Premium) or Azure (Power BI Embedded).

[@nicolasrojasgarzon8922]

Q: Does incremental refresh in dataflows work with a sharepoint folder?

[@rodyharper6549]

This is great. How do you setup a periodic full refresh in addition to the more regular incremental refreshes? Maybe you do 7 incremental refreshes in a day, but once a day you want a full refresh?

[@muthupandi830]

I have the same doubt. Did you got any idea? Or Do you have any doc, foe that?

[@miragliag]

Real Great! Can we change the Parameters after uploading? I for example I want to keep 8 years instead of 5?

[@GuyInACube]

No. You don't have direct access to the parameters within powerbi.com. You would have to change that within Power BI Desktop. although once it is in the service, those parameters get updated based on the last refresh that occurred.

[@mohammadimtiaz9334]

Ended up with below error The JSON DDL request failed with the following error: Failed to save modifications to the server. Error returned: 'Parameter substitution failed with following error: 'Neither mashup nor gateway connection string detected for incremental refresh parameter substitution'. Parameter substitution failed with following error: 'Neither mashup nor gateway connection string detected for incremental refresh parameter substitution'.

[@GuyInACube]

Did you define both the RangeStart and RangeEnd parameters?

[@peteflorenzano6604]

Great video! In reference to the content, I setup the date ranges and using an Azure SQL Data Warehouse back-end as my source. When performing the refresh via Desktop, I see the connection going to the database with the parameters that were created, however, after publishing the pbix to the service then refreshing manually, the connection that is made to the database is not using any parameters, but selecting all the data, despite my settings within the Incremental Refresh of storing rows for 3 years and refresh rows for the last 1 day with 'Detect data changes'

[@GuyInACube]

that is correct. The first refresh has to process the data. The first time it will pull all of the data to get a baseline. The first refresh will take a bit to process. After that, you will see your policy take effect.

[@peteflorenzano6604]

Great thank you. My refresh is timing out within the PowerBI service after three hours. Is there a setting where I can increase the timeout within the service itself? The timeout setting within the PowerQuery is set to 10 hours. Thank you :)

[@just4fun410]

HI Adam Can we use query (like last date or max date) in Rangeend parameter instead of giving end date, because if we give particular end date ,it means we are restricting data to that end date. If we reach end date again we need to change at PBIX. If m wrong can u give clarity.

[@GuyInACube]

I don't believe so as the parameters are overwritten once the service gets it after the first refresh. It may work for the initial refresh though. would need to try it.

[@TheHemanatvit]

This is a great addition to an already awesome tool! Quick questions about the advanced 'Detect Data changes', If I check this box and have Power BI look for changes in the last 3 weeks, Power BI engine will truncate and insert records for which the audit date column value changed?

[@ssohaibb]

Awesome information. We were having gateway timeout issues because the data, eventhough it only goes back a month, was about half a million rows and sharing the gateway with other reports was causing timeouts and failures. I am going to try this out. Thank you

[@yasielmesa1603]

Do gateway cluster, and make sure that the objects are well indexed will make a world of difference.

[@kishoredola7467]

Can i use ssas Live connection for incremental rwfeah?

[@annlaosun6260]

I'd like to know that too. Have you find out?

[@OzduSoleilDATA]

This is fascinating but I have a question about the cost. It looks like the minimum is $1000/month. So, this seems like an enterprise-level tool at that price. I've got a client with this exact need. We're using Dropbox and Power Query to process their data. Sometimes refreshes take over an hour and sometimes crashes when nearly complete. But I don't know that this small, family owned company is up for taking on $1000/mo.

[@GuyInACube]

You are correct on the cost for Premium capacity. This feature is coming for Pro/Shared Capacity in the future. I don't have a timeline though.

[@kirkjohns0n]

Hmm, am I missing something here? We're deploying embedded for our customers at work but when I looked at premium it started at ~4k/mo, not 1k. Is there some other pricing structure I should be aware of? We are using app owned embedded + RLS and are currently struggling with the time to refresh and the 1GB dataset limit. We've earmarked about 1k/mo to start powerBI, but I thought that would only get us the A-level SKUs and not premium. Am I missing something? Thanks! Great Videos!

[@elsimer]

I was told by MS that you can "test drive" Incremental Refresh on PBI Premium using A-level SKUs assuming that your dataset doesn't exceed the limitations on those SKUs

[@digikiwam]

@@kirkjohns0n I believe it is $4995 for one PowerBI Report Server Node. Since Premium includes Report Server, I am guessing this is the lowest tier for pricing on Premium. Microsoft's pricing structure on all it's products is somewhat complicated since they went DevOps. This "pay for what you use" is great in a sense, but confusing to figure out what your total costs will be, also makes it easy to get hit with "hidden fees" you didn't realize you incurred. Read the agreements twice, grab a fresh cup of your favorite beverage, then read the agreements twice more.

[@IndianYJ]

can yoiu do the incremental refresh in powerquery? or is it only a feature in power bi?

[@GuyInACube]

You define the Parameters within Power Query, but define the incremental refresh policy on a table in the data model. All within Power BI Desktop.

[@sifisongobese3113]

Hi Guys. Firstly thanks for update, Was having headache but its gone thanks to this update. Can we change the incremental refresh rate or rather update it. eg maybe change it to less or more years at any stage after publishing or we just re-setup on the desktop and publish again.

[@GuyInACube]

Glad the headache is gone :) you would need to change the settings within Desktop and re-deploy/re-process.

[@sifisongobese3113]

I seem to be stuck, I have done the necessary steps and published to the service, my only problem comes up when refreshing the dataset, it says "refresh failed due to gateway configuration issues. the dataset set requires a properly configured gateway in order to refresh. If your using enterprise mode ensure that you have added the following datasources in the Gateway Mangement Portal. > http: URL >Http:URL I have set the authentication level to : Anonymous I have set the privacy level setting to : blank this is on both desktop and service. Please advise on how to properly set the gateway.

[@rameshdurgam3589]

wow great video thank you

[@mauryavijai]

Excellent video. it help me to understand the concept very well.

[@JohnOttoKnoke]

Nice robe Dr. Adam, any hope this feature will be for Pro users?

[@GuyInACube]

Eventually. Not sure when. The May 2018 Power BI Desktop Blog indicated it will come at a later time - powerbi.microsoft.com/en-us/blog/power-bi-desktop-may-2018-feature-summary/#incrementalRefresh

[@GuyInACube]

haha! I had fun with the doctor robe at Microsoft Build. I'd start the day at the Power BI booth announcing the Power BI Doctor was in the house :)

[@zoloturu6322]

Good news! Thank you for the video! And I have question regarding this topic. What will be with such reports (with inremental refresh policy enabled) if we remove related workspace from Premium to Shared capacity? (data, structure changes etc.)

[@GuyInACube]

I got clarification from the engineering team. If you move the app workspace from Premium to Shared capacity, the dataset, with incremental refresh configured, will be in an unsupported state even if within 1GB. You will see a message saying to use the dataset it needs to be switched back to Premium. This is because incremental refresh is not currently supported within Shared capacity. Whenever that support does come, it should work.

[@zoloturu6322]

Thank you for response. At least report will not be corrupted and we will have ability to work with it once we move it back to Premium.

[@GuyInACube]

Yup!

[@SamuelRoshan]

Hi Adam, how do I address an old data coming back with an updated information, as there is no time limit on when an old row can be updated? Eg: I am performing Incremental refresh on the Orders table which has 5 years worth of data. I load orders from last 7 days only. But when Order that was posted 6 months ago returns with change in its "Status", how do I capture it and ensure latest changes are saved? Since we are only looking for changes made in the past 7 days, the row is identified by new record and is inserted into my Dataflow. This is resulting in duplication of Orders. Can there be an option where we also get to pick the Unique key column and ensure its not duplicated when its re-loaded?

[@akashjaiswal5203]

You need to address those changes in your ETL pipeline and not in BI. Add audit columns in your table and refresh the data accordingly in BI.

[@SamuelRoshan]

@@akashjaiswal5203 Yes thanks. And thats how I am able to perform incremental refreshes. The issue here is I have data from past coming back with updates. Since incremental refresh doesnt look for Primary/Unique key, they get duplicated

[@Rahul4u28]

Hi All, How to import large data set without memory error and timeout issue ? Am using Azure Table storage Data( which has upto 70 million rows) for one of my reports and trying incremental refresh by setting rangestart and rangeend parameter for an year. After setting Rangestart and Rangeend in power query. When close and apply the data is constantly loading and loading and its get struck after 40 - 60 Minutes. Which is purely a timeout issue. I did several checks like Memory usage, CPU performance, Clear local cache, reduce the range parameter to 6 month or 3 months but in spite of the am facing issues. Can someone assist me in this regard, is there anyone with whom I can connect over Microsoft teams for troubleshooting this issue. Looking for your help.

[@remeez]

This awesome, will this be coming to the rest api anytime soon?

[@GuyInACube]

What would you want from the REST API? You can already trigger a refresh from the REST API. That's really the only thing you can do in the Service with regards to this feature. Everything is configured within Power BI Desktop.

[@alexjunk6216]

Great content Adam and Christian! Any chance this incremental refresh setup is coming to AS? Cool new feature!

[@GuyInACube]

You can do partitioning in Analysis Services. That is what this is. :)

[@DranickMann]

If this only for PB Premium?

[@JohnOttoKnoke]

Same question here. Can you do it with Pro?

[@GHRichardson]

No. You have to have Premium. There's a diamond icon in the PBI dialog that they reference in the video, and they do also mention that it's a Premium only feature.

[@GuyInACube]

Geoff is correct! It is a Power BI Premium feature currently. The Power BI Desktop May 2018 blog indicates the plan is to bring it to Pro/Shared capacity eventually. No timeline was listed. powerbi.microsoft.com/en-us/blog/power-bi-desktop-may-2018-feature-summary/#incrementalRefresh

[@youssefzaki9142]

Thanks guys. Beautifully done

[@alitanveer]

Hi Adam. Great video and thank you for highlighting this potentially wonderful feature. I've been having a hell of time getting incremental refresh to work while using a MySQL query. I have a complicated query and I have to use several joins to get the data in the format I need. I've entered in the RangeStart and RangeEnd parameters into the WHERE clause of the query. It works fine as a filter to do the initial data load in PBI Desktop, but all data refreshes fail if I have incremental refresh turned on. Is there a different way of incorporating incremental refresh into a MySQL query? Thanks.

[@vamsichandemani2552]

Hey Ali, Got the solution to this problem?

[@alicedwonderland7733]

thank you so much for this video, guys! Wow, I did not know there are datasets out there more than 10 GB!!

[@alinafisher6256]

Hey! Thanks for the great video, as usual! Does anyone have good ideas on the repository for .pbix files since we're not able to download them back? Something with check ins/check outs

[@GuyInACube]

there are different systems out there. you could use GitHub or Azure DevOps for example. I'm sure there are others.