Introduction

Microsoft Fabric has a rich set of APIs to do many things from managing the environment to creating/read/updating/deleting (CRUD) items. In one of my customer projects, I needed to run notebooks via an API. The notebooks are orchestrators of other notebooks and were running around 175 notebooks each. It normally took around 20-40 minutes per orchestrator notebook to run.

To solve this, I created a Python notebook that calls the Fabric REST API, starts the orchestrator notebook, monitors its execution, and continues only after the process has completed. The implementation itself was straightforward, but I encountered a few unexpected issues along the way.

You might be wondering why I didn’t just use runMultiple() instead of using the APIs. In this particular scenario, I needed more flexibility than runMultiple() provided, which is why I chose to orchestrate the notebooks through the Fabric REST APIs. The most important reason was that runMultiple requires all the children to run under the same default lakehouse. While I normally try to not use default lakehouses, in this case each of the child orchestrators and all their children notebooks had been implemented with default lakehouses.

The main topics in this article are:

• Run a Fabric notebook using the Run On Demand Item Job API
• Authenticate using both interactive authentication and service principals
• Monitor long-running jobs using the Location header
• Handle Retry-After values safely
• Avoid token expiration issues in long-running processes

Architecture overview

Before going into the issues and how I solve them let´s look at how the process is. I’ll leave you with this image of the architecture but it really is just a Python notebook calling REST APIs and monitoring the responses to track progress.

Starting a notebook with the Run On Demand API

To run an item using the Fabric REST API you use the Run On Demand Item Job, Job Schedular endpoint (https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{itemId}/jobs/{jobType}/instances). You need to supply it with a workspace id, item id and item type. The easiest way to find the id´s is to open the item in the browser. In the URL you will find the workspace id after the groups/ and the item id at the end of the URL after the item type. For example, for notebooks you will find it after synapsenotebooks/. Here is an example of such an URL (with the id’s scrambled) https://app.fabric.microsoft.com/groups/dki84eu8-l98e-34iu-wo98-dkhf874jd7s9/synapsenotebooks/kfu87ds6-8fde-987r-sd98-dkoe876ste09.

The call is a POST call. If you need to override parameters, you can do so with a request body. Besides that, you need to supply it with a header, and you will need to acquire an access token for the Fabric APIs. You can use an interactive login if you are calling the endpoint from within Fabric like the below code shows. When running inside Fabric, notebookutils.credentials.getToken(‘pbi’) uses the identity under which the notebook is executing (the owner of the caller notebook).

Interactive “login” to get a token by using notebookutils:

token = notebookutils.credentials.getToken('pbi')

If you are running the notebook from a third-party tool or if you just prefer not to be reliant on named user accounts, you can use Service Principal instead. Then you acquire a access token in the below way.

Token gotten with service principal:

import msal 
import requests
 
# Service principal and tenant details
client_id = 'from Entra Id' 
tenant_id = 'The id of the tenant the item is in' 
secret = 'from Entra Id' 
authority = f'https://login.microsoftonline.com/{tenant_id}' 

# Use MSAL to get a token for Fabric APIs 
getToken = msal.ConfidentialClientApplication(
     client_id,
     authority=authority,
     client_credential=secret
 ) 

scope = ['https://api.fabric.microsoft.com/.default'] 
result = getToken.acquire_token_for_client(scopes=scope) 

if 'access_token' in result:     
    token = result['access_token']     
    print('Access token acquired successfully.') 
else:     
     print('Failed to acquire token:')     
     print(result)

When you have gotten a token, you add it to your header like below:

header = {'Content-Type':'application/json', 'Authorization':f'Bearer {token}'}

You can then call the Run On Demand Item Job endpoint like below (assuming you have the workspace and item ids in variables called workspaceId and itemId):

uri = f"https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{itemId}/jobs/instances?jobType=RunNotebook"

post_ItemRun = requests.post(url=uri, headers=header)

I left out a body to make it simpler but if you needed to supply the notebook with parameters you would deliver a JSON body

Monitoring the notebook execution

When you have called the endpoint, it will return with a header and a status. Status will just be if the call was successful or not (see more about HTTP status codes here: HTTP response status codes – HTTP | MDN). Hopefully you will have gotten a 201 which means that the call was successful, but the action is not completed. This is because the endpoint is ASYNC meaning that when you call it, the notebook will start but the call will not wait for it to finish. Instead, you will get a status message in the header including an URL you can call to check the current status of the run. The URL is in the Location variable in the header. One other key element is supplied in the header. That is how long you should wait until you check for the status. This is in the Retry-After element of the header. As you can see from the response header below it says I should wait at least 60 seconds until I check for the status.

{'Cache-Control': 'no-store, must-revalidate, no-cache', 'Pragma': 'no-cache', 'Content-Length': '0', 'Content-Type': 'application/octet-stream', 'Content-Encoding': 'gzip', 'Location': 'https://api.fabric.microsoft.com/v1/workspaces/08a4f5ad-e6df-4a03-abb7-6087bd0bfb99/items/7a8384a4-fb35-4d77-bbff-cd564324cc6f/jobs/instances/803c9e11-1eqe-4226-a917-4590dfe1c1c7', 'Retry-After': '60', 'x-ms-job-id': '803c9e11-1eqe-4226-a917-4590dfe1c1c7'', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'RequestId': 'bd9c4647-62a0-4e44-b408-029587b1d081', 'Access-Control-Expose-Headers': 'RequestId,Location,Retry-After', 'request-redirected': 'true', 'home-cluster-uri': 'https://wabi-west-europe-f-primary-redirect.analysis.windows.net/', 'Date': 'Mon, 29 Jun 2026 11:29:16 GMT'}

When I use the new check for status endpoint (the Location from above), I will get a similar header as a response including the time to wait before you check again. Here I had my first issue. In some instances, the return header does not contain a wait time and in others it has a seemingly random wait time. Sometimes it´s 60 sec, sometimes it´s 90 sec, sometimes it´s 10 sec and there does not seem to be any reason too it. I assume there is but I didn´t see it :-). My issue was that when the wait time was missing, I called it straight away as there was no wait time and if this happened few times in a row, I could get throttled. There is nothing in the documentation about throttling and when it occurs but there is some mechanism in place for it. Because my orchestration processes are long-running, I added a safeguard that enforces a minimum wait time of 60 seconds. This reduced the risk of hitting throttling limits when Retry-After was missing. This might not work in your case as you might have a short process, so you need to find an acceptable waiting time for your project.

if int(RetryAfter)<=60:
   RetryAfter = 60
time.sleep(int(RetryAfter))

Handling token expiration in long-running jobs

My main issue was that if I had a long running process of anything above 15 min (which my orchestration notebooks are) I randomly had an error saying 401 Unauthorized. Since this happened at random times, I was certain it was because I was being throttled. But after some back and forward with Simon Sabin (thank you Simon) I managed to find out that what was happening was that my token was expiring. It caught me off guard as it didn´t happen after fixed amount of time. Sometimes it happened after 30 min, sometimes after 45 min. But in the header response there is a status (TokenExpired) which says if your token has expired. You can either watch for that status in the response header, or you can brute force it like I did and try to get a new token if the HTTP status is 401 (or anything in the 400 range)

http_status = result["http_status"]
if http_status==401: # can use >=400
        token = notebookutils.credentials.getToken('pbi')
        header = {'Content-Type':'application/json', 'Authorization':f'Bearer {token}'}

Then I can keep on calling the status endpoint with the new token.

Lessons learned

• You cannot rely on the response even if it comes from Microsoft themselves. You have to safeguard against things like missing Retry-After although you would think it was in Microsoft interest to include it always
• There seems to be throttling somewhere in the system but it’s not documented so I have no way of saying if it’s just a feeling or true. I would imagine that Microsoft have it built in to safeguard themselves
• Tokens expire and they don’t do so after fixed amount of time (in my experience). I don’t know exactly from when they start to count but I assume it’s from the time I acquired the token. This caused me a lot of headaches as I refused to believe it was a token expiration as it was after different number of minutes each time.
• There are great minds out there. Expand your network and use it. There might be someone out there who can help you when you have hit a wall like Simon helped me.

Conclusion

After adding safeguards for missing Retry-After values and refreshing expired tokens, the solution has been running reliably in production for more than five months.

If you’re building long-running Fabric orchestrations using the REST APIs, hopefully these lessons will save you some debugging time and help you avoid the same pitfalls.

About the author

A, Gunnarsson (09/07/2026) Using the Fabric Rest API to run an item. (1) Using the Fabric Rest API to run an item | LinkedIn