DMF Blob Temporary Storage

Insight on how the Data Management Framework uses the Azure Blob to temporarily store exported files. Includes a code sample on how to delete these files from the Blob containers.

Background

When you import or export a file using the Data Management Framework in Microsoft Dynamics 365 Finance and Operations (D365FO) a temporary copy of the file is created in Azure Blob. Its temporary because the file created has an expiration date, currently hard-coded to 7 days (10080 minutes) and re-generated with a different GUID every time a user downloads the file.

To demonstrate this on a developer VM we will use the Azure Storage Emulator and the Azure Storage Explorer which you can download for free. In the following demo we will export and download the Customer Groups data entity.

Export Data Project

In this example, we create an export data project to export two entities (Customer groups and Vendor groups) in Excel format.

There are two options how to export the data. We can either click on the download action pane button or click on the export menu item button.

Both the export menu item and the download button will create the Excel files in the blob storage “dmf” folder as shown below.

Send File to User

The download button will take a step further by packaging the files in a single zip file and it send it to the user client browser. To send the file to the user a temporary file is created in the blob storage with a download link (URL).

Example download URL = http://127.0.0.1:10000/devstoreaccount1/temporary-file/%7BACCE3CFE-3EC5-431A-9245-003F298D3C01%7D/Export%20Demo_525CDCF8-5EE9-4D87-89E2-49F9023BA96F_DMFPackage.zip?sv=2014-02-14&sr=b&sig=2unM4CQ8jDUjQ5QWHV0vGsx9OJBHnLa22ekqUj4ZXEA%3D&st=2019-09-26T12%3A54%3A51Z&se=2019-09-26T13%3A59%3A51Z&sp=r

You can see the file in the temporary-file blob container of the Azure Emulator.

The download URL has an expiration time in minutes which you can specify from System Administration > Setup > System parameters > Blob link expiration timespan. If this is left to zero, a default expiration of 10 minutes is applied.

Download Link

When the export finishes executing, it will create the files in the “dmf” storage but will not send them to the user and therefore will not create the files in the “temporary-file” blob container yet. When the DMFExecutionHistoryList form opens, you will have a “Download file” that generates the file in the “temporary-file” blob container and a download URL is provided.

The two Blob containers “dmf” and “temporary-file” can be found in the #DMF macro as:

#define.DmfExportContainer(‘dmf’)

#define.TemporaryStorageContainer(‘temporary-file’)

The temporary container is also a public constant string in the FileUploadTemporaryStorageStrategy class.

Deleting the temporary files

In a recent project we had to export files using the Data Management Framework by executing the data project export functionality from code. One of the requirements was to delete the files from the Blob container once the file was sent to the user browser via the download link. Below is an extract of the code to achieve this.

Code Sample

First of all, file identifiers that are downloaded and therefore created in the “temporary-file” blob container are not stored anywhere in the D365FO database because they are created on the fly by generating a new GUID every time (refer to method uploadFile in class FileUploadTemporaryStorageStrategy). We therefore have to create our own log table and extend the method uploadFile which is generating the GUID to save these file ids in our custom table.

[ExtensionOf(classStr(FileUploadTemporaryStorageStrategy))]
public final class BFTFileUploadTemporaryStorageStrategy_Extension
{
    public FileUploadResultBase uploadFile(System.IO.Stream _stream, str _fileName, str _contentType, str _fileExtension, str _metaData)
    {
        FileUploadResultBase fileUploadResult = next uploadFile(_stream, _fileName, _contentType, _fileExtension, _metaData);
        
        if (fileUploadResult is FileUploadTemporaryStorageResult
            && fileUploadResult.getUploadStatus())
        {
            BFTFileUploadResult fileUpload; //Custom table to store uploaded file ids to the temporary blob
            FileUploadTemporaryStorageResult fileUploadResultTempStorage = fileUploadResult as FileUploadTemporaryStorageResult;
            fileUpload.Filename     = fileUploadResultTempStorage.getFileName();
            fileUpload.FileId       = fileUploadResultTempStorage.getFileId();
            fileUpload.insert();
        }
        return fileUploadResult;
    }
}

Next step is to create a class (can be batch executed nightly) that can delete these temporary files. In the example below we have a runnable class that loops all exported files in both the “dmf” and “temporary-file” containers that were exported and/or downloaded.

using Microsoft.DynamicsOnline.Infrastructure.Components.SharedServiceUnitStorage;
class DeleteDMFBlobFiles
{        
    public static void main(Args _args)
    {
        #DMF        
        var blobStorageService = new SharedServiceUnitStorage(SharedServiceUnitStorage::GetDefaultStorageContext());
        str azureStorageCategory = #DmfExportContainer;
        DMFEntityExportDetails entityExportDetails;
        while select entityExportDetails
        {
            if (entityExportDetails.SampleFilePath)
            {
                blobStorageService.deletedata(entityExportDetails.SampleFilePath, azureStorageCategory);
            }
                
            if (entityExportDetails.PackageFilePath)
            {
                blobStorageService.deletedata(entityExportDetails.PackageFilePath, FileUploadTemporaryStorageStrategy::AzureStorageCategory);
            }
        }
        BFTFileUploadResult bftFileUploadResultTemp;
        while select bftFileUploadResultTemp
        {
            blobStorageService.deletedata(bftFileUploadResultTemp.FileId, FileUploadTemporaryStorageStrategy::AzureStorageCategory);
        }
    }
}

Article written for Bluefort Malta