Data management overview - Finance & Operations | Dynamics 365 (2023)

  • Article
  • 18 minutes to read

Important

The functionality noted in this article is currently available in both the stand-alone Dynamics 365 Human Resources and the merged Finance infrastructure. Navigation might be different than noted while we make updates. If you need to find a specific page, you can use Search.

This article describes how you can use the data management framework to manage data entities and data entity packages in finance and operations.

The data management framework consists of the following concepts:

  • Data entities - A data entity is a conceptual abstraction and encapsulation of one or more underlying tables. A data entity represents a common data concept or functionality, for example, Customers or Vendors. Data entities are intended to be easily understood by users familiar with business concepts. After data entities are created, you can reuse them through the Excel Add-in, use them to define import/export packages, or use them for integrations.
  • Data project - A project that contains configured data entities, which include mapping and default processing options.
  • Data job - A job that contains an execution instance of the data project, uploaded files, schedule (recurrence), and processing options.
  • Job history - Histories of source to staging and staging to target jobs.
  • Data package - A single compressed file that contains a data project manifest and data files. This is generated from a data job and used for import or export of multiple files with the manifest.

The data management framework supports using data entities in the following core data management scenarios:

  • Data migration
  • Set up and copy configurations
  • Integration

Data entities

Data entities provide conceptual abstraction and encapsulation of underlying table schema that represent data concepts and functionalities. In Microsoft Dynamics AX 2012, most tables, like the Customer and Vendor tables, were de-normalized and split into multiple tables. This was beneficial from a database design point of view, but made it difficult for implementers and ISV's to use without a thorough understanding of the physical schema. Data entities were introduced as part of data management to be used as a layer of abstraction to easily understand by using business concepts. In previous versions there were multiple ways to manage data, such as Microsoft Excel Add-ins, AIF, and DIXF. The concept of data entities combines those different concepts into one. After data entities are created, you should be able to reuse them for an Excel Add-ins, import/export, or integration. The following table shows core data management scenarios.

Data Migration
  • Migrate reference, master, and document data from legacy or external systems.
Setup and copy configuration
  • Copy configuration between company/environments.
  • Configure processes or modules using the Lifecycle Services (LCS) environment.
Integration
  • Real-time service based integration.
  • Asynchronous integration.

Data migration

Using the data management framework, you can quickly migrate reference, master, and document data from legacy or external systems. The framework is intended to help you quickly migrate data by using the following features:

  • You can select only the entities you need to migrate.
  • If an import error occurs, you can skip selected records and choose to proceed with the import using only the good data, opting to then fix and import the bad data later. You will be able to partially continue and use errors to quickly find bad data.
  • You can move data entities straight from one system to another, without having to go through Excel, or XML.
  • Data imports can be easily scheduled using a batch, which offers flexibility when it is required to run. For example, you can migrate customer groups, customers, vendors, and other data entities in the system at any time.

Set up and copy configuration

You can use the data management framework to copy configurations between companies or environments, and configure processes or modules using Microsoft Dynamics Lifecycle Services (LCS).

Copying configurations is intended to make it easier to start a new implementation, even if your team doesn't deeply understand the structure of data that needs to be entered, or data dependencies, or which sequence to add data to an implementation.

The data management framework allows you to:

  • Move data between two similar systems
  • Discover entities and dependencies between entities for a given business process or module
  • Maintain a reusable library of data templates and datasets
  • Use data packages to create incremental data entities. Data entities can be sequenced inside the packages. You can name data packages, which can be easily identifiable during import or export. When building data packages, data entities can be mapped to staging tables in grids or by using a visual mapping tool. You can also drag-and-drop columns manually.
  • View data during imports, so you can compare data, and ensure that it is valid.

Working with data entities

The following sections provide quick snapshots of the different functionalities of data management using data entities. The goal is to help to you strategize and make effective decisions on how to best utilize the available tools during data migration. You will also find tips and tricks on how to effectively use each area during data migration. A list of available data entities for each area can also be found with the suggested data sequences, showing data dependencies. Microsoft provides data packages that can be found on Lifecycle Services (LCS) as an initial guide. The information in this document can be used as a guide for creating your own packages. The description of each data entity shows what the object contains and if it is needed during data migration.

Sequencing

There are two types of sequencing that should be considered when working with data entities.

  • Sequencing data entities within a data package
  • Sequencing the order of data package imports

Sequence data entities within a data package

  1. When a user adds data entities to a data project, by default, a sequence is set for the order in which the entities will load. The first entity added to the project will be set as the first entity to load, the next entity added will be second, the next entity will be third, and so on.

    (Video) Session-1: Data Management Introduction | Data Management Series | D365 for Operations | D365Zone

    For example, if a user added two entities in this order, Sales tax codes and Sales Tax groups, then Sales tax codes is assigned an entity sequence of 1.1.1, and Sales tax groups is assigned an entity sequence of 1.1.2. The sequence level indicates that the second entity will not start the import process until the first level is finished.

  2. To view or edit a sequence, click the Entity sequence button on the Action Pane of the data project.

    Data management overview - Finance & Operations | Dynamics 365 (1)

  3. In the Definition group entity sequence, you can see the execution units and the sequence. You can change sequence by selecting the data entity in the list, setting a different Execution unit or Sequence in level, and then clicking Update selected. After clicking Update selected, the entity will move up or down in the entity list.

Example

The following screenshot shows the entity sequence that is set for the Sales Tax CodeGroups data package.

Data management overview - Finance & Operations | Dynamics 365 (2)

In order to successfully import sales tax codes and groups, the sales tax codes and details have to be loaded first, before sales tax groups can be imported. Sales tax codes and groups are all in Execution unit = 1, but the sequences are in the order that they will be imported. Other related sales tax entities that are not dependent upon other data entities being loaded are included in the package. For example, sales tax exempt numbers is set in its own Execution unit = 2. This data entity will start loading immediately because there are no dependencies on other entities loading before it.

Sequence data package imports

In order to successfully load data, it's important to set the correct order for importing data packages, because of dependencies that exist within and across modules. The numbering format that has been created for the data packages within LCS are as follows:

  • First segment: Module
  • Second segment: Data type (setup, master, transaction)
  • Third segment: Sequence number

The following tables provide more information about the default numbering format.

Module numbers

Data type numbers

Sequence number

(Video) EP07 Data Management Framework for D365 FinOps

Data packages follow the sequence number, followed by the module abbreviation, and then a description. The following example shows General ledger data packages.

Mapping

When working with data entities, mapping an entity to a source is automatic. The automatic mapping of fields can be overridden if needed.

View mapping

To view how an entity is mapped, locate the tile for the entity in the project, and then click View map.

We provide mapping visualization view (default) and mapping details view. A red asterisk (*) identifies any required fields in an entity. These fields must be mapped in order to work with the entity. Other fields can be unmapped as required when working with the entity.

  • To unmap a field, highlight the field in either column (Entity or Source), click Delete selection, and then click Save. After saving, close the form to return to the project.

The field mapping from source to staging can also be edited after import using the same process.

Regenerate a map

If you have extended an entity (added fields) or if the automatic mapping appears to be incorrect, the mapping of the entity can be regenerated in the Mapping form.

  1. To do this, click Generate source mapping.

    A message will display asking, "Do you want to generate the mapping from scratch ?"

  2. Click Yes to regenerate the mapping.

Generate data

If you have fields in entities that you want the system to generate data for on import, instead of providing the data in the source file, you can use the auto-generated functionality in the mapping details for the entity. For example, if you want to import customers and customer address information, but the address information was not previously imported with the Global Address Book entities, you can have the entity auto-generate the party number upon import and the GAB information will be created. To access this functionality, view the map of the entity and click the Mapping details tab. Select the fields that you want to auto-generate. This will change the source field to Auto.

Turn off automatically generated number sequences

Many entities support automatic generation of identifiers based on number sequence setup. For example, when creating a product, the product number is automatically generated and the form does not allow you to edit values manually.

It is possible to enable manual assignment of number sequences for a specific entity.

(Video) Dynamics 365 Finance, Supply Chain Management Data Management

After you have enabled manual assignment, you can provide manually assigned numbers instead.

Export

Export is the process of retrieving data from a system using data entities. The export process is done through a project. When exporting, you have a lot of flexibility as to how the export project is defined. You can choose which data entities to export, but also the number of entities, the file format used (there are 14 different formats to choose for export), and apply a filter to each entity to limit what is exported. After the data entities have been pulled into the project, the sequencing and mapping described earlier can be performed for each export project.

After the project is created and saved you can export the project to create a job. During the export process, you can see a graphical view of the status of the job and the record count. This view shows multiple records so you can review the status of each record prior to downloading the actual files.

After the job is completed you can choose how to download the files: each data entity can be a separate file, or by combining the files into a package. If there are multiple data entities in the job, choosing the package option will speed up the upload process. The package is a zip file, containing a data file for each entity as well as a package header and manifest. These additional documents are used when importing in order to add the data files to the correct data entities and sequence the import process.

Import

Import is the process of pulling data into a system using data entities. The import process is done through the Import tile in the Data Management workspace. Data can be imported either for individual entities or for a group of logically related entities that are sequenced in the correct order. The file formats vary depending on the type of import. For an entity, it can be an Excel file that is comma-separated, tab-separated, or text. For a data package, it is a .zip file. In both cases, the files are exported using the above mentioned export process.

Import a data package

  1. Log into the environment using a login with sufficient privileges (typically this is the Administrator role).

  2. On the dashboard, click the Data Management workspace.

  3. Click the Import tile.

  4. On the next page, do the following:

    1. Provide a name.

    2. In the Source Data Format field, select Package.

      (Video) Dynamics 365 for Finance and Operations Data Management Tech Talk

    3. Click the Upload button and choose the appropriate package file from the location for the data being imported. This will import all the files from the package.

    4. Click Save, and then click Import.

Import multiple data packages

Use one of the following methods to import multiple data packages.

  • Create a new job for each package, and then repeat steps 4(a) through 4(d) above, for each package.

  • Create one job to import multiple packages in a sequence. Repeat steps 4(a) through 4(c) above, and then repeat step 4(c) for all packages that need to be imported. After you select the packages, execute step 4(d) to import the data from the selected data packages through a single job.

After you click Import, the data will be imported through staging tables. The progress of the import can be tracked using the Refresh button in the upper-right corner of the screen.

Troubleshoot data package processing

This section provides troubleshooting information for the different stages of data package processing.

  • Status and error details of a scheduled job can be found under the Job history section in the Data management form.
  • Status and error details of previous runs for data entities can be displayed by selecting a data project and clicking Job history. In the Execution history form, select a job, and click View staging data and View execution log. The previous runs include data project runs that were executed as batch jobs or manually.

Export process troubleshooting

  • If you get an error during the export process, click View execution log and review the log text, staging log details, and Infolog for more information.
  • If you get an error during the export process with a note directing you to not skip staging, turn off the Skip staging option, and then add the entity. If you are exporting multiple data entities, you can use the Skip staging button for individual data entities.

Import process troubleshooting

When uploading data entity files:

  • If data entities do not display in Selected files and entities after you click Upload during the import process, wait a few minutes, and then check whether the OLEDB driver is still installed. If not, then reinstall the OLEDB driver. The driver is Microsoft Access Database Engine 2010 Redistributable – AccessDatabaseEngine_x64.exe.
  • If data entities display in Selected Files and Entities with a warning after you click Upload during the import process, verify and fix the mapping of individual data entities by clicking View map. Update the mapping and click Save for each data entity.

During data entity import:

  • If data entities fail (shown with a red X or yellow triangle icon on the data entity tile) after you click Import, click View staging data on each tile under the Execution summary page to review the errors. Sort and scroll through the records with Transfer status = Error to display the errors in the Message section. Download the staging table. Fix a record (or all records) directly in staging by clicking Edit, Validate all, and Copy data to target, or fix the import file (not staging file) and reimport the data.
  • If data entities fail (shown with a red x or yellow triangle icon on the data entity tile) after you click Import, and View staging data shows no data, go back to the Execution summary page. Go to View execution log, select the data entity, and review the Log text, Staging log details, and Infolog for more information. Staging log details will display Error column (field) details and Log description will describe errors in detail.
  • If data entities fail, you can check the import file to see if there's an extra line in the file with text which displays, "This is a string that is inserted into Excel as a dummy cell to make the column to support more than 255 characters. By default, an Excel destination component will not support more than 255 characters. The default type of Excel will be set based on the first few rows". This line is added during data export. If this line exists, delete this line, re-package the data entity, and try to import.

Features flighted in data management and enabling flighted features

The following features are enabled via flighting. Flighting is a concept that allows a feature to be ON or OFF by default.

Flight nameDescription
DMFEnableAllCompanyExportEnables BYOD export from all companies in the same export job (supported for BYOD only and not files). By default, this is OFF. This flight is no longer needed after Platform update 27 because this feature can be turned ON using a parameter in data management framework parameters.
DMFExportToPackageForceSyncEnables synchronous execution of data package API export. By default, it's asynchronous.
EntityNamesInPascalCaseInXMLFilesEnables behavior where entity names are in Pascal Case in the XML files for entities. By default, the names are in upper case.
DMFByodMissingDeleteEnables the old behavior where under certain conditions, certain delete operations were not synced to BYOD using change tracking.
DMFDisableExportFieldsMappingCacheDisables caching logic when building target field mapping.
EnableAttachmentForPackageApiEnables attachments functionality in the package API.
FailErrorOnBatchForExportEnables fail on error at execution unit or level for export jobs.
IgnorePreventUploadWhenZeroRecordDisables 'prevent upload when zero records' functionality.
DMFInsertStagingLogToContainerBy default this is ON. This improves performance and functional issues with error logs in the staging table.
ExportWhileDataEntityListIsBeingRefreshedWhen enabled, additional validations are made on mappings when a job is scheduled while entity refresh is in progress. By default, this is OFF.
DMFDisableXSLTTransformationForCompositeEntityThis can disable the application of transformations on composite entities.
DMFDisableInputFileCheckInPackageImportAdditional validations are made to ensure if any entity file is missing from a data package, error message is shown. This is the default behavior. If required, this can be turned OFF by this flight.
FillEmptyXMLFileWhenExportingCompositeEntityPrior to Platform update 15, when exporting composite entities that did not have any records to export, the XML file generated did not have any schema elements. This behavior can be changed to output empty schema by enabling this flight. By default, the behavior will still be to output empty schema.
EnableNewNamingForPackageAPIExportA fix was made to ensure unique names are used for the execution ID when ExportToPackage is used for export scenarios. Duplicate execution ID's were being created when ExportToPackage was called in quick succession. To preserve compatibility, this behavior is OFF by default. Turning this flight ON will enable this new behavior where new naming convention for execution ID's will ensure unique names.
DMFDisableDoubleByteCharacterExportA fix was made to ensure that data can be exported when the format is configured to use code page 932 setting. If an issue is encountered in relation to double byte exports, this fix can be turned off by disabling this flight to unblock, if applicable.
DisablePendingRecordFromJobStatusA fix was made to ensure that pending records are taken into consideration while evaluating the final status of an import job. If implementations have a dependency on the status evaluation logic and this change is considered a breaking change for an implementation, this new logic can be disabled using this flight.
DMFDisableEnumFieldDefaultValueMappingA fix was made to ensure that default values set in advanced mapping for enum fields are successfully saved in the data package manifest file when generating the data package. This makes it possible for the data package to be used as a template for integrations when such advanced mappings are used. This fix is protected by this flight and can be disabled if the previous behavior is still needed (which is to always set the value to 0 in the data package manifest).
DMFXsltEnableScriptThis flight only applies to Platform update 34 and non-production environments. A fix was made in Platform update 34 to prevent scripting in XSLT. However, this resulted in breaking some functionality that was dependent on scripting. As a result, this flight has been turned ON by Microsoft in all production environments as a preventive measure. In non-production environments, this must be added by customers if they encounter XSLT failures related to scripting. From Platform update 35 onward, a code change was made to revert the Platform update 34 change so this flight does not apply from Platform update 35 onward. Even if you enabled this flight in Platform update 34, upgrading to Platform update 35 will not cause any negative impact due to this flight being ON from Platform update 34.
DMFExecuteSSISInProcThis flight is OFF by default. This is related to a code fix that was made to run SQL Server Integration Services (SSIS) out of in-process to optimize memory utilization of SSIS when running DIXF jobs. However, this change has caused a regression in a scenario where if the DIXF data project name has an apostrophe (') in it, then the job will fail with an error. If you encounter this issue, removing the (') in the data project name will resolve the failure. However, if for some reason the name cannot be changed, then this flight can be enabled to overcome this error. Enabling this flight will make SSIS run in-process as before, which could lead to higher memory consumption when running DIXF jobs.

The following steps enable a flight in a Tier-1 environment. Execute the following SQL command.

For enabling flights in a production or sandbox environment, a support case must be logged with Microsoft.

  • After running the SQL statement, ensure that the following is set in the web.config file on each of the AOS's.add key="DataAccess.FlightingServiceCatalogID" value="12719367"

  • After making the above change, perform an IISReset on all AOS's.

    (Video) Dynamics 365 Finance & Operations Data Migration and Data Management Best Practices

    INSERT INTO SYSFLIGHTING([FLIGHTNAME],[ENABLED],[FLIGHTSERVICEID],[PARTITION],[RECID],[RECVERSION])VALUES ('name', 1, 12719367, PARTITION, RECID, 1)
  • Partition - Partition ID from the environment, which can be obtained by querying (select) for any record. Every record will have a partition ID that must be copied and used here.

  • RecID - Same ID as partition. However, if multiple flights are enabled, then this can be partition ID + 'n' to ensure it has a unique value

  • RecVersion = 1

Additional resources

  • Data entities overview

FAQs

How does Data management work in Dynamics 365 for finance and operations? ›

You can export or import data in finance and operations apps by using the Data management workspace. Validate the data by staging the source data, and then move it to the target tables in the finance and operations apps database.

What is included in Dynamics 365 Finance and Operations? ›

Microsoft Dynamics 365 Finance & Operations

Dynamics 365 Finance: Includes budgeting, project management, financials, and accounting for large, international companies. Dynamics 365 Supply Chain Management: Includes engineering, manufacturing, warehousing, and distribution.

What is Data management in Dynamics 365? ›

The data management framework allows you to: Move data between two similar systems. Discover entities and dependencies between entities for a given business process or module.

What is DMF in d365fo? ›

Data Management Framework (DMF) entities will be available for benefits management. The entities will enable setup and enrollment scenarios and include the following: Entity support for setup entities. Entity support for transaction-level entities.

What are the 5 data management functions? ›

It is often referred to by its acronym, DBMS. The functions of a DBMS include concurrency, security, backup and recovery, integrity and data descriptions.

Does Dynamics 365 finance and operations use Dataverse? ›

Get your Finance and Operations apps data in Microsoft Dataverse and keep it up to date. We are making the dual-write framework a seamless experience by harmonizing the converging concepts between Finance and Operations apps and customer engagement apps in Dynamics 365.

Is Dynamics 365 a CRM or ERP? ›

Microsoft Dynamics 365 is a cloud-based business applications platform that combines components of customer relationship management (CRM) and enterprise resource planning (ERP), along with productivity applications and artificial intelligence tools.

Why choose Dynamics 365 Finance and Operations? ›

Better Associated Software Integration

With Microsoft Dynamics 365 for Finance and Operations, it is simpler to integrate and customize different modules and components of your business and productivity software, such as CRM, BI, different IoT applications, productivity tools, etc.

What are the 3 main processes of data management? ›

MDM helps ensure businesses don't use multiple, potentially inconsistent versions of data in different parts of business, including processes, operations, and analytics and reporting. The three key pillars to effective MDM include: data consolidation, data governance, and data quality management.

What are the 3 data management approaches? ›

Three approaches to implement data governance include 1) the Command and Control approach, 2) the Traditional approach, and 3) the Non-Invasive approach. This article compares and contrasts the approaches and quickly summarizes each approach.

What does MRP stand for in D365? ›

Planning Optimization support for materials requirements planning (MRP) - Dynamics 365 Release Plan | Microsoft Learn.

What is the benefit of DMF? ›

The primary benefits of a DMF for plastic use include legal protection for all entities involved, confidentiality of specific formulations, documented accuracy of chemistries, and communication of different parties involved.

What is primary key in D365FO? ›

The primary key is usually the type of key that other tables, called child tables, refer to when a foreign key field in those other tables need a relational identifier.

What are the 5 A's of data? ›

In order to understand at what point 'data' transitions into being 'big data', and what its key elements are, it is imperative that we study the 5 Vs associated with it: Velocity, Volume, Value, Variety, and Veracity.

What is the difference between Dataverse and Dynamics 365? ›

In short, Dataverse is a ready-to-use server that offers a security layer, a business layer, a data access layer and so on. Dynamics CRM solutions store their data on a Dynamics server, the business logic is implemented via plugins on Dataverse.

Is Dataverse a SQL database? ›

The Microsoft Dataverse business layer provides a Tabular Data Stream (TDS) endpoint that emulates a SQL data connection. The SQL connection provides read-only access to the table data of the target Dataverse environment thereby allowing you to execute SQL queries against the Dataverse data tables.

Does Dynamics 365 use SQL? ›

Dynamics 365 Server requires an instance of SQL Server Reporting Services be installed, running, and available. All installations of the supported SQL Server editions can be used as the reporting server.

What is Dynamics 365 in simple words? ›

Dynamics 365 is a portfolio of intelligent business applications that delivers superior operational efficiency and breakthrough customer experiences enabling businesses to become more agile and reduce complexity without increasing costs.

What is replacing Dynamics 365? ›

Dynamics 365 connectors must be replaced with a Microsoft Dataverse connector. Microsoft Dataverse – This is the newest connector and will, in time, become the only connector available for Dataverse usage. All new features, performance work, and enhancements will be available on this connector.

Is Dynamics 365 difficult to learn? ›

A lot of users find it easy to use Microsoft Dynamics 365, especially if they have used any other Microsoft apps before.

Which is better Salesforce or dynamics? ›

We recommend Salesforce for younger companies and smaller businesses because of its lower starting price and its extensive integrations with third-party applications. Microsoft Dynamics 365 is better for larger and more established businesses, especially ones that already use the Microsoft Office suite extensively.

What are the five 5 general data types? ›

Most modern computer languages recognize five basic categories of data types: Integral, Floating Point, Character, Character String, and composite types, with various specific subtypes defined within each broad category.

How is data processing used in finance? ›

Data analytics helps finance teams gather the information needed to gain a clear view of key performance indicators (KPIs). Examples include revenue generated, net income, payroll costs, etc. Data analytics allows finance teams to scrutinize and comprehend vital metrics, and detect fraud in revenue turnover.

What is data management and how it works? ›

Data management is the practice of collecting, organizing, protecting, and storing an organization's data so it can be analyzed for business decisions. As organizations create and consume data at unprecedented rates, data management solutions become essential for making sense of the vast quantities of data.

How do data management systems work? ›

Database Management Systems (DBMS) are software systems used to store, retrieve, and run queries on data. A DBMS serves as an interface between an end-user and a database, allowing users to create, read, update, and delete data in the database.

What is master data management in finance? ›

What is Master Data Management? Master data management (MDM) refers to collaboration across business units and departments in an organization in relation to the orchestration, enablement, and workflow of a given data domain. In the financial services sector, data domains typically include client, product, and assets.

Videos

1. How to Import Products in FinOps using data management.
(FinOps lyceum)
2. Dynamics 365 for finance and operations - Demo Data Packages
(Microsoft Dynamics 365)
3. Dynamics 365 Finance/SCM: Data Management TechTalk
(Michael Cobbin - Dynamics 365)
4. Make The Most Of Data Management In Dynamics 365 Finance & Operations D365FO
(AXcademy)
5. Data Management work space Import in Dynamics 365 Finance and Operations
(Dynamics Post)
6. Dynamics 365 Finance & Operations - Module 11: Introduction to Advanced Development Topics
(Microsoft Dynamics 365)
Top Articles
Latest Posts
Article information

Author: Duncan Muller

Last Updated: 01/07/2023

Views: 6315

Rating: 4.9 / 5 (59 voted)

Reviews: 90% of readers found this page helpful

Author information

Name: Duncan Muller

Birthday: 1997-01-13

Address: Apt. 505 914 Phillip Crossroad, O'Konborough, NV 62411

Phone: +8555305800947

Job: Construction Agent

Hobby: Shopping, Table tennis, Snowboarding, Rafting, Motor sports, Homebrewing, Taxidermy

Introduction: My name is Duncan Muller, I am a enchanting, good, gentle, modern, tasty, nice, elegant person who loves writing and wants to share my knowledge and understanding with you.