Data Science Workflow Process
In today’s rapidly evolving digital landscape, data has emerged as the new oil, a potent and invaluable resource that drives innovation, fuels industries, and shapes the very fabric of our society.
Just as oil, when extracted, needs refinement to unlock its full potential, so does data require the sophisticated and transformative process of data analytics.
Much like a refinery purifies crude oil, data analytics acts as the refining mechanism for raw data, distilling it into valuable insights and knowledge that can power informed decision-making and strategic planning. This is the very reason for which the Data Science Department of TD Africa Distributions Limited was formed.
The metaphorical relationship between data and oil underscores the critical role data analytics plays in our modern world. It highlights that data, like oil, exists in abundance, but its true value lies hidden beneath the surface.
TD Africa, being Africa’s foremost indigenous distributor of technology products, has a vast amount of data collected from her interactions with her multiple partners.
To extract this value, we rely on the refinery of data analytics, which employs advanced techniques, algorithms, and tools to convert vast and often complex datasets into actionable intelligence.
This intelligence empowers businesses, organizations, and individuals to navigate the intricate landscape of the digital age with precision, efficiency, and effectiveness. As such, the data science team was set up to help extract this intelligence.
The Data Science department has a mandate to help harness the potentials of the wealth of data produced by all processes and activities carried out on TD Africa’s robust cloud infrastructure by ensuring the availability of analytical solutions near real-time to drive decision making across the entirety of the business.
This manual provides an ample description of the system and processes required for the efficient and effective management of all DATA SCIENCE related activities as carried out by the department.
AUTHORITY / RESPONSIBILITY
The Head of Department is responsible for ensuring total compliance with the requirements stated in this procedural manual to maintain consistent workflow within the department.
DEPARTURE FROM PROCEDURE
Adherence to all procedures stated in this manual is obligatory for all employees of the department and as such, due authorization must be obtained from the Head of Department before any deviation from these procedures can occur.
TD AFRICA VISION, MISSION, AND QUALITY POLICY STATEMENT
Our Vision
Our vision is to be a global leader in distribution, empowering Africa with transformative technologies.
Our Mission
To ensure accessibility, affordability and useability of technology across Africa, empowering individuals and businesses, nurturing local innovations and steering a sustainable digital evolution
TD QUALITY POLICY STATEMENT
- We are committed to empowering all our stakeholders with exceptional experiences in the technology ecosystem. At TD Africa we know that poor quality products can cost loss of market, time and credibility. To uphold our quality policy, we maintain zero tolerance for deficiencies. We understand the importance of prompt quote response, quality products and timely delivery.
INTER- RELATIONSHIP WITH OTHER DEPARTMENT
| S/N | DEPARTMENT | DESCRIPTION OF FUNCTIONS | PROCESS INPUTS | PROCESS OUTPUTS |
| 1 | All Departments | Handles all Support Issues around: Dashboards, Reports, Data Apps, and Power Bl Administration | Issues are communicated via the MST. | Assignment of Data Scientist to treat. |
Creation of Dashboards, Reports and Analytical solutions. | Management instruction. Verbal/formal | Power Bl Dashboard, Report or Software Solution. | ||
| Assignment of permission or access to various dashboards or reports to Users. | Management or workspace administrator. Verbal/formal | Access granted to user with required access restrictions. | ||
| Training of Staff after implementation of new technology, new process, or modified process. | Implementation of new processes or modification to current processes. | Communication of date of training to employees and training on communicated dates. | ||
| 2 | MIS | Collaborate on Solution Deployments for D365F&O | Notice of deployment and request for timelines / approval | Confirmation of timelines and deployment of artifacts. |
HANDLING POWER BI USERS AND SUPPORT
ONBOARDING NEW USERS:
REQUEST FOR POWER Bl DASHBOARD ACCESS
Line manager sends a mail to datateam@tdafrica.com or calls the HOD to recommend the user for onboarding with access to a specified dashboard or report. The communication should detail:
- User job role
- User Email ID
- The Dashboard to be
- Once we receive the communication, the setup begins.
SETUP PROCESS
The setup process is as follows:
- A mail is sent to tdmis@tdafrica.com to assign a Power Bl Pro License to the User (an exception could be made where the user has a free trial license in which case, they would be required to send a mail to MIS after the trial is exhausted requesting a Power Bl Pro License with the Data team in copy).
- The user is then onboarded on Power
- The user is added to the required workspace, or the desired report or dashboard is shared with the user only with the appropriate access permissions.
- The user receives an onboarding email from Microsoft Power Bl detailing the access granted and a link to the dashboard or report.
REMOVAL OF USER ACCESS
On exit of a staff from the system (who has access to a workspace, dashboard, or report):
- HR sends a mail notifying us of staff
- We revoke all access granted to the exited user from all dashboards and reports.
SUPPORT
For a support request,
- A ticket, A call, mail request from the User about an
- Confirmation of fault claim by Data Science personnel via phone, in-person, or by remote
- If Data Scientist finds that it is a knowledge gap, they guide the user on possible
Corrections:
- If the issue is considered requiring correction and the scope is on Power Bl, the correction will be effected within 48 hours (if the power bi file will need to be edited and redeployed) or within a few hours (if the power bi file can be edited on web).
- If the issues are deemed to require a configuration on Dynamics, the issue will be added to a queue of planned Dynamics configurations and depending on the importance and complexity of the issue discussed as well as the time commitment required to execute, it would be ranked and executed in due time
DASHBOARD/REPORT DEVELOPMENT AND DEPLOYMENT
REPORT DEVELOPMENT
Dashboard development will always begin on assignment by the HOD. This could either be an aftermath of conversation with management, sales team, other HODs or experimental development of a possible solution for the company.
The processes involved in the development of analytical solutions for Dynamics for Finance and Operations are detailed below:
- Complete understanding of the intended solution must be
- Conduct proof of concept
- Check for availability of the
- Find all tables required to build the desired
- Create Tables, Forms and Data Entities to store the data where the data is not already stored in the ERP. Create views to consolidate all data
- Expose data to the entity store or OData sources (as required) using Aggregate Measurements and Dimensions.
- Never attempt to create a deployment package until all codes have been tested and have been found to be error-free on your development machine.
DEVELOPMENT FOR D365F&O ON VISUAL STUDIO
Development is a vital part of what we do on the Data Science desk because it is how we make data from Dynamics available for analytics on Power BI.
Development must be done by following these:
- All development work must be done in your personalized official development environment.
- All development, when done, must be certified by the Head of Department.
- Upon the certification of said model by the HOD, the project files as well as the package files should then be moved to TD DEV environment.
- At this point, the deployment process
DEPLOYMENT PROCESS
All deployments must be carried out in a supervised manner as deployments can potentially cause chaos in the ERP. All deployments must follow the deployment Pipeline which is stated below.
DEV ENVIRONMENT
With permission from the HOD, after all relevant verifications must have been done on the code, the deployment package or project files should be moved into the TD DEV environment for further testing.
- Proper communication must exist with technical consultants to ensure that all packages on TD DEV are safe for deployment.
- Once confirmed, a full build of the package to be deployed must be done on the TD DEV If successful, a full build of all packages on the TD DEV environment must be performed.
- If that is successful, a deployment package can thereafter be created consisting of all the pre-approved models sitting on the TD DEV environment.
- With the HOD’s permission, the deployment package may be moved from TD DEV environment to the Asset Library of the Microsoft Dynamics Life Cycle Services (LCS).
- MIS department must be contacted to ensure the availability of the Sandbox (TD SAT) environment for a test deployment.
SAT ENVIRONMENT
With the permission of the MIS department, the deployment package may be pushed to the TD SAT environment through LCS.
- If package deployment is unsuccessful, a rollback of the SAT environment must be performed immediately.
- A log of the failed deployment must be collected and analyzed to find the cause of the failure and only until that error is resolved should the package be allowed to be recreated and taken back to the beginning of the deployment cycle for a fresh
- If the deployment is successful, the MIS department must be
- The MIS department would be required to run all necessary environmental tests on the SAT environment to check for any unexpected outcomes.
- If errors are found, the last successful deployment before the recent change must be pushed to the SAT environment to correct the errors found and the codes must be sent back for further investigation and the root cause of the error established and resolved. Only until the error is completely resolved should the codes be allowed to begin the deployment cycle again.
- If errors are not found, the MIS department will be required to provide information on available times for a deployment to the Production Environment (TD PROD) to be performed.
PRODUCTION ENVIRONMENT
The production environment is the target environment for all codes and must be, again, checked for inconsistencies after each deployment.
- The deployed package must be examined thoroughly to ascertain that they are working perfectly in the production
- After packages are deployed successfully to Production, Power Bl files must then be moved for deployment to production.
DEPLOYMENT OF POWER BI FILES:
VIA MICROSOFT DYNAMICS LCS:
- The Power BI file must have been developed using Direct Query connection to the “localhost” server and database in the dev environment.
- The Power Bl file requiring the deployment package must be moved to the Power Bl files storage of the Asset Library.
- The File must be deployed to Microsoft Dynamics 365 for Finance and Operations using the Deploy Power Bl Files menu item of the System Administration Module of the production instance of Dynamics.
- The check box beside the file should be checked and the Deploy Power Bl file button at the top clicked.
- If deployment fails, the Power Bl file must be checked for adherence to the deployment rules for Dynamics 365 for Finance and Operations and if any rule was broken it must be fixed before any further attempted deployment is carried out.
- If deployment succeeds, the Power Bl Dashboard or Report can be accessed on Power Bl Web for further tests configuration and
- Dashboards or Reports deployed via this route will be deployed to the “My Workspace” workspace from where it can be shared.
VIA POWER BI DESKTOP:
- To deploy a Power BI file via Power BI Desktop’s “Publish” button, the file must have been developed using a connection other than the one described in A. above. Preferably, use connection to dataflows via Import mode.
- Save the document using the agreed naming convention.
- On the Power BI Desktop, click “Publish.”
- Sign-in to the admin account on the Dev environment and select the Workspace you want to deploy the file to.
- Click “Select”
- If there are no issues with the deployment, it should return a successful message with a link to view the report on Power BI Service.
NAMING CONVENTION
- Every new item created using x++ on Visual Studio for Dynamics 365 Finance & Operations must have a name that begins with TDL.
- Every Power BI File must be named according to what problem it solves or who will use it.