what prevent invalid data into database
Answers
Answered by
0
<<HEY MATE HERE'S YOUR ANSWER>>
Problem
We receive data from thousands of sources, some of which provide direct access and some of which manually enter their data into their system or our system. We run a multi-server and database ETL flow that obtains the original data, transforms the data to meaningful information, and finally presents the data to clients. Our clients have identified errors in our data (the presentation layer of our data), causing them to question other data we present to them. We’ve been able to identify that many of these errors come from sources that tend to use manual entry in their data flows, but we want to prevent these errors from ever making it into the presentation layer where our clients see the data.
Solution
We want to catch possible invalid data points as early as possible in our ETL flow, as clients will start to doubt our data if they see errors in the final data. If our data flow has 3 nodes, where the first node is the import node, the second node is the transformation node, and the third node is the presentation node, we may not identify some errors until the final node. However, it’s possible that we can catch these errors earlier in our ETL flow.
In this tip, we’ll look at some of the considerations for both data rules and logic to handle these rules.
HOPE IT HELPS
LLZ MRK AS BRAINLIST
Problem
We receive data from thousands of sources, some of which provide direct access and some of which manually enter their data into their system or our system. We run a multi-server and database ETL flow that obtains the original data, transforms the data to meaningful information, and finally presents the data to clients. Our clients have identified errors in our data (the presentation layer of our data), causing them to question other data we present to them. We’ve been able to identify that many of these errors come from sources that tend to use manual entry in their data flows, but we want to prevent these errors from ever making it into the presentation layer where our clients see the data.
Solution
We want to catch possible invalid data points as early as possible in our ETL flow, as clients will start to doubt our data if they see errors in the final data. If our data flow has 3 nodes, where the first node is the import node, the second node is the transformation node, and the third node is the presentation node, we may not identify some errors until the final node. However, it’s possible that we can catch these errors earlier in our ETL flow.
In this tip, we’ll look at some of the considerations for both data rules and logic to handle these rules.
HOPE IT HELPS
LLZ MRK AS BRAINLIST
Similar questions