FarmMaps/Documentatie
13
0
Fork 0
Code Pull Requests Releases Activity

updated workflow and component pages, restructure index

Browse Source
This commit is contained in:
auke
2020-04-08 18:49:50 +02:00
parent 208e776dd0
commit 35ced21791
5 changed files with 153 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
Workflow.md Normal file
View File

@@ -0,0 +1,43 @@
# FarmMaps Workflow
This page explains how FarmMaps processes incoming data so that it can be used in the FarmMaps application, and data becomes easily retrievable through the interface.
Depending on the type of data, the system might process the data differently, how this works is explained below.
## General workflow
As mentioned earlier, the REST API is the primary API endpoint.
The workflow starts with registering a new upload, and uploading the file in chunks.
Once a file has been uploaded succesfully through the REST API, FarmMaps will automatically start to process the file step by step.
In the processing workflow, there are 3 common, not dependent on the type of file.
After the common steps, the workflow starts tasks for the specific type of file.
So, when a file has been uploaded completely, FarmMaps will:
* Start a stitching task to reasemble the chunks back into a single file.
* Start an identification task to identify the type of file, and identify what the next processing step should be.
* Start the file-type specific workflow, starting with the first task defined in the file-type specific workflow.
Each time a task is started, a separate process is created called a worker.
This worker then executes the task and reports back to the main FarmMaps workflow once it has completed.
The main workflow will then start the next task untill all processing steps defined for that specific type of file are completed.
Once the primary processing is completed, the data is available for viewing through the FarmMaps GUI.
Also, other tasks can be started that use the pre-processed data.
These are tasks such as generating a task map, or creating derived datasets from the pre-processed data.
## Stitching task
Because files are uploaded in chunks in FarmMaps, these chunks need to re-assembled into the original file.
This task is started automatically.
## Identification task
Once a file is re-assembled, FarmMaps will try to identify the file.
It does so by looking at the file extension (i.e. `.csv`, `.shp`, `.tif`), the binary structure, and the content structure.
Once it recognizes the type of file that was uploaded, FarmMaps stores this information as metadata in the database, with a reference to the file.
The system can then look up the correct file specific workflow, and related tasks.
### File specific task
Once the file has been identified and the correct file-type specific workflow is known, FarmMaps will start the with the first task listed in that workflow.
It uses the resulting file of the stitching task, and depending on the type of task generally creates a new file with the converted data.
A good example would be a `.csv` file that was uploaded, that would be converted to a `.shp` or shapefile.
Once the data conversion has completed, the database is updated to point to the new file instead of the original `.csv`.
The system does however keep track of the previous version and also keeps this in store, so the same data can easily be retrieved in different formats or versions.
When the next file-specific task starts, this step repeats and another "version" of the same data is created.