Skip to content

Uploading Data

CSV files can be uploaded easily into Datameer to then be provided as Datameer datasets. They can be viewed and used inside a Project's Workbench.

General Considerations#

  • Any files you upload will live in your company's Datameer environment and all the other Datameer users in your organization will be able to read and use them in their own work. Please consider the confidentiality of files before uploading them to Datameer.
  • Currently only files in CSV format can be uploaded.
  • Quoted fields are uploaded without the quotation marks.
  • A CSV file with incorrect rows will skip those rows and provide a separate data set (skippedlines) which is also added to the Project and shows the errors in the Data Grid when selecting this node in the Flow Area.

Uploading Permissions#

When uploading CSV data to Datameer, the following permission has to be granted in your Snowflake instance:

  • 'CREATE_TABLE'

If not granted, CSV file uploads will result in an error.

Uploading CSV Data#

To upload a CSV file:

  1. In the Data Browser, click on the "Upload CSV" icon. The 'Upload CSV' dialog opens.

  2. Drag and drop your file or click on "Add File" and select the CSV file.

  3. If needed, rename the file. Then select the new destination from the dropdown and confirm with "Upload". The CSV is now uploaded and displayed in the Data Browser as well as in the Flow Area.

CSV Upload Limitations#

When uploading CSV data, please note the following:

Datasets that contain wrong delimiters

When your data contains wrong delimiters, such as ';', the data is converted into STRING data and the following column is left empty. Note that the file must contain at least 100 rows to being displayed in the 'Preview'.

Datasets that contain empty values

The 'Preview' shows the empty fields blank but the rest of the row is displayed as expected.

Datasets with unvalid data

  • DATE data can not be parsed and will be converted into STRING data (you can transform into a date within datameer later on)
  • if the dataset contains 95% of a file type, the rest of the data will be converted to the same type, e.g. a column contains 95% INTEGER data and 5% DATE or STRING data, the column will be converted to INTEGER
  • if a column contains 50% STRING and 50% INTEGER data, it will be converted to STRING data
  • if a column contains 50% INTEGER data and 50% DECIMAL, it will be converted to DECIMAL

Data size limitations

  • datasets that are larger than 100 MB can not be uploaded and throw an error
  • dataset files with more than 512 columns will not upload to Datameer
  • datasets containing more than 3 million rows will upload but are not processed by Datameer and will display with an error next to their names