com.alpine.plugin.core.spark.reporting
NullDataReportingUtils
Related Doc:
package reporting
returns a count of rows where "expression" is false, and true.
returns a count of rows where "expression" is false, and true.
- dataframe to count
- remove condition, must be possible to negate. See "filterNullDataAndReportGeneral" for limitations on the remove condition.
(goodCount, nullCount) where goodCount is the number of rows for which "removeCondition" is false and null count is the number of rows where it was true.
Given a dataFrame, the parameters and an instance of sparkRuntimeUtils, filters out all the rows containing null values.
Given a dataFrame, the parameters and an instance of sparkRuntimeUtils, filters out all the rows containing null values. Writes those rows to a file according to the values of the 'dataToWriteParam' and the 'badDataPathParam' (provided in the HdfsParameterUtils class). The method returns the data frame which does not contain nulls as well as a string containing an HTML formatted table with the information about the what data was removed and if/where it was stored. The message is generated using the 'AddendumWriter' object in the Plugin Core module.
Dirty Data: Spark SQL cannot process CSV files with dirty data (i.e. String values in numeric columns. We use the Drop Malformed option, so in the case of dirty data, the operator will not fail, but will silently remove those rows.
Same as 'filterNullDataAndReport' but rather than using the .anyNull function in the dataFrame class allows the user to define a function which returns a boolean for each row is it contains data which should be removed.
Same as 'filterNullDataAndReport' but rather than using the .anyNull function in the dataFrame class allows the user to define a function which returns a boolean for each row is it contains data which should be removed.
Dirty Data: Spark SQL cannot process CSV files with dirty data (i.e. String values in numeric columns. We use the Drop Malformed option, so in the case of dirty data, the operator will not fail, but will silently remove those rows.
could be a processed version of input data or data comming directly from operator
for writing the null data if necessary
- Either DoNotCount(), DoNotWrite() or WriteAll(path) contains information about whether and where to wrtie/ count null data
- a column condition. Data will be kept if the condition is NOT true, and removed if this condition is true. For example if the condition is col("dependent").isNull then we will remove all data that is null in the "dependent column. This expression must be nullable. It will fail if you use "col("dependent").isNull.or(col("dependent").equals(0). Because the expression !("col("dependent").isNull.or(col("dependent").equals(0)) fails on null values since both if conditions will be evaluated. Use instead !("col("dependent").isNull.or(col("dependent").eqNullSafe(0))
Return the text of the message about writing null data to file.
Return the text of the message about writing null data to file. Method does not actually filter or write.
Based on Data frames, so they should be faster Use newer language