Spark DataFrame - Select n random rows

lte__ :

I have a dataframe with multiple thousands of records, and I'd like to randomly select 1000 rows into another dataframe for demoing. How can I do this in Java?

Thank you!

T. Gawęda :

You can try sample () method. Unfourtunatelly you must give there not a number, but fraction. You can write function like this:

def getRandom (dataset : Dataset[_], n : Int) = {
    val count = dataset.count();
    val howManyTake = if (count > n) n else count;
    dataset.sample(0, 1.0*howManyTake/count).limit (n)
}

Explanation: we must take a fraction of data. If we have 2000 rows and you want to get 100 rows, we must have 0.5 of total rows. If you want to get more rows than there are in DataFrame, you must get 1.0. limit () function is invoked to make sure that rounding is ok and you didn't get more rows than you specified.

Edit: I see in other answer the takeSample method. But remember:

1. It'a a method of RDD, not Dataset, so you must do: dataset.rdd.takeSample(0, 1000, System.currentTimeMilis()).toDF() takeSample will collect all values.
2. Remember that if you want to get very many rows then you will have problems with OutOfMemoryError as takeSample is collecting results in driver. Use it carefully

Guess you like

Origin http://10.200.1.11:23101/article/api/json?id=448104&siteId=1

Spark DataFrame - Select n random rows

pandas select rows by condition for all of dataframe columns

Why select after a join raises an exception in java spark dataframe?

Duplicating n rows in a DataFrame using 2nd level index?

[Python Fennel Bean Series] How does PANDAS select DataFrame rows based on column values

choosing rows by values in DataFrame

insert dataframe into rows for each group in another dataframe

Convert dataframe of list in columns to rows

Update dataframe rows trough loop

How to add rows to Pandas DataFrame

Spark DataFrame in comparison with Pandas

Spark of RDD and DataFrame

spark notes of DataFrame

Programming of 10 spark DataFrame

spark - DataFrame Api exercise

spark - RDD to DataFrame

pandas gets the number of rows, columns, and elements of the dataframe

propagate calculation along pandas dataframe rows

drop rows in pandas dataframe that are not meeting the conditions

Merge multiple rows into one in a pandas dataframe?

DataFrame deletes all rows or columns with a certain value

Spark dataframe in comparison with Pandas in dataframe / conversion

Conversion between DataFrame in Spark and DataFrame in Pandas

SELECT - Remove several rows from a table or view

Combining Multiple Rows in MYSQL SELECT query

Mysql - select rows which have linked values

Select rows of a matrix using binary array

Select oldest timestamp in MySQL when grouping rows

pandas select rows based on multiple datetime columns

MySQL SELECT fist X rows without an index