随机数生成

  随机数生成在随机算法、性能测试中非常有用，spark.mllib支持生成随机的RDD,RDD的独立同分布(iid)的值来自于给定的分布：均匀分布、标准正太分布、泊松分布。

  RandomRDDs提供工厂方法生成随机的双精度RDD或者向量RDD。下面的例子生成了一个随机的双精度RDD，它的值来自于标准的正太分布N(0,1)。

  1. import org.apache.spark.SparkContext
  2. import org.apache.spark.mllib.random.RandomRDDs._
  3. val sc: SparkContext = ...
  4. // Generate a random double RDD that contains 1 million i.i.d. values drawn from the
  5. // standard normal distribution `N(0, 1)`, evenly distributed in 10 partitions.
  6. val u = normalRDD(sc, 1000000L, 10)
  7. // Apply a transform to get a random double RDD following `N(1, 4)`.
  8. val v = u.map(x => 1.0 + 2.0 * x)

  normalRDD的实现如下面代码所示。

  1. def normalRDD(
  2.       sc: SparkContext,
  3.       size: Long,
  4.       numPartitions: Int = 0,
  5.       seed: Long = Utils.random.nextLong()): RDD[Double] = {
  6.     val normal = new StandardNormalGenerator()
  7.     randomRDD(sc, normal, size, numPartitionsOrDefault(sc, numPartitions), seed)
  8.   }
  9. def randomRDD[T: ClassTag](
  10.       sc: SparkContext,
  11.       generator: RandomDataGenerator[T],
  12.       size: Long,
  13.       numPartitions: Int = 0,
  14.       seed: Long = Utils.random.nextLong()): RDD[T] = {
  15.     new RandomRDD[T](sc, size, numPartitionsOrDefault(sc, numPartitions), generator, seed)
  16.   }
  17. private[mllib] class RandomRDD[T: ClassTag](sc: SparkContext,
  18.     size: Long,
  19.     numPartitions: Int,
  20.     @transient private val rng: RandomDataGenerator[T],
  21.     @transient private val seed: Long = Utils.random.nextLong) extends RDD[T](sc, Nil)