8 集合常用方法和函数操作
foreach
foreach 方法的原型：
// f 返回的类型是Unit， foreach 返回的类型是Unit
def foreach[U](f: Elem => U)该方法接受一个函数 f 作为参数， 函数 f 的类型为Elem => U，即 f 接受一个参数，参数的类型为容器元素的类型Elem，f 返回结果类型为 U。foreach 遍历集合的每个元素，并将f 应用到每个元素上。
res176: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7)
scala> arr.foreach(t=>{
| if(t%2==0)
| println(t)
| else
| println(t*2)
| })
2
2
6
4
10
6
scala> val arr = Array(Array(11,22,33,44),Array(1,2,3,4))
arr: Array[Array[Int]] = Array(Array(11, 22, 33, 44), Array(1, 2, 3, 4))
scala> arr.foreach(arr=> arr.foreach(t=>{
| if(t%2==0)
| println(t)
| else
| println(t*2)
| }))map
map 操作
map操作是针对集合的典型变换操作，它将某个函数应用到集合中的每个元素，并产生一个结果集合；
map方法返回一个与原集合类型大小都相同的新集合，只不过元素的类型可能不同。
map带有返回值，集合类型和元类型一样，元素的个数不会发生变化，元素本身会变化
scala> arr
res193: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.map(t=>t*2)
res194: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> arr.foreach(t=>println(t*2))
2
4
6
8
10
12
14
16
18
scala> arr.map(t=> if(t<5) t+1 else t )
res196: Array[Int] = Array(2, 3, 4, 5, 5, 6, 7, 8, 9)
scala> arr.map(t=> if(t<5) t+1 else t ).map(t=> if(t%2==0)t else t*2)
res197: Array[Int] = Array(2, 6, 4, 10, 10, 6, 14, 8, 18)
scala> arr
res198: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.map(t=> if(t%2==0) (t,"even") else (t,"odd"))
res199: Array[(Int, String)] = Array((1,odd), (2,even), (3,odd), (4,even), (5,odd), (6,even), (7,odd), (8,even), (9,odd))
//多重集合的基础使用
//最高成绩
scala> val arr = Array(("zhangsan",Array(100,120,50,45,34)),("zhaosi",Array(90,98,34,32,13)))
arr: Array[(String, Array[Int])] = Array((zhangsan,Array(100, 120, 50, 45, 34)), (zhaosi,Array(90, 98, 34, 32, 13)))
scala> arr.map(tp=>(tp._1,tp._2.max))
res200: Array[(String, Int)] = Array((zhangsan,120), (zhaosi,98))
//多层嵌套array的处理
//每个人的及格成绩有几个
scala> arr
res201: Array[(String, Array[Int])] = Array((zhangsan,Array(100, 120, 50, 45, 34)), (zhaosi,Array(90, 98, 34, 32, 13)))
scala> arr.map(tp=>(tp._1,tp._2.map(t=>if(t>=60) 1 else 0).sum))
res202: Array[(String, Int)] = Array((zhangsan,2), (zhaosi,2))
//平均成绩
scala> arr
res203: Array[(String, Array[Int])] = Array((zhangsan,Array(100, 120, 50, 45, 34)), (zhaosi,Array(90, 98, 34, 32, 13)))
scala> arr.map(tp=> (tp._1,tp._2.sum * 1.0 / tp._2.size))
res204: Array[(String, Double)] = Array((zhangsan,69.8), (zhaosi,53.4))flatten
当有一个集合的集合，然后你想对这些集合的所有元素进行操作时，就会用到 flatten；
List(List(1,2), List(3,4)) —–> List(1,2,3,4)
List(Array(1,2),Array(3,4)) —–> List(1,2,3,4)
List(Map(“a”->1,“b”->2), Map(“c”->3,“d”->4)) —–> List((a,1), (b,2), (c,3), (d,4))
scala> val arr = Array(List(1,2,3),List(4,5,6))
arr: Array[List[Int]] = Array(List(1, 2, 3), List(4, 5, 6))
scala> arr.flatten
res4: Array[Int] = Array(1, 2, 3, 4, 5, 6)
scala> val arr1 = Array(1,2,3,4,5,6,Array(7,8,9))
arr1: Array[Any] = Array(1, 2, 3, 4, 5, 6, Array(7, 8, 9))
scala> arr1.flatten
<console>:13: error: No implicit view available from Any => Traversable[U].
arr1.flatten
^
scala> val set = Set(1,2,3)
set: scala.collection.immutable.Set[Int] = Set(1, 2, 3)
scala> val set1 = Set(set,set,set)
set1: scala.collection.immutable.Set[scala.collection.immutable.Set[Int]] = Set(Set(1, 2, 3))
scala> val set1 = Set(Set(1,2,3),Set(4,5,6))
set1: scala.collection.immutable.Set[scala.collection.immutable.Set[Int]] = Set(Set(1, 2, 3), Set(4, 5, 6))
scala> set1.flatten
res6: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 3, 4)
scala> ((1,2),(3,4))
res7: ((Int, Int), (Int, Int)) = ((1,2),(3,4))
scala> res7.flatten
<console>:13: error: value flatten is not a member of ((Int, Int), (Int, Int))
res7.flatten
^
scala> val arr1 =Array(Array(arr,arr))
arr1: Array[Array[Array[List[Int]]]] = Array(Array(Array(List(1, 2, 3), List(4, 5, 6)), Array(List(1, 2, 3), List(4, 5, 6))))
scala> arr1.flatten
res9: Array[Array[List[Int]]] = Array(Array(List(1, 2, 3), List(4, 5, 6)), Array(List(1, 2, 3), List(4, 5, 6)))
scala> arr1.flatten.flatten
res10: Array[List[Int]] = Array(List(1, 2, 3), List(4, 5, 6), List(1, 2, 3), List(4, 5, 6))
scala> arr1.flatten.flatten.flatten
res11: Array[Int] = Array(1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6)注意：flatten 不支持元组
// 下面的方法报错
val list = List((1,2), (3,4))
list.flattenflatMap 操作
flatMap的执行过程： map –> flatten
scala> val arr = Array(1,2,3,4)
arr: Array[Int] = Array(1, 2, 3, 4)
scala> val arr1 = Array(arr,arr)
arr1: Array[Array[Int]] = Array(Array(1, 2, 3, 4), Array(1, 2, 3, 4))
scala> arr1.flatten
res14: Array[Int] = Array(1, 2, 3, 4, 1, 2, 3, 4)
scala> arr1.flatten.map(t=>t*2)
res15: Array[Int] = Array(2, 4, 6, 8, 2, 4, 6, 8)
scala> val arr = Array("a b c d e","a d e f g")
arr: Array[String] = Array(a b c d e, a d e f g)
scala> arr.map(t=> t.split(" "))
res16: Array[Array[String]] = Array(Array(a, b, c, d, e), Array(a, d, e, f, g))
scala> res16.flatten
res17: Array[String] = Array(a, b, c, d, e, a, d, e, f, g)
scala> val arr = Array(("a","b"),("c","d"))
arr: Array[(String, String)] = Array((a,b), (c,d))
scala> arr.map(t=>Array(t._1,t._2))
res18: Array[Array[String]] = Array(Array(a, b), Array(c, d))
scala> res18.flatten
res20: Array[String] = Array(a, b, c, d)
scala> arr
res21: Array[(String, String)] = Array((a,b), (c,d))
//加入flatMap
scala> arr.map(t=> Array(t._1,t._2)).flatten
res22: Array[String] = Array(a, b, c, d)
scala> arr.flatMap(t=>Array(t._1,t._2))
res23: Array[String] = Array(a, b, c, d)
scala> val arr = Array("a b c d e","a d e f g")
arr: Array[String] = Array(a b c d e, a d e f g)
scala> arr.flatMap(t=> t.split(" "))
res24: Array[String] = Array(a, b, c, d, e, a, d, e, f, g)注意：同flatten一样，不支持元组
练习题:
val arr = Array("zhangsan 60 90 100 120","zhaosi 100 90 98 76 59")
//想要得到的结果
zhangsan,60
zhangsan,90
zhaosi,100
zhaosi,90 ....
//思路：首先是堆叠问题要flatten 第一个元素和后面的所有元素做拼接，head tail
scala> val arr = Array("zhangsan 60 90 100 120","zhaosi 100 90 98 76 59")
arr: Array[String] = Array(zhangsan 60 90 100 120, zhaosi 100 90 98 76 59)
scala> arr.map(t=> t.split(" "))
res25: Array[Array[String]] = Array(Array(zhangsan, 60, 90, 100, 120), Array(zhaosi, 100, 90, 98, 76, 59))
scala> res25.map(arr=> arr.tail.map(t=>(arr.head,t)))
res26: Array[Array[(String, String)]] = Array(Array((zhangsan,60), (zhangsan,90), (zhangsan,100), (zhangsan,120)), Array((zhaosi,100), (zhaosi,90), (zhaosi,98), (zhaosi,76), (zhaosi,59)))
scala> res26.flatten
res27: Array[(String, String)] = Array((zhangsan,60), (zhangsan,90), (zhangsan,100), (zhangsan,120), (zhaosi,100), (zhaosi,90), (zhaosi,98), (zhaosi,76), (zhaosi,59))
scala> arr.map(t=> t.split(" ")).flatMap(arr=> arr.tail.map(t=> (arr.head,t)))
res29: Array[(String, String)] = Array((zhangsan,60), (zhangsan,90), (zhangsan,100), (zhangsan,120), (zhaosi,100), (zhaosi,90), (zhaosi,98), (zhaosi,76), (zhaosi,59))filter
遍历一个集合并从中获取满足指定条件的元素组成一个新的集合；
元素不会发生变化，集合也不会发生变化
scala> val arr1 = Array(1,2,3,4,5,6,7,8,9)
arr1: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr1.filter
filter filterNot
scala> arr1.filter
def filter(p: Int => Boolean): Array[Int]
scala> arr1.filter(t=> if(t>5) true else false)
res205: Array[Int] = Array(6, 7, 8, 9)
scala> arr1.filter(t=> t>5)
res206: Array[Int] = Array(6, 7, 8, 9)高级使用
scala> arr
res207: Array[(String, Array[Int])] = Array((zhangsan,Array(100, 120, 50, 45, 34)), (zhaosi,Array(90, 98, 34, 32, 13)))
scala> arr.map(t=> (t._1,t._2.filter(t=> t>=60).size))
res208: Array[(String, Int)] = Array((zhangsan,2), (zhaosi,2))filterNot
scala> arr
res207: Array[(String, Array[Int])] = Array((zhangsan,Array(100, 120, 50, 45, 34)), (zhaosi,Array(90, 98, 34, 32, 13)))
scala> arr1
res209: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr1.filter(t=> t%2==0)
res210: Array[Int] = Array(2, 4, 6, 8)
scala> arr1.filterNot(t=> t%2==0)
res211: Array[Int] = Array(1, 3, 5, 7, 9)
scala> arr.map(t=> (t._1,t._2.filterNot(t=> t<60).size))
res212: Array[(String, Int)] = Array((zhangsan,2), (zhaosi,2))sorted、sortBy、sortWith
sorted：按照元素自身进行排序；
sortBy: 按照应用函数 f 之后产生的元素进行排序；
sortWith:传入ab两个值，然后返回比较规则比如a>b
val arr = Array(1,9,2,8,3,7,4,6,5)
arr: Array[Int] = Array(1, 9, 2, 8, 3, 7, 4, 6, 5)
scala> arr.sort
sortBy sortWith sorted
scala> arr.sorted
res30: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.sorted.reverse
res31: Array[Int] = Array(9, 8, 7, 6, 5, 4, 3, 2, 1)
scala> val arr = Array(Array(1,2,3),Array(3,4,5,6,7,8))
arr: Array[Array[Int]] = Array(Array(1, 2, 3), Array(3, 4, 5, 6, 7, 8))
scala> arr.sorted
<console>:13: error: No implicit Ordering defined for Array[Int].
arr.sorted
^
scala>
scala> val arr1 = Array(1,9,2,8,3,7,4,6,5)
arr1: Array[Int] = Array(1, 9, 2, 8, 3, 7, 4, 6, 5)
scala> arr1.sortBy(t=> t)
res33: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr1.sortBy(t=> -t)
res34: Array[Int] = Array(9, 8, 7, 6, 5, 4, 3, 2, 1)
scala>
scala> val arr2 = Array(("zhangsan",28000),("lisi",25000),("wangwu",26000))
arr2: Array[(String, Int)] = Array((zhangsan,28000), (lisi,25000), (wangwu,26000))
scala> arr2.sorted
res35: Array[(String, Int)] = Array((lisi,25000), (wangwu,26000), (zhangsan,28000))
scala> val arr2 = Array(("zhangsan",28000),("lisi",25000),("wangwu",26000),("azhen",30000))
arr2: Array[(String, Int)] = Array((zhangsan,28000), (lisi,25000), (wangwu,26000), (azhen,30000))
scala> arr2.sorted
res36: Array[(String, Int)] = Array((azhen,30000), (lisi,25000), (wangwu,26000), (zhangsan,28000))
scala> arr2.sortBy(t=> t._2)
res37: Array[(String, Int)] = Array((lisi,25000), (wangwu,26000), (zhangsan,28000), (azhen,30000))
scala> arr2.sortBy(t=> -t._2)
res38: Array[(String, Int)] = Array((azhen,30000), (zhangsan,28000), (wangwu,26000), (lisi,25000))
scala>
scala> arr1
res39: Array[Int] = Array(1, 9, 2, 8, 3, 7, 4, 6, 5)
scala> arr
res40: Array[Array[Int]] = Array(Array(1, 2, 3), Array(3, 4, 5, 6, 7, 8))
scala> arr.sortBy(t=> t.sum/t.size)
res41: Array[Array[Int]] = Array(Array(1, 2, 3), Array(3, 4, 5, 6, 7, 8))
scala> arr.sortBy(t=> -t.sum/t.size)
res42: Array[Array[Int]] = Array(Array(3, 4, 5, 6, 7, 8), Array(1, 2, 3))
scala>
scala> arr.map(t=> t.sortBy(e=> -e))
res43: Array[Array[Int]] = Array(Array(3, 2, 1), Array(8, 7, 6, 5, 4, 3))
scala>
scala> arr1
res44: Array[Int] = Array(1, 9, 2, 8, 3, 7, 4, 6, 5)
scala> arr1.sortWith((a,b)=> a<b)
res45: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr1.sortWith((a,b)=> a>b)
res46: Array[Int] = Array(9, 8, 7, 6, 5, 4, 3, 2, 1)
scala> arr
res47: Array[Array[Int]] = Array(Array(1, 2, 3), Array(3, 4, 5, 6, 7, 8))
scala> arr.sortWith((a,b)=> a.sum/a.length > b.sum/b.length)
res48: Array[Array[Int]] = Array(Array(3, 4, 5, 6, 7, 8), Array(1, 2, 3))
scala> arr2
res49: Array[(String, Int)] = Array((zhangsan,28000), (lisi,25000), (wangwu,26000), (azhen,30000))
scala> arr2.sortWith((a,b)=> a._2 > b._2)
res50: Array[(String, Int)] = Array((azhen,30000), (zhangsan,28000), (wangwu,26000), (lisi,25000))并行集合
通过list.par 会将集合变成并行集合，可以利用多线程来进行运算。
def main(args: Array[String]): Unit = {
val arr = Array(1,2,3,4,5,6,7,8,9)
arr.par.map(t=>{
val name = Thread.currentThread().getName
(name,t)
}).foreach(println)
}
reduce、reduceLeft、reduceRight
reduce：reduce(op: (A1, A1) => A1): A1 。reduce操作是按照从左到右的顺序进行规约。(((1+2)+3)+4)+5
reduceLeft：reduceLeft[B >: A](f: (B, A) => B): B。是按照从左到右的顺序进行规约。 (((1+2)+3)+4)+5
reduceRight：reduceRight[B >: A](op: (A, B) => B): B。是按照从右到左的顺序进行规约。1+(2+(3+(4+5)))
单线程下： reduce 和 reduceLeft一样
并行集合运行下： reduce利用CPU数运行， reduceLeft 有方向，只能单线程运行
reduce
scala> arr.reduce((a,b)=> a+b)
res4: Int = 45
scala> arr.reduce((a,b)=> a*b)
res5: Int = 362880
scala> arr.reduce((a,b)=> a-b)
res6: Int = -43
scala> arr.reduce((a,b)=> a*(a-b))
res7: Int = 0
scala> arr.reduce((a,b)=> if(a>b) a else b)
res8: Int = 9
scala> arr.reduce((a,b)=> if(a<b) a else b)
res9: Int = 1
scala> val arr1 = Array(("a",10),("b",20),("c",15))
arr1: Array[(String, Int)] = Array((a,10), (b,20), (c,15))
scala> arr1.reduce((a,b)=> a._2 + b._2)
<console>:13: error: type mismatch;
found : Int
required: (String, Int)
arr1.reduce((a,b)=> a._2 + b._2)
^
scala> arr1.reduce((a,b)=> ("",a._2+b._2))
res11: (String, Int) = ("",45)def main(args: Array[String]): Unit = {
val arr = Array(1,2,3,4,5,6,7,8,9)
val num = arr.par.reduce((a,b) => {
val name = Thread.currentThread().getName
println(name,a,b)
a+b
})
println(num)
}
reduceLeft和reduceRight
val arr = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
// val num1 = arr.reduceLeft((a, b) => a + b)
// val num2 = arr.reduceRight((a, b) => a + b)
// println(num1)
// println(num2)
val num = arr.par.reduce((a, b) => {
val name = Thread.currentThread().getName
println(name,a, b)
a - b
})
val num1 = arr.par.reduceLeft((a, b) => {
val name = Thread.currentThread().getName
println(name, a, b)
a - b
})
val num2 = arr.par.reduceRight((a, b) => {
val name = Thread.currentThread().getName
println(name, a, b)
a - b
})
println(num1)
println(num2)
}
类型展示
scala> arr.reduce((a,b)=> a+b)
res3: Int = 15
scala> arr.reduceLeft((a,b)=> a+b)
res4: Int = 15
scala> arr.reduceLeft((a:AnyVal,b:Int) => (a.asInstanceOf[Int]+b).asInstanceOf[AnyVal])
res5: AnyVal = 15
scala> arr.reduce((a:AnyVal,b:Int) => (a.asInstanceOf[Int]+b).asInstanceOf[AnyVal])
<console>:13: error: type mismatch;
found : (AnyVal, Int) => AnyVal
required: (AnyVal, AnyVal) => AnyVal
arr.reduce((a:AnyVal,b:Int) => (a.asInstanceOf[Int]+b).asInstanceOf[AnyVal])
^
scala> arr.reduceRight
reduceRight reduceRightOption
scala> arr.reduceRight
override def reduceRight[B >: Int](op: (Int, B) => B): B
scala> arr.reduceRight((a:Int,b:AnyVal) => (a+b.asInstanceOf[Int]).asInstanceOf[AnyVal])
res7: AnyVal = 15如何简写？
scala> val list = List(1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)
scala> list.reduce((a:Int,b:Int) => {a + b})
res18: Int = 15
scala> list.reduce((a:Int,b:Int) => {println(s"${a} + ${b} = ${a + b}");a + b})
1 + 2 = 3
3 + 3 = 6
6 + 4 = 10
10 + 5 = 15
res19: Int = 15
scala> list.sum
res20: Int = 15
// 求最大值
scala> list.max
res21: Int = 5
// 用reduce实现求最大值
scala> list.reduce((a:Int,b:Int) => {println(s"${a} vs ${b}");if(a > b) a else b})
1 vs 2
2 vs 3
3 vs 4
4 vs 5
res22: Int = 5
scala> list.reduce((a:Int,b:Int) => {a + b})
res23: Int = 15
scala> list.reduce(_ + _)
res24: Int = 15
scala> list.par.reduce(_ + _)
res25: Int = 15fold, foldLeft, foldRight
fold：fold[A1 >: A](z: A1)(op: (A1, A1) => A1): A1 。带有初始值的reduce，从一个初始值开始，从左向右将两个元素合并成一个，最终把列表合并成单一元素。((((10+1)+2)+3)+4)+5
foldLeft：foldLeft[B](z: B)(f: (B, A) => B): B 。带有初始值的reduceLeft。((((10+1)+2)+3)+4)+5
foldRight：foldRight[B](z: B)(op: (A, B) => B): B。带有初始值的reduceRight。1+(2+(3+(4+(5+10))))
fold代码
scala> arr
res8: Array[Int] = Array(1, 2, 3, 4, 5)
scala> arr.fold
fold foldLeft foldRight
scala> arr.fold
def fold[A1 >: Int](z: A1)(op: (A1, A1) => A1): A1
scala> arr.fold(0)((a,b)=> a+b)
res9: Int = 15
scala> val arr = Array(("a",10),("b",20),("c",15))
arr: Array[(String, Int)] = Array((a,10), (b,20), (c,15))
scala> arr.fold(0)((a,b)=>a+b._2)
<console>:13: error: value _2 is not a member of Any
arr.fold(0)((a,b)=>a+b._2)
^
scala> arr.fold(("",0))((a,b)=>a._2+b._2)
<console>:13: error: type mismatch;
found : Int
required: (String, Int)
arr.fold(("",0))((a,b)=>a._2+b._2)
^
scala> arr.fold(("",0))((a,b)=>("",a._2+b._2))
res12: (String, Int) = ("",45)//fold多线程
def main(args: Array[String]): Unit = {
val arr = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
val num = arr.par.fold(0)((a,b) => {
val name = Thread.currentThread().getName
println(name,a,b)
a+b
})
println(num)
}
foldLeft和foldRight
scala> val arr = Array(1,2,3,4,5,6,7,8,9)
arr: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.foldLeft(0)((a,b)=>a-b)
res14: Int = -45
scala> arr.foldRight(0)((a,b)=> a-b)
res15: Int = 5
scala> arr.foldRight(0)((a,b)=> {
| println(a,b)
| a-b
| })
(9,0)
(8,9)
(7,-1)
(6,8)
(5,-2)
(4,7)
(3,-3)
(2,6)
(1,-4)
res16: Int = 5
scala> arr.fold
def fold[A1 >: Int](z: A1)(op: (A1, A1) => A1): A1
scala> arr.foldLeft
override def foldLeft[B](z: B)(op: (B, Int) => B): B
scala> val arr = Array(("a",10),("b",20),("c",15))
arr: Array[(String, Int)] = Array((a,10), (b,20), (c,15))
scala> arr.foldLeft(0)((a,b)=>a+b._2)
res17: Int = 45
scala> arr.foldRight
override def foldRight[B](z: B)(op: ((String, Int), B) => B): B
scala> arr.foldRight(0)((a,b)=> b+a._2)
res18: Int = 45 def main(args: Array[String]): Unit = {
val arr = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
val num = arr.par.foldRight(0)((a, b) => {
val name = Thread.currentThread().getName
println(name, a, b)
a + b
})
println(num)
}
aggregate
将每个分区里面的元素进行聚合，然后用combine函数将每个分区的结果和初始值进行combine操作；
val list = List(1,2,3,4,5)
// 当集合不是并行集合时，combop函数不执行
val sum: Int = list.aggregate(0)((a: Int, b: Int) => {
println(s"step1:a:${a}, b:${b}")
a + b
},
(a: Int, b: Int) => {
println(s"step2:a:${a}, b:${b}")
a + b
}
)
println(sum)
//-------运行结果-----------------------------
step1:a:0, b:1
step1:a:1, b:2
step1:a:3, b:3
step1:a:6, b:4
step1:a:10, b:5
15
//--------------------------------------------------------------
val list = List(1,2,3,4,5)
// 当集合是并行集合时，combop函数执行
// step1:做基础聚合， step2:在step1 基础上做聚合，相当于combiner
val sum2: Int = list.par.aggregate(0)((a: Int, b: Int) => {
println(s"step1:a:${a}, b:${b}")
a + b
},
(a: Int, b: Int) => {
println(s"step2:a:${a}, b:${b}")
a + b
}
)
println(sum2)
//-------运行结果-----------------------------
15
step1:a:0, b:1
step1:a:0, b:4
step1:a:0, b:5
step1:a:0, b:2
step1:a:0, b:3
step2:a:1, b:2
step2:a:4, b:5
step2:a:3, b:9
step2:a:3, b:12
15总结：
reduce/reduceLeft/reduceRight: 认为每个元素类型一样
fold: 带有初始值的reduce，初始值类型和元素类型一样；并行集合下注意初始值的设定；
foldLeft/foldRight： 初始值类型和元素类型可以不一样，规约结果和初始值类型一致；并行集合下是单线程运算
aggregate：初始值类型和元素类型可以不一样，规约结果和初始值类型一致；并行集合下是利用CPU核数运算
groupBy、grouped
groupBy：将list 按照某个元素内的字段分组，返回map。 List((k,v),(k,v)) –> Map(k, List(k,v))
grouped：按列表按照固定的大小进行分组，返回迭代器。List(1,2,3,4,5) –> Iterator[List[A]]
scala> val arr = Array(1,2,3,4,5,6,7,8,9,10)
arr: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
scala> arr.groupBy
def groupBy[K](f: Int => K): scala.collection.immutable.Map[K,Array[Int]]
scala> arr.groupBy(t=> if(t%2 == 0) "even" else "odd")
res21: scala.collection.immutable.Map[String,Array[Int]] = Map(odd -> Array(1, 3, 5, 7, 9), even -> Array(2, 4, 6, 8, 10))
scala> arr.groupBy(t=> t%2==0)
res22: scala.collection.immutable.Map[Boolean,Array[Int]] = Map(false -> Array(1, 3, 5, 7, 9), true -> Array(2, 4, 6, 8, 10))
scala> val arr = Array("hello","hehe","world","hahah","what","kitty","kill","zoo","zookeeper")
arr: Array[String] = Array(hello, hehe, world, hahah, what, kitty, kill, zoo, zookeeper)
scala> arr.groupBy(t=> if(t.startswith("w")) "w" else ....)
<console>:1: error: illegal start of simple expression
arr.groupBy(t=> if(t.startswith("w")) "w" else ....)
^
scala> arr.groupBy(t=> t.substring(0,1))
res23: scala.collection.immutable.Map[String,Array[String]] = Map(h -> Array(hello, hehe, hahah), w -> Array(world, what), k -> Array(kitty, kill), z -> Array(zoo, zookeeper))
scala> val arr2 = Array(("zhangsan",100),("lisi",98),("wangwu",45),("zhaosi",36))
arr2: Array[(String, Int)] = Array((zhangsan,100), (lisi,98), (wangwu,45), (zhaosi,36))
scala> arr2.groupBy(t=> t._2 >= 60)
res24: scala.collection.immutable.Map[Boolean,Array[(String, Int)]] = Map(true -> Array((zhangsan,100), (lisi,98)), false -> Array((wangwu,45), (zhaosi,36)))grouped分组
scala> val arr = Array(1,2,3,4,5,6,7,8,9)
arr: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.grouped(3)
res25: Iterator[Array[Int]] = <iterator>
scala> res25.foreach(println)
[I@161c8305
[I@696f286d
[I@1aebf638
scala> res25.foreach(t=>println(t.toList))
scala> arr.grouped(3).foreach(t=> println(t.toList))
List(1, 2, 3)
List(4, 5, 6)
List(7, 8, 9)
scala> arr
res29: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.grouped(2)
res30: Iterator[Array[Int]] = <iterator>
scala> res30.map(t=> t.reverse)
res31: Iterator[Array[Int]] = <iterator>
scala> res31.flatten
res32: Iterator[Int] = <iterator>
scala> res32.mkString(" ")
res33: String = 2 1 4 3 6 5 8 7 9mapValues
对map映射里每个key的value 进行操作。
val map = Map("b" -> List(1,2,3), "a" -> List(4,5,6))
// 对每个key的value 求和
val n1 = map.mapValues(_.sum)
println(n1)group by 和 mapValues 组合
scala> arr
res34: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> //奇数和偶数的平均值
scala> arr.groupBy(t=> t%2==0)
res35: scala.collection.immutable.Map[Boolean,Array[Int]] = Map(false -> Array(1, 3, 5, 7, 9), true -> Array(2, 4, 6, 8))
scala> res35.map(t=>(t._1,t._2.sum * 1.0/t._2.size))
res36: scala.collection.immutable.Map[Boolean,Double] = Map(false -> 5.0, true -> 5.0)
scala> val arr1 = Array("hello","hello","tom","world","tom","world")
arr1: Array[String] = Array(hello, hello, tom, world, tom, world)
scala> arr1.groupBy(t=> t)
res37: scala.collection.immutable.Map[String,Array[String]] = Map(hello -> Array(hello, hello), tom -> Array(tom, tom), world -> Array(world, world))
scala> res37.map(t=> (t._1,t._2.size))
res38: scala.collection.immutable.Map[String,Int] = Map(hello -> 2, tom -> 2, world -> 2)
scala> val map = Map(("zhangsan",20000),("lisi",30000),("wangwu",18000))
map: scala.collection.immutable.Map[String,Int] = Map(zhangsan -> 20000, lisi -> 30000, wangwu -> 18000)
scala> map.map(t=>(t._1,t._2+1000))
res39: scala.collection.immutable.Map[String,Int] = Map(zhangsan -> 21000, lisi -> 31000, wangwu -> 19000)
scala> map.mapValues(t=> t+1000)
res40: scala.collection.immutable.Map[String,Int] = Map(zhangsan -> 21000, lisi -> 31000, wangwu -> 19000)
scala> arr1
res41: Array[String] = Array(hello, hello, tom, world, tom, world)
scala> arr1.groupBy(t=> t)
res42: scala.collection.immutable.Map[String,Array[String]] = Map(hello -> Array(hello, hello), tom -> Array(tom, tom), world -> Array(world, world))
scala> res42.mapValues(t=> t.size)
res43: scala.collection.immutable.Map[String,Int] = Map(hello -> 2, tom -> 2, world -> 2)
scala> //操作的数据必须是map集合
scala> //不能是List((k,v),(k,v))
scala> val list = List(("a",10),("b",20))
list: List[(String, Int)] = List((a,10), (b,20))
scala> list.mapValues(t=> t+1)
<console>:13: error: value mapValues is not a member of List[(String, Int)]
list.mapValues(t=> t+1)
^diff, union, intersect
diff : 两个集合的差集；
union : 两个集合的并集；
intersect: 两个集合的交集；
scala> val arr = Array(1,2,3,4,5)
arr: Array[Int] = Array(1, 2, 3, 4, 5)
scala> val arr2 = Array(3,4,5,6,7)
arr2: Array[Int] = Array(3, 4, 5, 6, 7)
scala> arr intersect arr2
res45: Array[Int] = Array(3, 4, 5)
scala> arr diff arr2
res46: Array[Int] = Array(1, 2)
scala> arr2 diff arr
res47: Array[Int] = Array(6, 7)
scala> arr union arr2
res48: Array[Int] = Array(1, 2, 3, 4, 5, 3, 4, 5, 6, 7)
scala> val arr = Array(("zhangsan","math"),("lisi","math"),("zhangsan","chinese"),("zhaosi","english"))
arr: Array[(String, String)] = Array((zhangsan,math), (lisi,math), (zhangsan,chinese), (zhaosi,english))
scala> val arr1 = Array("zhangsan","lisi","wangwu","zhaosi","guangkun")
arr1: Array[String] = Array(zhangsan, lisi, wangwu, zhaosi, guangkun)
scala> arr.map(t=>t._1)
res49: Array[String] = Array(zhangsan, lisi, zhangsan, zhaosi)
scala> res49 intersect arr1
res50: Array[String] = Array(zhangsan, lisi, zhaosi)
scala> arr1 diff res49
res51: Array[String] = Array(wangwu, guangkun)实现 wordcount
实现统计单词的个数
scala> val list = List("hello tom hello jack","hello tom hello world world world")
list: List[String] = List(hello tom hello jack, hello tom hello world world world)
scala> list.flatMap(t=> t.split(" "))
res52: List[String] = List(hello, tom, hello, jack, hello, tom, hello, world, world, world)
scala> res52.groupBy(t=> t)
res53: scala.collection.immutable.Map[String,List[String]] = Map(hello -> List(hello, hello, hello, hello), tom -> List(tom, tom), jack -> List(jack), world -> List(world, world, world))
scala> res53.mapValues(t=> t.size)
res54: scala.collection.immutable.Map[String,Int] = Map(hello -> 4, tom -> 2, jack -> 1, world -> 3)
scala> res54.foreach(println)
(hello,4)
(tom,2)
(jack,1)
(world,3)idea版本统计单词次数
def main(args: Array[String]): Unit = {
val it: Iterator[String] = Source.fromFile("data/word.txt").getLines() //hello tom hello world
val data1:Iterator[String] = it.flatMap(t=> t.split(" "))
val data2:Iterator[(String,Int)] = data1.map(t=>(t,1))
val list:List[(String,Int)] = data2.toList
val map:Map[String,List[(String,Int)]] = list.groupBy(t=> t._1)
val res:Map[String,Int] = map.mapValues(t=> t.size)
res.foreach(println)
}wordcount的改版
def main(args: Array[String]): Unit = {
val it: Iterator[String] = Source.fromFile("data/word.txt").getLines() //hello tom hello world
val data1: Iterator[String] = it.flatMap(t => t.split(" "))
val data2: Iterator[(String, Int)] = data1.map(t => (t, 1))
val list: List[(String, Int)] = data2.toList
val map: Map[String, List[(String, Int)]] = list.groupBy(t => t._1)
// val res:Map[String,Int] = map.mapValues(t=> t.size)
// val res = map.mapValues(t => t.map(e => e._2).sum)
// val res = map.mapValues(t => t.reduce((a, b) => ("", a._2 + b._2))._2)
// val res = map.mapValues(t => t.foldLeft(0)((a, b) => a + b._2))
val res = map.mapValues(t=> t.par.aggregate(0)((a,b)=> a+b._2,(a,b)=> a+b))
res.foreach(println)
}下划线的使用
scala> val arr = Array(1,2,3,4,5,6,7,8,9)
arr: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> val doubleValue = (x:Int) => x*2
doubleValue: Int => Int = $Lambda$1398/517536015@131936bf
scala> arr.map(doubleValue)
res56: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> arr.map((x:Int)=>x*2)
res57: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> arr.map(t => t*2)
res58: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> arr.map(_*2)
res59: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> //下划线不能单独使用
scala> arr.map(t=> t)
res60: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.map(_)
<console>:13: error: missing parameter type for expanded function ((x$1: <error>) => arr.map(x$1))
arr.map(_)
^
scala> arr.map(+_)
res62: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.map(-_)
res63: Array[Int] = Array(-1, -2, -3, -4, -5, -6, -7, -8, -9)
scala> //下划线代表同一个语义的时候不能使用两次
scala> arr
res64: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.map(t=> if(t%2==0) t else t*2)
res65: Array[Int] = Array(2, 2, 6, 4, 10, 6, 14, 8, 18)
scala> arr.map(if(_%2==0) _ else _*2)
<console>:13: error: missing parameter type for expanded function ((x$1: <error>) => x$1.$percent(2).$eq$eq(0))
arr.map(if(_%2==0) _ else _*2)
^
<console>:13: error: missing parameter type for expanded function ((x$3: <error>) => x$3.$times(2))
arr.map(if(_%2==0) _ else _*2)
^
scala> arr.reduce((a,b)=> a+b)
res67: Int = 45
scala> arr.reduce(_+_)
res68: Int = 45
scala> //如果使用下划线报错了，t=>这个方式肯定不会出错wordcount使用下划线
scala> val arr = Array(1,2,3,4,5,6,7,8,9)
arr: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> val doubleValue = (x:Int) => x*2
doubleValue: Int => Int = $Lambda$1398/517536015@131936bf
scala> arr.map(doubleValue)
res56: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> arr.map((x:Int)=>x*2)
res57: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> arr.map(t => t*2)
res58: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> arr.map(_*2)
res59: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18)
scala> //下划线不能单独使用
scala> arr.map(t=> t)
res60: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.map(_)
<console>:13: error: missing parameter type for expanded function ((x$1: <error>) => arr.map(x$1))
arr.map(_)
^
scala> arr.map(+_)
res62: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.map(-_)
res63: Array[Int] = Array(-1, -2, -3, -4, -5, -6, -7, -8, -9)
scala> //下划线代表同一个语义的时候不能使用两次
scala> arr
res64: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> arr.map(t=> if(t%2==0) t else t*2)
res65: Array[Int] = Array(2, 2, 6, 4, 10, 6, 14, 8, 18)
scala> arr.map(if(_%2==0) _ else _*2)
<console>:13: error: missing parameter type for expanded function ((x$1: <error>) => x$1.$percent(2).$eq$eq(0))
arr.map(if(_%2==0) _ else _*2)
^
<console>:13: error: missing parameter type for expanded function ((x$3: <error>) => x$3.$times(2))
arr.map(if(_%2==0) _ else _*2)
^
scala> arr.reduce((a,b)=> a+b)
res67: Int = 45
scala> arr.reduce(_+_)
res68: Int = 45
scala> //如果使用下划线报错了，t=>这个方式肯定不会出错
