PySpark Review 박영택.

Presentation on theme: "PySpark Review 박영택."— Presentation transcript:

1 PySpark Review 박영택

2 RDD (Resilient Distributed Dataset)

3 RDD 생성 RDD를 생성하는 첫번째 방법 예시 파일, 파일의 집합을 통해 생성 sc.textFile(“myfile.txt”)
sc.textFile(“mydata/*.log”) sc.textFile(“myfile1.txt, myfile2.txt”) 파일, 파일의 집합을 통해 생성 Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt RDD: newRDD Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven > newRDD = sc.textFile(“test.txt”) > newRDD.count() 4

4 RDD 생성 RDD를 생성하는 두번째 방법 메모리에 있는 데이터를 통해 생성 > num = [1,2,3,4]
list: num > num = [1,2,3,4] > rdd = sc.parallelize(num) RDD: rdd 1 2 3 4

5 RDD 생성 RDD를 생성하는 세번째 방법 RDD를 통해 생성
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt > newRDD = sc.textFile(“test.txt") > newRDD_uc = newRDD.map(lambda line: \ line.upper()) > newRDD_uc.count() 4 RDD: newRDD Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven RDD: newRDD_uc Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven

6 RDD Operations RDD 함수의 종류 Actions : 값을 리턴
Transformations : 현재의 것 에 기초하여 새로운 RDD를 정의한다.

7 RDD 함수: Transformations (1)
Transformation 함수는 이미 존재하는 RDD를 통해 새로운 RDD를 생성 RDD는 불변(immutable) RDD에 있는 데이터는 절대 바꿀수 없다. 필요에 따라 데이터를 수정하는 시퀀스를 변환한다.

8 RDD 함수: Transformations (2)
map(function) : 주어진 RDD의 각 레코드 (라인)별로 기능을 수행하여 새로운 RDD를 생성 Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt RDD: newRDD > newRDD = sc.textFile(“test.txt") > newRDD_uc = newRDD.map(lambda line: line.upper()) > newRDD_uc.count() 4 Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven RDD: newRDD_uc TIME CAN BRING YOU DOWN TIME CAN BEND YOUR KNEES WOULD YOU KNOW MY NAME IF I SAW YOU IN HEAVEN

9 RDD 함수: Transformations (3)
filter(function) : 주어진 RDD를 라인(레 코드)별로 조건에 맞는 라인으로 새로운 RDD를 생 성 Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt RDD: newRDD Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven > newRDD = sc.textFile(“test.txt") > newRDD _ft = newRDD.filter(lambda line: \ line.startswith(’T')) > newRDD_ft.count() 2 RDD: newRDD_ft Time can bring you down Time can bend your knees

10 RDD 함수: Actions (1) 주요 Action 함수 count() : RDD의 요소의 갯수를 반환
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt > newRDD= sc.textFile(“test.txt”) > newRDD.count() 4 RDD: newRDD Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven

11 RDD 함수: Actions (2) 주요 Action 함수
take(n) : RDD의 첫번째 요소부터 n개의 요소를 리스트로 반환 Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt > newRDD= sc.textFile(“test.txt”) > newRDD.take(2) [ u’Time can bring you down’, u'Time can bend your knees' ] RDD: newRDD Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven list Time can bring you down Time can bend your knees

12 RDD 함수: Actions (3) 주요 Action 함수 collect(n) : RDD의 모든 요소를 반환
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt > newRDD = sc.textFile(“test.txt”) > newRDD .collect() [u" Time can bring you down", u' Time can bend your knees', u' Would you know my name', u" If I saw you in heaven"] RDD: newRDD Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven list Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven

13 RDD 함수: Actions (4) Some common actions
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt Some common actions saveAsTextFile(path) : RDD를 파일로 저장 RDD: newRDD > newRDD= sc.textFile("test.txt") > newRDD_ft = newRDD.filter(lambda line: \ line.startswith(’T')) > newRDD_ft.saveAsTextFile(“output”) Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven RDD: newRDD_ft Time can bring you down Time can bend your knees Time can bring you down Time can bend your knees File: part-0000

14 Lazy Execution (1) RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않 음 >
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않 음 >

15 Lazy Execution (2) RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않음
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않음 RDD: newRDD > newRDD = sc.textFile(”text.txt")

16 Lazy Execution (3) RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않 음
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않 음 RDD: newRDD > newRDD = sc.textFile(”test.txt") > newRDD_uc = newRDD.map(lambda line: line.upper()) RDD: newRDD_uc

17 Lazy Execution (4) RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않 음
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않 음 RDD: newRDD > newRDD = sc.textFile(”test.txt") > newRDD_uc = newRDD.map(lambda line: line.upper()) > newRDD_filt = \ newRDD_uc.filter(lambda line: \ line.startswith(’T')) RDD: newRDD_uc RDD: newRDD_filit

18 Lazy Execution (5) RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않 음
Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt RDD의 데이터는 action 함수로 인한 작업이 수행 될 때까지, 처리되지 않 음 RDD: newRDD > newRDD = sc.textFile(”test.txt") > newRDD_uc = newRDD.map(lambda line: line.upper()) > newRDD_filt = \ newRDD_uc.filter(lambda line: \ line.startswith(’T')) > newRDD_filt.count() 2 Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven RDD: newRDD_uc TIME CAN BRING YOU DOWN TIME CAN BEND YOUR KNEES WOULD YOU KNOW MY NAME IF I SAW YOU IN HEAVEN RDD: newRDD_filit Time can bring you down Time can bend your knees

19 Chaining Transformations
> newRDD = sc.textFile("test.txt") > newRDD_uc = newRDD.map(lambda line: line.upper()) > newRDD_filt = newRDD_uc.filter(lambda line: line.startswith(’T')) > newRDD_filt.count() 2 > sc.textFile(“test.txt”).map(lambda line: line.upper()) \ .filter(lambda line: line.startswith(‘T’)).count() 2

20 Example: Passing Named Functions (1)
Anonymous Functions 식별자(함수명)없는 인라인 함수 Best for short, one-off functions Python: lamda x: … Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt > def toUpper(s) : return s.upper() > newRDD = sc.textFile(“test.txt”) > newRDD.map(toUpper).take(2) > newRDD.map(lambda line: line.upper()).take(2)

21 Example: Passing Named Functions (2)
Anonymous Functions 식별자(함수명)없는 인라인 함수 Best for short, one-off functions Python: lamda x: … Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt > def toUpper(s) : return s.upper() > newRDD = sc.textFile(“test.txt”) > newRDD.map(toUpper).take(2) > newRDD.map(lambda line: line.upper()).take(2)

22 Some Other General RDD Operations (1)
Transformations flatMap(function) : base RDD의 각 라인별 엘리먼트를 각 엘리먼 트 단위로 매핑 > sc.textFile(“test.txt”) \ .map(lambda line: line.split()) \ [ [“Time”, “can”, “bring”, “you”, “down”] [”Time”, “can”, “bend”, “your”, “knees”] [“Would”, “you”, “know”, “my”, “name”] [“If”, “I”, “saw”, “you”, “in”, “heaven”] ] Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt

23 Some Other General RDD Operations (1)
Transformations flatMap(function) : base RDD의 각 라인별 엘리먼트를 각 엘리먼 트 단위로 매핑 > sc.textFile(“test.txt”) \ .flatMap(lambda line: line.split()) [ [“Time”, “can”, “bring”, “you”, “down”] [”Time”, “can”, “bend”, “your”, “knees”] [“Would”, “you”, “know”, “my”, “name”] [“If”, “I”, “saw”, “you”, “in”, “heaven”] ] Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt [“Time”, “can”, “bring”, “you”, “down”, ”Time”, “can”, “bend”, “your”, “knees”, “Would”, “you”, “know”, “my”, “name”, “If”, “I”, “saw”, “you”, “in”, “heaven”]

24 Some Other General RDD Operations (2)
Transformations distinct: 중복제거 > sc.textFile(“test.txt”) \ .flatMap(lambda line: line.split()) .distinct() Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven File: test.txt [“Time”, “can”, “bring”, “you”, “down”, ”Time”, “can”, “bend”, “your”, “knees”, “Would”, “you”, “know”, “my”, “name”, “If”, “I”, “saw”, “you”, “in”, “heaven”] [“Time”, “can”, “bring”, “you”, “down”, “bend”, “your”, “knees”, “Would”, “know”, “my”, “name”, “If”, “I”, “saw”, “in”, “heaven”]

25 Some Other General RDD Operations (4)
Other RDD operations first: RDD의 첫 번째 엘리먼트를 리턴 > newRDD.first() “Time can bring you down” RDD: newRDD Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven

26 Some Other General RDD Operations (5)
Other RDD operations top(n): 현재 RDD 상태에서 가장 큰 n개의 엘리먼트를 리턴 > newRDD.top(2) “Time can bring you down” “Time can bend your knees” RDD: newRDD Time can bring you down Time can bend your knees Would you know my name If I saw you in heaven