ScalaPB生成scala的protobuf文件

环境: sbt, scala 2.10.4

 

1.

vi project/scalapb.sbt

addSbtPlugin(“com.thesamet” % “sbt-protoc” % “0.99.1”)

libraryDependencies += “com.trueaccord.scalapb” %% “compilerplugin” % “0.5.43”

 

2.

vi build.sbt

PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)

// If you need scalapb/scalapb.proto or anything from google/protobuf/*.proto
//ScalaPB looks for protocol buffer files in src/main/protobuf, but this can be customized. Running the compile command in sbt will both generate Scala sources from your protos and compile them.

libraryDependencies += “com.trueaccord.scalapb” %% “scalapb-runtime” % com.trueaccord.scalapb.compiler.Version.scalapbVersion % “protobuf”

 

3.
mkdir src/main/protobuf

vi src/main/protobuf/hello.proto

syntax = “proto3”;
package example;

message HelloRequest {
string name = 1;
}

sbt assembly
生成的scala文件放在 target/scala-2.10/src_managed/main/example/hello/HelloRequest.scala

4.

使用如下
import hello._

val h = HelloRequest().withName(“hq”)
val hba = h.toByteArray
println(hba) //serialize
println(HelloRequest.parseFrom(hba).name) //unserialize

spark streaming读取kafka上的protobuf格式的数据

1. 通过proto文件生成java文件夹

vi test1.proto

syntax = “proto2”;
package example;

message Hello{
required string name = 1;
required int32 id = 2;
}

生成Test1.java
protoc –java_out=pbdir test1.proto

 

2. 将Test1.java拷贝到src/main/java/example目录下

 

3. 通过spark streaming读取kafka上的pb数据
import Test1._

createKafkaStream(ssc, pb_topic, kafkaParams1).map(r => r._2).map(r => {val p = Hello.parseFrom(r.getBytes); p.getId + “\\t” + p.getName})