量化自我和极简主义的窝藏点
环境: elaticSearch 1.6, elasticsearch-sql-1.4.9
下载elasticsearch-sql-1.4.9到相应目录下，离线安装插件。
bin/plugin –url file:///home/adadmin/elasticsearch-1.6.1/plugins/elasticsearch-sql-1.4.9.zip –installl elasticsearch-sql
重启ElasticSearch服务才可以使用
http://192.168.56.101:9200/_plugin/sql/
环境：spark 1.6， 存储是一个http get的服务
在build.sbt中添加”org.apache.httpcomponents” % “httpclient” % “4.5.2” ，记得第一个分隔符是%，而不是%%。
经过多次尝试，最终代码如下：
agg_wd_business.foreach(d => {
val httpParams = new BasicHttpParams()
HttpConnectionParams.setConnectionTimeout(httpParams, 50)
HttpConnectionParams.setSoTimeout(httpParams, 50)
val client = new DefaultHttpClient(httpParams)
val request = new HttpGet(“http://xxx.xxx.xxx.xxx:9010/rt?” + URLEncoder.encode(d, “UTF-8”))
request.addHeader(“Connection”, “close”)
try{
val response = client.execute(request)
val handler = new BasicResponseHandler()
handler.handleResponse(response).trim.toString
}catch{
case ex: SocketTimeoutException => None
case ex: Exception => None
}
})
发送一个http get请求，设置超时，设置为短连接，并不保证请求一定成功。由于生成的数据有30万左右，得调用http get这么次，而nginx搭配的服务并不能快速地响应。
之前解决问题的时候忘了做笔记，后来遇到相同的问题的时候，不记得之前有个解决方案。年纪大了好多东西都得记得做，有时候忘了这忘了呢，以后做这种技术的话最好还是自己做一下笔记，用wordpress
环境: sbt， scala 2.10.4
1.
vi project/scalapb.sbt
addSbtPlugin(“com.thesamet” % “sbt-protoc” % “0.99.1”)
libraryDependencies += “com.trueaccord.scalapb” %% “compilerplugin” % “0.5.43”
2.
vi build.sbt
PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
// If you need scalapb/scalapb.proto or anything from google/protobuf/*.proto
//ScalaPB looks for protocol buffer files in src/main/protobuf, but this can be customized. Running the compile command in sbt will both generate Scala sources from your protos and compile them.
libraryDependencies += “com.trueaccord.scalapb” %% “scalapb-runtime” % com.trueaccord.scalapb.compiler.Version.scalapbVersion % “protobuf”
3.
mkdir src/main/protobuf
vi src/main/protobuf/hello.proto
syntax = “proto3”;
package example;
message HelloRequest {
string name = 1;
}
sbt assembly
生成的scala文件放在 target/scala-2.10/src_managed/main/example/hello/HelloRequest.scala
4.
使用如下
import hello._
val h = HelloRequest().withName(“hq”)
val hba = h.toByteArray
println(hba) //serialize
println(HelloRequest.parseFrom(hba).name) //unserialize
1. 通过proto文件生成java文件夹
vi test1.proto
syntax = “proto2”;
package example;
message Hello{
required string name = 1;
required int32 id = 2;
}
生成Test1.java
protoc –java_out=pbdir test1.proto
2. 将Test1.java拷贝到src/main/java/example目录下
3. 通过spark streaming读取kafka上的pb数据
import Test1._
createKafkaStream(ssc, pb_topic, kafkaParams1).map(r => r._2).map(r => {val p = Hello.parseFrom(r.getBytes); p.getId + “\\t” + p.getName})
在build.sbt中添加”org.apache.httpcomponents” %% “httpclient” % “4.5.2”
编译的时候出现报错：
sbt.ResolveException: unresolved dependency: org.apache.httpcomponents#httpclient_2.10;4.5.2: not found
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.httpcomponents#httpclient_2.10;4.5.2: not found
在stackoverflow找到一个解决方法
Change the first %% to a single %. The double character version is for fetching cross-built libraries, and yours isn’t.
去掉一个%，修改如下：
“org.apache.httpcomponents” % “httpclient” % “4.5.2”