rJava安装

环境: R 64bit,  Rstudio, JDK 64bit,  Windows 7 64bit
由于想使用rmmseg4j来用中文分词,因而必须安装rJava。但发现出现问题:
Error : .onLoad failed in loadNamespace() for 'rJava', details:
  call: inDL(x, as.logical(local), as.logical(now), ...)
  error: 无法载入共享目标对象‘D:/program/R/R-2.15.1/library/rJava/libs/x64/rJava.dll’::
  LoadLibrary failure:  %1 不是有效的 Win32 应用程序。

此外: 警告信息:
程辑包‘rJava’是用R版本2.15.3 来建造的 
错误: ‘rJava’程辑包/名字空间载入失败,


问题描述: R查找dll文件有问题

解决方法: 添加C:\\Program Files\\Java\\jdk1.6.0_10\\jre\\bin\\server环境变量path,即将jvm.dll这个库加入。 然后重启RStudio

R语言XML包readHTMLTable中文乱码

环境: Windows 7,  Ubuntu 12,  RStudio Desktop
问题: 使用安装在windows 7 上的RStudio desktop, 用包XML中的readHTMLTable读取网页上的<table>数据,例:
library(XML)
u = ‘http://tech.163.com/special/00094IGJ/top1000.html’
url = htmlParse(u, encoding=”GB2312″)
tables = readHTMLTable(url)
raw = tables[[6]]
查看raw中文显示乱码, 查看sessionInfo(),
R version 2.15.1 (2012-06-22)
Platform. x86_64-pc-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=Chinese (Simplified)_People's Republic of China.936 
[2] LC_CTYPE=Chinese (Simplified)_People's Republic of China.936   
[3] LC_MONETARY=Chinese (Simplified)_People's Republic of China.936
[4] LC_NUMERIC=C                                                   
[5] LC_TIME=Chinese (Simplified)_People's Republic of China.936    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

other attached packages:
[1] XML_3.95-0.1

loaded via a namespace (and not attached):
[1] tools_2.15.1

这个与操作相关, 可以尝试更改Sys.setlocale("LC_CTYPE", "UTF-8"),但报“操作系统报告说无法执行将本地化设成"UTF-8"的请求”。


在Ubuntu中使用RStudio却能正确显示中文,查看sessionInfo()
R version 2.14.1 (2011-12-22)
Platform. x86_64-pc-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C         LC_TIME=C           
 [4] LC_COLLATE=C         LC_MONETARY=C        LC_MESSAGES=C       
 [7] LC_PAPER=C           LC_NAME=C            LC_ADDRESS=C        
[10] LC_TELEPHONE=C       LC_MEASUREMENT=C ""LC_IDENTIFICATION"" =C 

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods  
[7] base     

loaded via a namespace (and not attached):
[1] tools_2.14.1



造成的原因推测是XML包编码方式与操作系统的字符编码相关。 有高手知道的具体原因的请帮忙解答下。

python 无法使用numpy

环境: Ubuntu 12.4

>>> import numpy
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
File “/usr/lib/python2.7/dist-packages/numpy/__init__.py”, line 137, in <module>
import add_newdocs
File “/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py”, line 9, in <module>
from numpy.lib import add_newdoc
File “/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py”, line 13, in <module>
from polynomial import *
File “/usr/lib/python2.7/dist-packages/numpy/lib/polynomial.py”, line 11, in <module>
import numpy.core.numeric as NX
AttributeError: ‘module’ object has no attribute ‘core’
>>>

解决方法:   sudo apt-get remove libopenblas-base

Python PIL图片数据操作

环境: Ubuntu 12.4

模式

1  1位像素,黑和白,存成8位的像素
L  8位像素,黑白
P  8位像素,使用调色板映射到任何其他模式
RGB  3×8位像素,真彩
RGBA  4×8位像素,真彩+透明通道
CMYK  4×8位像素,颜色隔离
YCbCr  3×8位像素,彩色视频格式
I  32位整型像素
F  32位浮点型像素

导入图片处理库
>>>from PIL import Image
加载图片文件
>>>im = Image.open(‘aa.jpeg’)
输出文件的格式,大小,模式
>>> print im.format, im.size, im.mode
JPEG (510, 333) RGB

显示图片
>>> im.show()

白化图片
>>> im.convert(‘1’).show()

获取图片pixel数据
方法一:
>>> import numpy
>>> numpy.asarray(im)[100, 100]

array([254, 167, 175], dtype=uint8)

方法二:
>>> im.getpixel((100, 100))
(254, 167, 175)

方法三:

>>> im.load()[100, 100]
(254, 167, 175)

几何转变提供resize,rotate等方法,用以重定义图片大小,对图片进行旋转等操作。
>>> im.resize((250, 150)).show()
>>> im.rotate(45).show()

图片加强

滤镜
ImageFilter模块提供了很多预定义的图片加强滤镜
>>>import ImageFilter
>>> im.filter(ImageFilter.DETAIL).show()

>>>import ImageEnhance
>>>ImageEnhance.Contrast(im).enhance(1.5).show()

通道
一个图片包含一个或多个数据通道, 如果这些通道有相同的长度和宽度, 可以进入叠加
>>> r,g,b = im.split()
>>>r.sohw()
>>>g.show()
>>>b.show()
>>>im2 = Image.merge(‘RGB’, (b,g,r))

crontab调度Python脚本

环境:Red Hat Enterprise Linux Server release 5.4 (Tikanga),  python 2.7
由于本人对shell脚本语法相当不喜欢,一般与操作系统打交道用python脚本。在用crontab调度python脚有两个地方要注意:
1.  在python脚本中增加#!/usr/bin/env python来声明执行程序在系统环境变量中的名字
2.  在crontab中增加. $HOME/.bash_profile来调用bash_profile来声明环境变量,例: 19 16 * * * . $HOME/.bash_profile;/home/oradev/wget_test.py
注:  文件wget_test.py为可执行状态 — chmod +x wget_test.py
        该文件的目录必须在环境变量PATH 内。
还遇到一问题就是脚本不执行也不报错,最终发现python有两个版本,一个版本在/usr/bin/python是2.4.3,将其删除并指定软链接 ln -s /usr/local/bin/python2.7 /usr/bin/python

使用CDH4 Maven Repository

环境: CentOS

在使用Maven编译一些与hadoop相关的产品时候需要使用hadoop相关版本对应的核心组件,而自己使用的大多数都是CDH版本。因而需要从些版本上下载相应的包。

相应的解决方法是在pom.xml增加如下:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <repositories>
    <repository>
      <id>cloudera</id>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
    </repository>
  </repositories>

</project>
以下显示的是project name, groupId, artifactId, and version required to access each CDH4 artifact.
Project groupId artifactId version
Hadoop org.apache.hadoop hadoop-annotations 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-archives 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-assemblies 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-auth 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-client 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-common 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-datajoin 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-dist 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-distcp 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-extras 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-gridmix 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-hdfs 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-mapreduce-client-app 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-mapreduce-client-common 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-mapreduce-client-core 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-mapreduce-client-hs 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-mapreduce-client-jobclient 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-mapreduce-client-shuffle 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-mapreduce-examples 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-rumen 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-api 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-applications-distributedshell 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-applications-unmanaged-am-launcher 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-client 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-common 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-server-common 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-server-nodemanager 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-server-resourcemanager 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-server-tests 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-server-web-proxy 2.0.0-cdh4.2.0
  org.apache.hadoop hadoop-yarn-site 2.0.0-cdh4.2.0
Hadoop MRv1 org.apache.hadoop hadoop-core 2.0.0-mr1-cdh4.2.0
  org.apache.hadoop hadoop-examples 2.0.0-mr1-cdh4.2.0
  org.apache.hadoop hadoop-minicluster 2.0.0-mr1-cdh4.2.0
  org.apache.hadoop hadoop-streaming 2.0.0-mr1-cdh4.2.0
  org.apache.hadoop hadoop-test 2.0.0-mr1-cdh4.2.0
  org.apache.hadoop hadoop-tools 2.0.0-mr1-cdh4.2.0
Hive org.apache.hive hive-anttasks 0.10.0-cdh4.2.0
  org.apache.hive hive-builtins 0.10.0-cdh4.2.0
  org.apache.hive hive-cli 0.10.0-cdh4.2.0
  org.apache.hive hive-common 0.10.0-cdh4.2.0
  org.apache.hive hive-contrib 0.10.0-cdh4.2.0
  org.apache.hive hive-exec 0.10.0-cdh4.2.0
  org.apache.hive hive-hbase-handler 0.10.0-cdh4.2.0
  org.apache.hive hive-hwi 0.10.0-cdh4.2.0
  org.apache.hive hive-jdbc 0.10.0-cdh4.2.0
  org.apache.hive hive-metastore 0.10.0-cdh4.2.0
  org.apache.hive hive-pdk 0.10.0-cdh4.2.0
  org.apache.hive hive-serde 0.10.0-cdh4.2.0
  org.apache.hive hive-service 0.10.0-cdh4.2.0
  org.apache.hive hive-shims 0.10.0-cdh4.2.0
HBase org.apache.hbase hbase 0.94.2-cdh4.2.0
ZooKeeper org.apache.zookeeper zookeeper 3.4.5-cdh4.2.0
Sqoop org.apache.sqoop sqoop 1.4.2-cdh4.2.0
Pig org.apache.pig pig 0.10.0-cdh4.2.0
  org.apache.pig pigsmoke 0.10.0-cdh4.2.0
  org.apache.pig pigunit 0.10.0-cdh4.2.0
Flume 1.x org.apache.flume flume-ng-configuration 1.3.0-cdh4.2.0
  org.apache.flume flume-ng-core 1.3.0-cdh4.2.0
  org.apache.flume flume-ng-embedded-agent 1.3.0-cdh4.2.0
  org.apache.flume flume-ng-node 1.3.0-cdh4.2.0
  org.apache.flume flume-ng-sdk 1.3.0-cdh4.2.0
  org.apache.flume flume-ng-tests 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-channels flume-file-channel 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-channels flume-jdbc-channel 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-channels flume-recoverable-memory-channel 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-clients flume-ng-log4jappender 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-legacy-sources flume-avro-source 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-legacy-sources flume-thrift-source 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-sinks flume-hdfs-sink 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-sinks flume-irc-sink 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-sinks flume-ng-elasticsearch-sink 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-sinks flume-ng-hbase-sink 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-sources flume-jms-source 1.3.0-cdh4.2.0
  org.apache.flume.flume-ng-sources flume-scribe-source 1.3.0-cdh4.2.0
Oozie org.apache.oozie oozie-client 3.3.0-cdh4.2.0
  org.apache.oozie oozie-core 3.3.0-cdh4.2.0
  org.apache.oozie oozie-examples 3.3.0-cdh4.2.0
  org.apache.oozie oozie-hadoop 2.0.0-cdh4.2.0.oozie-3.3.0-cdh4.2.0
  org.apache.oozie oozie-hadoop-distcp 2.0.0-mr1-cdh4.2.0.oozie-3.3.0-cdh4.2.0
  org.apache.oozie oozie-hadoop-test 2.0.0-mr1-cdh4.2.0.oozie-3.3.0-cdh4.2.0
  org.apache.oozie oozie-hbase 0.94.2-cdh4.2.0.oozie-3.3.0-cdh4.2.0
  org.apache.oozie oozie-sharelib-distcp 3.3.0-cdh4.2.0
  org.apache.oozie oozie-sharelib-distcp-yarn 3.3.0-cdh4.2.0
  org.apache.oozie oozie-sharelib-hive 3.3.0-cdh4.2.0
  org.apache.oozie oozie-sharelib-oozie 3.3.0-cdh4.2.0
  org.apache.oozie oozie-sharelib-pig 3.3.0-cdh4.2.0
  org.apache.oozie oozie-sharelib-sqoop 3.3.0-cdh4.2.0
  org.apache.oozie oozie-sharelib-streaming 3.3.0-cdh4.2.0
  org.apache.oozie oozie-sharelib-streaming-yarn 3.3.0-cdh4.2.0
  org.apache.oozie oozie-tools 3.3.0-cdh4.2.0
Mahout org.apache.mahout mahout-buildtools 0.7-cdh4.2.0
  org.apache.mahout mahout-core 0.7-cdh4.2.0
  org.apache.mahout mahout-examples 0.7-cdh4.2.0
  org.apache.mahout mahout-integration 0.7-cdh4.2.0
  org.apache.mahout mahout-math 0.7-cdh4.2.0
Whirr org.apache.whirr whirr-build-tools 0.8.0-cdh4.2.0
  org.apache.whirr whirr-cassandra 0.8.0-cdh4.2.0
  org.apache.whirr whirr-cdh 0.8.0-cdh4.2.0
  org.apache.whirr whirr-chef 0.8.0-cdh4.2.0
  org.apache.whirr whirr-cli 0.8.0-cdh4.2.0
  org.apache.whirr whirr-core 0.8.0-cdh4.2.0
  org.apache.whirr whirr-elasticsearch 0.8.0-cdh4.2.0
  org.apache.whirr whirr-examples 0.8.0-cdh4.2.0
  org.apache.whirr whirr-ganglia 0.8.0-cdh4.2.0
  org.apache.whirr whirr-hadoop 0.8.0-cdh4.2.0
  org.apache.whirr whirr-hama 0.8.0-cdh4.2.0
  org.apache.whirr whirr-hbase 0.8.0-cdh4.2.0
  org.apache.whirr whirr-mahout 0.8.0-cdh4.2.0
  org.apache.whirr whirr-pig 0.8.0-cdh4.2.0
  org.apache.whirr whirr-puppet 0.8.0-cdh4.2.0
  org.apache.whirr whirr-solr 0.8.0-cdh4.2.0
  org.apache.whirr whirr-yarn 0.8.0-cdh4.2.0
  org.apache.whirr whirr-zookeeper 0.8.0-cdh4.2.0
DataFu com.linkedin.datafu datafu 0.0.4-cdh4.2.0
Sqoop2 org.apache.sqoop sqoop-client 1.99.1-cdh4.2.0
  org.apache.sqoop sqoop-common 1.99.1-cdh4.2.0
  org.apache.sqoop sqoop-core 1.99.1-cdh4.2.0
  org.apache.sqoop sqoop-docs 1.99.1-cdh4.2.0
  org.apache.sqoop sqoop-spi 1.99.1-cdh4.2.0
  org.apache.sqoop.connector sqoop-connector-generic-jdbc 1.99.1-cdh4.2.0
  org.apache.sqoop.repository sqoop-repository-derby 1.99.1-cdh4.2.0
HCatalog org.apache.hcatalog hcatalog-core 0.4.0-cdh4.2.0
  org.apache.hcatalog hcatalog-pig-adapter 0.4.0-cdh4.2.0
  org.apache.hcatalog hcatalog-server-extensions 0.4.0-cdh4.2.0
  org.apache.hcatalog webhcat 0.4.0-cdh4.2.0
  org.apache.hcatalog webhcat-java-client 0.4.0-cdh4.2.0

sqoop的插件oraoop

环境: Centos 5.7,  CDH 4.3, sqoop 1.6

从http://downloads.cloudera.com/connectors/oraoop-1.6.0-cdh4.tgz 下载oraoop,解压生成
[oracle@xxx ~]$ ls oraoop-1.6.0
bin  conf  docs  install.sh  version.txt

设置环境参数vi ~/.bash_profile
export SQOOP_CONF_DIR=/etc/sqoop/conf
export SQOOP_HOME=/u01/cloudera/parcels/CDH/lib/sqoop
export HADOOP_CLIENT_OPTS=”-Xmx2048m $HADOOP_CLIENT_OPTS”

然后执行安装脚本./install.sh, 测试安装效果:

[oracle@xxx ~]$ sqoop list-tables –verbose –connect jdbc:oracle:thin:@xxx:8521:biprod –username xxx –password xxx
14/09/23 18:39:49 DEBUG tool.BaseSqoopTool: Enabled debug logging.
14/09/23 18:39:49 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/09/23 18:39:49 DEBUG util.ClassLoaderStack: Checking for existing class: com.quest.oraoop.OraOopManagerFactory
14/09/23 18:39:49 DEBUG util.ClassLoaderStack: Class is already available. Skipping jar /u01/cloudera/parcels/CDH/lib/sqoop/lib/oraoop-1.6.0.jar
14/09/23 18:39:49 DEBUG sqoop.ConnFactory: Added factory com.quest.oraoop.OraOopManagerFactory in jar /u01/cloudera/parcels/CDH/lib/sqoop/lib/oraoop-1.6.0.jar specified by /etc/sqoop/conf/managers.d/oraoop
14/09/23 18:39:49 DEBUG sqoop.ConnFactory: Loaded manager factory: com.quest.oraoop.OraOopManagerFactory
14/09/23 18:39:49 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
14/09/23 18:39:49 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.quest.oraoop.OraOopManagerFactory
14/09/23 18:39:49 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
14/09/23 18:39:49 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:oracle:thin:@xxx:8521
14/09/23 18:39:49 DEBUG manager.OracleManager$ConnCache: Instantiated new connection cache.
14/09/23 18:39:49 INFO manager.SqlManager: Using default fetchSize of 1000
14/09/23 18:39:49 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.OracleManager@52f6438d
14/09/23 18:39:49 DEBUG manager.OracleManager: Creating a new connection for jdbc:oracle:thin:@xxx:8521:biprod, using username: SQOOP_USER
14/09/23 18:39:49 DEBUG manager.OracleManager: No connection paramenters specified. Using regular API for making connection.
14/09/23 18:40:01 INFO manager.OracleManager: Time zone has been set to GMT
14/09/23 18:40:02 DEBUG manager.OracleManager$ConnCache: Caching released connection for jdbc:oracle:thin:@xxx:8521:biprod/SQOOP_USER
OS_ZHIXIN_CHG
T1