2014年6月30日月曜日

perlのcpanモジュールをchefで入れてみた

先日書いたブログでは
cpanコマンドで一つずつモジュールをインストールしていましたが、
やはりめんどくさいのでchefで入れてみました。

やはりそういう風に考える人はいるみたいで、recipe探せばすぐ見つかりましたね。
https://github.com/melezhik/cookbooks/tree/master/cpan

入れたいモジュールをこんな感じで羅列するだけで簡単に入れられました。
%w{
  ExtUtils::CBuilder
  ExtUtils::MakeMaker
  Test::Deep
  CPAN::Meta
  Module::Build
  JSON
  JSON::XS
  Try::Tiny
  URI
  File::Temp
  Test::Requires
  ExtUtils::Constant
  Socket
  Test::SharedFork
  Time::HiRes
  Test::TCP
  Class::Accessor::Lite
  Encode
  HTTP::Parser::XS
  MIME::Base64
  Mozilla::CA
  Scalar::Util
  Furl
  Net::Hadoop::WebHDFS
  Readonly
  Crypt::CBC
  Path::Class
  YAML::Syck
  Crypt::Rijndael
  Test::Exception
  Text::Glob
}.each do |mod|
  cpan_client "#{mod}" do
    action 'install'
    install_type 'cpan_module'
    user 'root'
    group 'root'
  end
end


参考URL
https://github.com/melezhik/cookbooks/tree/master/cpan

2014年6月27日金曜日

perlでcpanモジュールを入れてみた。

perlは全然やったことないし、あんまり好きでもないのだけれど、
なんかperlでライブラリを提供されたので、perlのスクリプトを書くことになってしまい、
cpanとかよくわからないけど触ってみたのでブログ書いてみた。

用意してもらったライブラリはHadoopにwebHDFSでファイルをアップするためのラッパー。
# perl test.pl 
Can't locate Net/Hadoop/WebHDFS.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/share/perl5/sample/WebHDFS.pm line 6.
BEGIN failed--compilation aborted at /usr/local/share/perl5/sample/WebHDFS.pm line 6.
Compilation failed in require at test.pl line 1.
BEGIN failed--compilation aborted at test.pl line 1.
とりあえず使ってみたら、Can't locate Net/Hadoop/WebHDFS.pmと言われた。
モジュールがないんだろうなということで、cpanで入れてみた。
途中「yes」と入れるところがあったりするが、ずらずらーっと一気に流れていく。
# cpan Net::Hadoop::WebHDFS
 
 
CPAN is the world-wide archive of perl resources. It consists of about
300 sites that all replicate the same contents around the globe. Many
countries have at least one CPAN site already. The resources found on
CPAN are easily accessible with the CPAN.pm module. If you want to use
CPAN.pm, lots of things have to be configured. Fortunately, most of
them can be determined automatically. If you prefer the automatic
configuration, answer 'yes' below.
 
If you prefer to enter a dialog instead, you can answer 'no' to this
question and I'll let you configure in small steps one thing after the
other. (Note: you can revisit this dialog anytime later by typing 'o
conf init' at the cpan prompt.)
Would you like me to configure as much as possible automatically? [yes]
 
 
The following questions are intended to help you with the
configuration. The CPAN module needs a directory of its own to cache
important index files and maybe keep a temporary mirror of CPAN files.
This may be a site-wide or a personal directory.
 
 
 
First of all, I'd like to create this directory. Where?
 
 
CPAN build and cache directory? [/root/.cpan]
 
 
Unless you are accessing the CPAN on your filesystem via a file: URL,
CPAN.pm needs to keep the source files it downloads somewhere. Please
supply a directory where the downloaded files are to be kept.
 
 
Download target directory? [/root/.cpan/sources]
 
 
 
Directory where the build process takes place? [/root/.cpan/build]
 
 
Until version 1.88 CPAN.pm never trusted the contents of the build_dir
directory between sessions. Since 1.88_58 CPAN.pm has a YAML-based
mechanism that makes it possible to share the contents of the
build_dir/ directory between different sessions with the same version
of perl. People who prefer to test things several days before
installing will like this feature because it safes a lot of time.
 
If you say yes to the following question, CPAN will try to store
enough information about the build process so that it can pick up in
future sessions at the same state of affairs as it left a previous
session.
 
 
Store and re-use state information about distributions between
CPAN.pm sessions? [no]
 
 
CPAN.pm can store customized build environments based on regular
expressions for distribution names. These are YAML files where the
default options for CPAN.pm and the environment can be overridden and
dialog sequences can be stored that can later be executed by an
Expect.pm object. The CPAN.pm distribution comes with some prefab YAML
files that cover sample distributions that can be used as blueprints
to store one own prefs. Please check out the distroprefs/ directory of
the CPAN.pm distribution to get a quick start into the prefs system.
 
 
Directory where to store default options/environment/dialogs for
building modules that need some customization? [/root/.cpan/prefs]
 
 
Normally CPAN.pm keeps config variables in memory and changes need to
be saved in a separate 'o conf commit' command to make them permanent
between sessions. If you set the 'auto_commit' option to true, changes
to a config variable are always automatically committed to disk.
 
 
Always commit changes to config variables to disk? [no]
 
 
CPAN.pm can limit the size of the disk area for keeping the build
directories with all the intermediate files.
 
 
Cache size for build directory (in MB)? [100]
 
 
The CPAN indexes are usually rebuilt once or twice per hour, but the
typical CPAN mirror mirrors only once or twice per day. Depending on
the quality of your mirror and your desire to be on the bleeding edge,
you may want to set the following value to more or less than one day
(which is the default). It determines after how many days CPAN.pm
downloads new indexes.
 
 
Let the index expire after how many days? [1]
 
 
By default, each time the CPAN module is started, cache scanning is
performed to keep the cache size in sync. To prevent this, answer
'never'.
 
 
Perform cache scanning (atstart or never)? [atstart]
 
 
To considerably speed up the initial CPAN shell startup, it is
possible to use Storable to create a cache of metadata. If Storable is
not available, the normal index mechanism will be used.
 
Note: this mechanism is not used when use_sqlite is on and SQLLite is
running.
 
 
Cache metadata (yes/no)? [yes]
 
 
CPAN::SQLite is a layer between the index files that are downloaded
from the CPAN and CPAN.pm that speeds up metadata queries and reduces
memory consumption of CPAN.pm considerably.
 
 
Use CPAN::SQLite if available? (yes/no)? [no]
 
 
The CPAN module can detect when a module which you are trying to build
depends on prerequisites. If this happens, it can build the
prerequisites for you automatically ('follow'), ask you for
confirmation ('ask'), or just ignore them ('ignore'). Please set your
policy to one of the three values.
 
 
Policy on building prerequisites (follow, ask or ignore)? [ask]
 
 
When a module declares another one as a 'build_requires' prerequisite
this means that the other module is only needed for building or
testing the module but need not be installed permanently. In this case
you may wish to install that other module nonetheless or just keep it
in the 'build_dir' directory to have it available only temporarily.
Installing saves time on future installations but makes the perl
installation bigger.
 
You can choose if you want to always install (yes), never install (no)
or be always asked. In the latter case you can set the default answer
for the question to yes (ask/yes) or no (ask/no).
 
 
Policy on installing 'build_requires' modules (yes, no, ask/yes,
ask/no)? [ask/yes]
 
 
CPAN packages can be digitally signed by authors and thus verified
with the security provided by strong cryptography. The exact mechanism
is defined in the Module::Signature module. While this is generally
considered a good thing, it is not always convenient to the end user
to install modules that are signed incorrectly or where the key of the
author is not available or where some prerequisite for
Module::Signature has a bug and so on.
 
With the check_sigs parameter you can turn signature checking on and
off. The default is off for now because the whole tool chain for the
functionality is not yet considered mature by some. The author of
CPAN.pm would recommend setting it to true most of the time and
turning it off only if it turns out to be annoying.
 
Note that if you do not have Module::Signature installed, no signature
checks will be performed at all.
 
 
Always try to check and verify signatures if a SIGNATURE file is in
the package and Module::Signature is installed (yes/no)? [no]
 
 
The goal of the CPAN Testers project (http://testers.cpan.org/) is to
test as many CPAN packages as possible on as many platforms as
possible.  This provides valuable feedback to module authors and
potential users to identify bugs or platform compatibility issues and
improves the overall quality and value of CPAN.
 
One way you can contribute is to send test results for each module
that you install.  If you install the CPAN::Reporter module, you have
the option to automatically generate and email test reports to CPAN
Testers whenever you run tests on a CPAN package.
 
See the CPAN::Reporter documentation for additional details and
configuration settings.  If your firewall blocks outgoing email,
you will need to configure CPAN::Reporter before sending reports.
 
 
Email test reports if CPAN::Reporter is installed (yes/no)? [no]
 
 
When a distribution has already been tested by CPAN::Reporter on
this machine, CPAN can skip the test phase and just rely on the
test report history instead.
 
Note that this will not apply to distributions that failed tests
because of missing dependencies.  Also, tests can be run
regardless of the history using "force".
 
 
Do you want to rely on the test report history (yes/no)? [no]
 
 
At the time of this writing (2009-03) there are three YAML
implementations working: YAML, YAML::Syck, and YAML::XS. The latter
two are faster but need a C compiler installed on your system. There
may be more alternative YAML conforming modules. When I tried two
other players, YAML::Tiny and YAML::Perl, they seemed not powerful
enough to work with CPAN.pm. This may have changed in the meantime.
 
 
Which YAML implementation would you prefer? [YAML]
 
 
Warning (maybe harmless): 'YAML' not installed.
Both YAML.pm and YAML::Syck are capable of deserialising code. As this
requires a string eval, which might be a security risk, you can use
this option to enable or disable the deserialisation of code via
CPAN::DeferredCode. (Note: This does not work under perl 5.6)
 
 
Do you want to enable code deserialisation (yes/no)? [no]
 
 
 
 
The CPAN module will need a few external programs to work properly.
Please correct me, if I guess the wrong path for a program. Don't
panic if you do not have some of them, just press ENTER for those. To
disable the use of a program, you can type a space followed by ENTER.
 
 
Where is your bzip2 program? [/usr/bin/bzip2]
 
 
 
Where is your gzip program? [/bin/gzip]
 
 
 
Where is your tar program? [/bin/tar]
 
 
 
Where is your unzip program? [/usr/bin/unzip]
 
 
 
Where is your make program? [/usr/bin/make]
 
 
 
Where is your curl program? [/usr/bin/curl]
 
 
Warning: lynx not found in PATH[/usr/local/maven/bin;/usr/local/sbin;/usr/local/bin;/sbin;/bin;/usr/sbin;/usr/bin;/home/user/core/bin;/usr/local/groovy/bin;/usr/java/default/bin;/root/bin;/opt/infra/ruby/bin]
 
Where is your lynx program? []
 
 
 
Where is your wget program? [/usr/bin/wget]
 
 
Warning: ncftpget not found in PATH[/usr/local/maven/bin;/usr/local/sbin;/usr/local/bin;/sbin;/bin;/usr/sbin;/usr/bin;/home/user/core/bin;/usr/local/groovy/bin;/usr/java/default/bin;/root/bin;/opt/infra/ruby/bin]
 
Where is your ncftpget program? []
 
 
Warning: ncftp not found in PATH[/usr/local/maven/bin;/usr/local/sbin;/usr/local/bin;/sbin;/bin;/usr/sbin;/usr/bin;/home/user/core/bin;/usr/local/groovy/bin;/usr/java/default/bin;/root/bin;/opt/infra/ruby/bin]
 
Where is your ncftp program? []
 
 
Warning: ftp not found in PATH[/usr/local/maven/bin;/usr/local/sbin;/usr/local/bin;/sbin;/bin;/usr/sbin;/usr/bin;/home/user/core/bin;/usr/local/groovy/bin;/usr/java/default/bin;/root/bin;/opt/infra/ruby/bin]
 
Where is your ftp program? []
 
 
 
Where is your gpg program? [/usr/bin/gpg]
 
 
 
Where is your patch program? [/usr/bin/patch]
 
 
Warning: applypatch not found in PATH[/usr/local/maven/bin;/usr/local/sbin;/usr/local/bin;/sbin;/bin;/usr/sbin;/usr/bin;/home/usercore/bin;/usr/local/groovy/bin;/usr/java/default/bin;/root/bin;/opt/infra/ruby/bin]
 
Where is your applypatch program? []
 
 
 
What is your favorite pager program? [/usr/bin/less]
 
 
 
What is your favorite shell? [/bin/bash]
 
 
When CPAN.pm uses the tar command, which switch for the verbosity
shall be used? Choose 'none' for quiet operation, 'v' for file
name listing, 'vv' for full listing.
 
 
Tar command verbosity level (none or v or vv)? [v]
 
 
When CPAN.pm loads a module it needs for some optional feature, it
usually reports about module name and version. Choose 'v' to get this
message, 'none' to suppress it.
 
 
Verbosity level for loading modules (none or v)? [v]
 
 
When CPAN.pm extends @INC via PERL5LIB, it prints a list of
directories added (or a summary of how many directories are
added).  Choose 'v' to get this message, 'none' to suppress it.
 
 
Verbosity level for PERL5LIB changes (none or v)? [v]
 
 
When the CPAN shell is started it normally displays a greeting message
that contains the running version and the status of readline support.
 
 
Do you want to turn this message off? [no]
 
 
When you have Module::Build installed and a module comes with both a
Makefile.PL and a Build.PL, which shall have precedence?
 
The main two standard installer modules are the old and well
established ExtUtils::MakeMaker (for short: EUMM) which uses the
Makefile.PL. And the next generation installer Module::Build (MB)
which works with the Build.PL (and often comes with a Makefile.PL
too). If a module comes only with one of the two we will use that one
but if both are supplied then a decision must be made between EUMM and
MB. See also http://rt.cpan.org/Ticket/Display.html?id=29235 for a
discussion about the right default.
 
Or, as a third option you can choose RAND which will make a random
decision (something regular CPAN testers will enjoy).
 
 
In case you can choose between running a Makefile.PL or a Build.PL,
which installer would you prefer (EUMM or MB or RAND)? [MB]
 
 
Every Makefile.PL is run by perl in a separate process. Likewise we
run 'make' and 'make install' in separate processes. If you have
any parameters (e.g. PREFIX, UNINST or the like) you want to
pass to the calls, please specify them here.
 
If you don't understand this question, just press ENTER.
 
Typical frequently used settings:
 
    PREFIX=~/perl    # non-root users (please see manual for more hints)
 
 
Parameters for the 'perl Makefile.PL' command? [INSTALLDIRS=site]
 
 
Parameters for the 'make' command? Typical frequently used setting:
 
    -j3              # dual processor system (on GNU make)
 
 
Your choice: []
 
 
Do you want to use a different make command for 'make install'?
Cautious people will probably prefer:
 
    su root -c make
 or
    sudo make
 or
    /path1/to/sudo -u admin_account /path2/to/make
 
 
or some such. Your choice: [/usr/bin/make]
 
 
Parameters for the 'make install' command?
Typical frequently used setting:
 
    UNINST=1         # to always uninstall potentially conflicting files
 
 
Your choice: []
 
 
A Build.PL is run by perl in a separate process. Likewise we run
'./Build' and './Build install' in separate processes. If you have any
parameters you want to pass to the calls, please specify them here.
 
Typical frequently used settings:
 
    --install_base /home/xxx             # different installation directory
 
 
Parameters for the 'perl Build.PL' command? [--installdirs site]
 
 
Parameters for the './Build' command? Setting might be:
 
    --extra_linker_flags -L/usr/foo/lib  # non-standard library location
 
 
Your choice: []
 
 
Do you want to use a different command for './Build install'? Sudo
users will probably prefer:
 
    su root -c ./Build
 or
    sudo ./Build
 or
    /path1/to/sudo -u admin_account ./Build
 
 
or some such. Your choice: [./Build]
 
 
Parameters for the './Build install' command? Typical frequently used
setting:
 
    --uninst 1                           # uninstall conflicting files
 
 
Your choice: []
 
 
Sometimes you may wish to leave the processes run by CPAN alone
without caring about them. Because the Makefile.PL or the Build.PL
sometimes contains question you're expected to answer, you can set a
timer that will kill a 'perl Makefile.PL' process after the specified
time in seconds.
 
If you set this value to 0, these processes will wait forever. This is
the default and recommended setting.
 
 
Timeout for inactivity during {Makefile,Build}.PL? [0]
 
 
Normaly, CPAN.pm continues processing the full list of targets and
dependencies, even if one of them fails.  However, you can specify 
that CPAN should halt after the first failure. 
 
 
Do you want to halt on failure (yes/no)? [no]
 
 
 
 
If you're accessing the net via proxies, you can specify them in the
CPAN configuration or via environment variables. The variable in
the $CPAN::Config takes precedence.
 
 
Your ftp_proxy? []
 
 
 
Your http_proxy? []
 
 
 
Your no_proxy? []
 
 
 
Shall we always set the FTP_PASSIVE environment variable when dealing
with ftp download (yes/no)? [yes]
 
 
CPAN.pm changes the current working directory often and needs to
determine its own current working directory. Per default it uses
Cwd::cwd but if this doesn't work on your system for some reason,
alternatives can be configured according to the following table:
 
    cwd         Cwd::cwd
    getcwd      Cwd::getcwd
    fastcwd     Cwd::fastcwd
    backtickcwd external command cwd
 
 
Preferred method for determining the current working directory? [cwd]
 
 
The prompt of the cpan shell can contain the current command number
for easier tracking of the session or be a plain string.
 
 
Do you want the command number in the prompt (yes/no)? [yes]
 
 
When using Term::ReadLine, you can turn ornaments on so that your
input stands out against the output from CPAN.pm.
 
 
Do you want to turn ornaments on? [yes]
 
 
The next option deals with the charset (aka character set) your
terminal supports. In general, CPAN is English speaking territory, so
the charset does not matter much but some CPAN have names that are
outside the ASCII range. If your terminal supports UTF-8, you should
say no to the next question. If it expects ISO-8859-1 (also known as
LATIN1) then you should say yes. If it supports neither, your answer
does not matter because you will not be able to read the names of some
authors anyway. If you answer no, names will be output in UTF-8.
 
 
Your terminal expects ISO-8859-1 (yes/no)? [yes]
 
 
If you have one of the readline packages (Term::ReadLine::Perl,
Term::ReadLine::Gnu, possibly others) installed, the interactive CPAN
shell will have history support. The next two questions deal with the
filename of the history file and with its size. If you do not want to
set this variable, please hit SPACE RETURN to the following question.
 
If you have one of the readline packages (Term::ReadLine::Perl,
Term::ReadLine::Gnu, possibly others) installed, the interactive CPAN
shell will have history support. The next two questions deal with the
filename of the history file and with its size. If you do not want to
set this variable, please hit SPACE RETURN to the following question.
 
 
File to save your history? [/root/.cpan/histfile]
 
 
 
Number of lines to save? [100]
 
 
The 'd' and the 'm' command normally only show you information they
have in their in-memory database and thus will never connect to the
internet. If you set the 'show_upload_date' variable to true, 'm' and
'd' will additionally show you the upload date of the module or
distribution. Per default this feature is off because it may require a
net connection to get at the upload date.
 
 
Always try to show upload date with 'd' and 'm' command (yes/no)? [no]
 
 
During the 'r' command CPAN.pm finds modules without version number.
When the command finishes, it prints a report about this. If you
want this report to be very verbose, say yes to the following
variable.
 
 
Show all individual modules that have no $VERSION? [no]
 
 
During the 'r' command CPAN.pm finds modules with a version number of
zero. When the command finishes, it prints a report about this. If you
want this report to be very verbose, say yes to the following
variable.
 
 
Show all individual modules that have a $VERSION of zero? [no]
 
 
If you have never defined your own C in your configuration
then C will be hesitant to use the built in default sites for
downloading. It will ask you once per session if a connection to the
internet is OK and only if you say yes, it will try to connect. But to
avoid this question, you can choose your favorite download sites once
and get away with it. Or, if you have no favorite download sites
answer yes to the following question.
 
 
If no urllist has been chosen yet, would you prefer CPAN.pm to connect
to the built-in default sites without asking? (yes/no)? [yes]
 
 
Autoconfigured everything but 'urllist'.
Please call 'o conf init urllist' to configure your CPAN server(s) now!
 
 
 
commit: wrote '/usr/share/perl5/CPAN/Config.pm'
CPAN: Storable loaded ok (v2.20)
CPAN: LWP::UserAgent loaded ok (v5.833)
Warning: no success downloading '/root/.cpan/sources/authors/01mailrc.txt.gz.tmp30757'. Giving up on it. at /usr/share/perl5/CPAN/Index.pm line 225
Fetching with LWP:
  http://www.perl.org/CPAN/authors/01mailrc.txt.gz
Going to read '/root/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................DONE
Fetching with LWP:
  http://www.perl.org/CPAN/modules/02packages.details.txt.gz
Going to read '/root/.cpan/sources/modules/02packages.details.txt.gz'
  Database was generated on Thu, 19 Jun 2014 05:41:02 GMT
..............
  New CPAN.pm version (v2.05) available.
  [Currently running version is v1.9402]
  You might want to try
    install CPAN
    reload cpan
  to both upgrade CPAN.pm and run the new version without leaving
  the current session.
 
 
..............................................................DONE
Fetching with LWP:
  http://www.perl.org/CPAN/modules/03modlist.data.gz
Going to read '/root/.cpan/sources/modules/03modlist.data.gz'
DONE
Going to write /root/.cpan/Metadata
Running install for module 'Net::Hadoop::WebHDFS'
CPAN: Data::Dumper loaded ok (v2.124)
'YAML' not installed, falling back to Data::Dumper and Storable to read prefs '/root/.cpan/prefs'
Running make for T/TA/TAGOMORIS/Net-Hadoop-WebHDFS-0.5.tar.gz
Fetching with LWP:
  http://www.perl.org/CPAN/authors/id/T/TA/TAGOMORIS/Net-Hadoop-WebHDFS-0.5.tar.gz
CPAN: Digest::SHA loaded ok (v5.47)
Fetching with LWP:
  http://www.perl.org/CPAN/authors/id/T/TA/TAGOMORIS/CHECKSUMS
Checksum for /root/.cpan/sources/authors/id/T/TA/TAGOMORIS/Net-Hadoop-WebHDFS-0.5.tar.gz ok
CPAN: Archive::Tar loaded ok (v1.58)
Net-Hadoop-WebHDFS-0.5/Build.PL
Net-Hadoop-WebHDFS-0.5/Changes
Net-Hadoop-WebHDFS-0.5/LICENSE
Net-Hadoop-WebHDFS-0.5/META.json
Net-Hadoop-WebHDFS-0.5/META.yml
Net-Hadoop-WebHDFS-0.5/README.md
Net-Hadoop-WebHDFS-0.5/cpanfile
Net-Hadoop-WebHDFS-0.5/lib/Net/Hadoop/WebHDFS.pm
Net-Hadoop-WebHDFS-0.5/t/Net-Hadoop-WebHDFS.t
Net-Hadoop-WebHDFS-0.5/MANIFEST
CPAN: File::Temp loaded ok (v0.22)
 
  CPAN.pm: Going to build T/TA/TAGOMORIS/Net-Hadoop-WebHDFS-0.5.tar.gz
 
Can't locate CPAN/Meta.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at Build.PL line 15.
BEGIN failed--compilation aborted at Build.PL line 15.
Warning: No success on command[/usr/bin/perl Build.PL --installdirs site]
Warning (usually harmless): 'YAML' not installed, will not store persistent state
  TAGOMORIS/Net-Hadoop-WebHDFS-0.5.tar.gz
  /usr/bin/perl Build.PL --installdirs site -- NOT OK
Running Build test
  Make had some problems, won't test
Running Build install
  Make had some problems, won't install
Could not read '/root/.cpan/build/Net-Hadoop-WebHDFS-0.5-pJ80FW/META.yml'. Falling back to other methods to determine prerequisites
CPAN: Module::Build loaded ok (v0.35)

結局エラーになったようで、
Could not read '/root/.cpan/build/Net-Hadoop-WebHDFS-0.5-pJ80FW/META.yml'
と出ていたので、中身を見てみる。
どうやら必要なモジュールなどがここに書かれているようだ。
# cat /root/.cpan/build/Net-Hadoop-WebHDFS-0.5-pJ80FW/META.yml 
---
abstract: 'Client library for Hadoop WebHDFS and HttpFs'
author:
  - 'TAGOMORI Satoshi '
build_requires:
  ExtUtils::MakeMaker: 6.36
  Test::Deep: 0
  Test::More: 0
configure_requires:
  CPAN::Meta: 0
  CPAN::Meta::Prereqs: 0
  ExtUtils::MakeMaker: 0
  Module::Build: 0.38
dynamic_config: 0
generated_by: 'Minilla/v0.6.6, CPAN::Meta::Converter version 2.120921'
license: perl
meta-spec:
  url: http://module-build.sourceforge.net/META-spec-v1.4.html
  version: 1.4
name: Net-Hadoop-WebHDFS
no_index:
  directory:
    - t
    - xt
    - inc
    - share
    - eg
    - examples
    - author
provides:
  Net::Hadoop::WebHDFS:
    file: lib/Net/Hadoop/WebHDFS.pm
    version: 0.5
requires:
  Furl: 0
  JSON: 0
  JSON::XS: 0
  Test::Deep: 0
  Test::More: 0
  Try::Tiny: 0
  URI: 0
resources:
  bugtracker: https://github.com/tagomoris/Net-Hadoop-WebHDFS/issues
  homepage: https://github.com/tagomoris/Net-Hadoop-WebHDFS
  repository: git://github.com/tagomoris/Net-Hadoop-WebHDFS.git
version: 0.5
x_contributors:
  - 'TAGOMORI Satoshi '

そしてエラーが出てはymlを見るってのを繰り返して、
最終的にこれだけのモジュールを入れたら、ちゃんと動くようになりました。
CPAN::Meta
ExtUtils::MakeMaker
Module::Build
ExtUtils::CBuilder
Test::Deep
JSON
JSON::XS
Try::Tiny
URI
File::Temp
Test::Requires
ExtUtils::Constant
Socket
Test::SharedFork
Time::HiRes
Test::TCP
Class::Accessor::Lite
Encode
HTTP::Parser::XS
MIME::Base64
Mozilla::CA
Scalar::Util
Furl
Net::Hadoop::WebHDFS


参考URL
http://d.hatena.ne.jp/perlcodesample/20100504/1270894115
http://tech.thekyo.jp/cpan%E3%83%A2%E3%82%B8%E3%83%A5%E3%83%BC%E3%83%AB%E3%80%8Cdatetime%E3%80%8D%E3%81%AE%E3%82%A4%E3%83%B3%E3%82%B9%E3%83%88%E3%83%BC%E3%83%AB/910/
http://deps.cpantesters.org/?module=Net%3A%3AHadoop%3A%3AWebHDFS&perl=5.20.0&os=any+OS

2014年6月23日月曜日

railsをサーバーに載せたらエラー出たときの対応

Macでrailsのアプリケーションを書いてて、一通りできたのでCentOSに載せたらエラー出たので、そのときの対応です。

環境
MacOS 10.9
rails4.1.0
CentOS6.5

まずはgithubからcheckoutしてきてbundle installを実施
# bundle install --path vendor/bundle
Fetching gem metadata from https://rubygems.org/..........
Fetching additional metadata from https://rubygems.org/..
Installing rake (10.3.2)
Installing i18n (0.6.9)
Using json (1.8.1)
Installing minitest (5.3.4)
Installing thread_safe (0.3.4)
Installing tzinfo (1.2.1)
Installing activesupport (4.1.0)
Installing builder (3.2.2)
Installing erubis (2.7.0)
Installing actionview (4.1.0)
Installing rack (1.5.2)
Installing rack-test (0.6.2)
Installing actionpack (4.1.0)
Installing mime-types (1.25.1)
Installing polyglot (0.3.5)
Installing treetop (1.4.15)
Installing mail (2.5.4)
Installing actionmailer (4.1.0)
Installing activemodel (4.1.0)
Installing arel (5.0.1.20140414130214)
Installing activerecord (4.1.0)
Installing coffee-script-source (1.7.0)
Installing execjs (2.2.0)
Installing coffee-script (2.2.0)
Installing thor (0.19.1)
Installing railties (4.1.0)
Installing coffee-rails (4.0.1)
Installing hike (1.2.3)
Installing multi_json (1.10.1)
Installing jbuilder (2.1.1)
Installing jquery-rails (3.1.0)
Using bundler (1.5.3)
Installing tilt (1.4.1)
Installing sprockets (2.11.0)
Installing sprockets-rails (2.1.3)
Installing rails (4.1.0)
Installing rdoc (4.1.1)
Installing sass (3.2.19)
Installing sass-rails (4.0.3)
Installing sdoc (0.4.0)
Installing spring (1.1.3)
Installing turbolinks (2.2.2)
Installing uglifier (2.5.1)
Your bundle is complete!
It was installed into ./vendor/bundle
Post-install message from rdoc:
Depending on your version of ruby, you may need to install ruby rdoc/ri data:

 
<= 1.8.6 : unsupported
 = 1.8.7 : gem install rdoc-data; rdoc-data --install
 = 1.9.1 : gem install rdoc-data; rdoc-data --install
>= 1.9.2 : nothing to do! Yay!

serverを起動するとエラーになった。
# bundle exec rails s
/usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/execjs-2.2.0/lib/execjs/runtimes.rb:51:in `autodetect': Could not find a JavaScript runtime. See https://github.com/sstephenson/execjs for a list of available runtimes. (ExecJS::RuntimeUnavailable)
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/execjs-2.2.0/lib/execjs.rb:5:in `'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/execjs-2.2.0/lib/execjs.rb:4:in `'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/uglifier-2.5.1/lib/uglifier.rb:3:in `require'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/uglifier-2.5.1/lib/uglifier.rb:3:in `'
 from /opt/infra/ruby/lib64/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:76:in `require'
 from /opt/infra/ruby/lib64/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:76:in `block (2 levels) in require'
 from /opt/infra/ruby/lib64/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:72:in `each'
 from /opt/infra/ruby/lib64/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:72:in `block in require'
 from /opt/infra/ruby/lib64/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:61:in `each'
 from /opt/infra/ruby/lib64/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler/runtime.rb:61:in `require'
 from /opt/infra/ruby/lib64/ruby/gems/2.1.0/gems/bundler-1.5.3/lib/bundler.rb:131:in `require'
 from /usr/local/app/app/script/simulator/config/application.rb:14:in `'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/railties-4.1.0/lib/rails/commands/commands_tasks.rb:79:in `require'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/railties-4.1.0/lib/rails/commands/commands_tasks.rb:79:in `block in server'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/railties-4.1.0/lib/rails/commands/commands_tasks.rb:76:in `tap'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/railties-4.1.0/lib/rails/commands/commands_tasks.rb:76:in `server'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/railties-4.1.0/lib/rails/commands/commands_tasks.rb:40:in `run_command!'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/railties-4.1.0/lib/rails/commands.rb:17:in `'
 from /usr/local/app/app/script/simulator/bin/rails:8:in `require'
 from /usr/local/app/app/script/simulator/bin/rails:8:in `'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/spring-1.1.3/lib/spring/client/rails.rb:27:in `load'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/spring-1.1.3/lib/spring/client/rails.rb:27:in `call'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/spring-1.1.3/lib/spring/client/command.rb:7:in `call'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/spring-1.1.3/lib/spring/client.rb:26:in `run'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/spring-1.1.3/bin/spring:48:in `'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/spring-1.1.3/lib/spring/binstub.rb:11:in `load'
 from /usr/local/app/app/script/simulator/vendor/bundle/ruby/2.1.0/gems/spring-1.1.3/lib/spring/binstub.rb:11:in `'
 from /usr/local/app/app/script/simulator/bin/spring:16:in `require'
 from /usr/local/app/app/script/simulator/bin/spring:16:in `'
 from bin/rails:3:in `load'
 from bin/rails:3:in `
'

Could not find a JavaScript runtimeのエラーが出てるようなのでJSのモジュールがあるか確認したところなさそう。
ls vendor/bundle/ruby/2.1.0/gems/
actionmailer-4.1.0   arel-5.0.1.20140414130214   execjs-2.2.0        mime-types-1.25.1  rails-4.1.0       sdoc-0.4.0             tilt-1.4.1
actionpack-4.1.0     builder-3.2.2               hike-1.2.3          minitest-5.3.4     railties-4.1.0    spring-1.1.3           treetop-1.4.15
actionview-4.1.0     coffee-rails-4.0.1          i18n-0.6.9          multi_json-1.10.1  rake-10.3.2       sprockets-2.11.0       turbolinks-2.2.2
activemodel-4.1.0    coffee-script-2.2.0         jbuilder-2.1.1      polyglot-0.3.5     rdoc-4.1.1        sprockets-rails-2.1.3  tzinfo-1.2.1
activerecord-4.1.0   coffee-script-source-1.7.0  jquery-rails-3.1.0  rack-1.5.2         sass-3.2.19       thor-0.19.1            uglifier-2.5.1
activesupport-4.1.0  erubis-2.7.0                mail-2.5.4          rack-test-0.6.2    sass-rails-4.0.3 

Gemfileに以下を追加
gem 'therubyracer'
再度インストールを実施。無事に入ったげ。
# bundle install --path vendor/bundle
Fetching gem metadata from https://rubygems.org/...........
Fetching additional metadata from https://rubygems.org/..
Resolving dependencies...
Using rake (10.3.2)
Using i18n (0.6.9)
Using json (1.8.1)
Using minitest (5.3.4)
Using thread_safe (0.3.4)
Using tzinfo (1.2.1)
Using activesupport (4.1.0)
Using builder (3.2.2)
Using erubis (2.7.0)
Using actionview (4.1.0)
Using rack (1.5.2)
Using rack-test (0.6.2)
Using actionpack (4.1.0)
Using mime-types (1.25.1)
Using polyglot (0.3.5)
Using treetop (1.4.15)
Using mail (2.5.4)
Using actionmailer (4.1.0)
Using activemodel (4.1.0)
Using arel (5.0.1.20140414130214)
Using activerecord (4.1.0)
Using bundler (1.5.3)
Using coffee-script-source (1.7.0)
Using execjs (2.2.0)
Using coffee-script (2.2.0)
Using thor (0.19.1)
Using railties (4.1.0)
Using coffee-rails (4.0.1)
Using hike (1.2.3)
Using multi_json (1.10.1)
Using jbuilder (2.1.1)
Using jquery-rails (3.1.0)
Installing libv8 (3.16.14.3)
Using tilt (1.4.1)
Using sprockets (2.11.0)
Using sprockets-rails (2.1.3)
Using rails (4.1.0)
Using rdoc (4.1.1)
Installing ref (1.0.5)
Using sass (3.2.19)
Using sass-rails (4.0.3)
Using sdoc (0.4.0)
Using spring (1.1.3)
Installing therubyracer (0.12.1)
Using turbolinks (2.2.2)
Using uglifier (2.5.1)
Your bundle is complete!
It was installed into ./vendor/bundle

再度サーバーを起動。無事に起動した。
# bundle exec rails s
=> Booting WEBrick
=> Rails 4.1.0 application starting in development on http://0.0.0.0:3000
=> Run `rails server -h` for more startup options
=> Notice: server is listening on all interfaces (0.0.0.0). Consider using 127.0.0.1 (--binding option)
=> Ctrl-C to shutdown server
[2014-06-19 10:27:39] INFO  WEBrick 1.3.1
[2014-06-19 10:27:39] INFO  ruby 2.1.0 (2013-12-25) [x86_64-linux-gnu]
[2014-06-19 10:27:39] INFO  WEBrick::HTTPServer#start: pid=29686 port=3000


参考URL
http://mymemo.weby117.com/develop/ruby-geminstall-error_3.html

2014年6月12日木曜日

メモリが足りなくてchefのエラーではまった

chefのエラーではまったので書いておきます。

# knife solo cook dev1
Running Chef on dev1...
Checking Chef version...
Uploading the kitchen...
Generating solo config...
Running Chef...
Starting Chef Client, version 11.10.0
[2014-06-11T11:29:18+09:00] WARN: unable to detect ipaddress
[2014-06-11T11:29:18+09:00] WARN: unable to detect macaddress
[2014-06-11T11:29:18+09:00] WARN: unable to detect ip6address
[2014-06-11T11:29:18+09:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 0.524407356 seconds
[2014-06-11T11:29:18+09:00] ERROR: Unable to determine node name: configure node_name or configure the system's hostname and fqdn
[2014-06-11T11:29:18+09:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
ERROR: RuntimeError: chef-solo failed. See output above.

ネットで調べたらnode_nameを設定すればよいとかいくつか出てきたけど、
いろいろやってみてもどうもそれではなさそうでした。
なので、別のマシンに同じレシピを流してみたところちゃんと流れていったので、
違いを調べてみたところサーバーの空きメモリの量が原因のようでした。
メモリを大量に喰ってるプロセスを停止させてからもう一度レシピを流すとちゃんと流れました。
メモリが足りないって内容のエラーが出ないってのが判断が難しいですね。

JJUG ナイトセミナー 「6.11 ドメイン駆動設計特集! 」の勉強会に行ってきました

勉強会に行くのは数年ぶりだったのですが、
久々なので実装メインよりは設計とか思想メイン的な勉強会をってことで、
JJUG ナイトセミナー 「6.11 ドメイン駆動設計特集! 」@グリーに行ってきました。
詳しいことはもうブログ書かれてる方がいたので、そちらを見ていただいた方がよさそうです。
よしだのブログ(http://blog.yoslab.com/entry/2014/06/11/201103)

1.コードに語らせるために
和智 右桂氏 (グロースエクスパートナーズ株式会社)

DDDとはDomain Driven Designのこと。
Domain=業務と考えてよい。
ユビキタス言語とは図やドキュメント、会話の中で同一の言語を用いること。
チーム内のすべてのコミュニケーションとコードにおいて、言語を厳格に用いるのが大事。
会話ではお客さん、設計書では顧客、コードでは客、みたいな別々の言葉にするのではなく、統一するのがよい。
「エリック・エヴァンスのドメイン駆動設計」をまだ読んでない方は、
第一部と第三部を先に読むのがよいそうです。
第二部は具体的すぎるのでついていけないかもしれないと。


2.DDDで実践する時に役に立つ話し
加藤 潤一氏 (グリー株式会社)

GREEチャット(限定公開)でDDDを実践。
メンバーは8人。
ユビキタス言語と実装をプロトタイプで説明。
DDDを浸透させていくためにほぼ毎日読書会を実施。
言語を合わせていくために議論をかなり重ねたらしい。
大規模サービスでよくやるシャーディングなどとは相性があまりよくない。
テーブル設計はするが、通常のやり方とは視点が違うので、
ドメインモデルをまず決めて、それをテーブルの形式にするにはどうエンコードしたらよいかを考えていくような形になる。


個人的に気になったところだけを書き出してみました。
何度も重ねて話されてたのは、関係者の言語を合わせていくことが大事ということでした。
これまでDDDでの開発をやったことはないし、
実は「エリック・エヴァンスのドメイン駆動設計」読んだことがないので、
今度読んでみようと思います。
久々の勉強会としてはかなり有意義な時間が過ごせたかなと。

2014年6月6日金曜日

Macにpgadminをhomebrewで入れる

ちょっとpostgresqlにつなぐ必要ができたのですが、
あまりに何のDBかわからず怖すぎたのでpgadminⅢを入れてつないでみました。

Macにはhomebrewという便利なのがあるので、まずはあるか検索。
$ brew search pgadmin
homebrew/php/phppgadmin
普通に検索するとphpのやつが引っかかります。
なのでvirtualboxとかで有名なcaskでも検索。
$ brew cask search pgadmin
==> Partial matches
pgadmin3
引っかかりました。さすがですね。
早速インストールしてみます。
$ brew cask install pgadmin3
==> Downloading http://ftp.postgresql.org/pub/pgadmin3/release/v1.18.1/osx/pgadmin3-1.18.1.dmg
######################################################################## 100.0%
==> Symlinking App 'pgAdmin3.app' to '/Users/xxxx/Applications/pgAdmin3.app'
🍺  pgadmin3 installed to '/opt/homebrew-cask/Caskroom/pgadmin3/1.18.1' (412 files, 65M)
完了すると、書いてあるようにユーザーのアプリケーションフォルダにappファイルができます。
そこから実行してみると、ちゃんと起動しました。
homebrew便利ですね。

2014年6月3日火曜日

macのchromeでjavaアプレット動かす

HPのサーバーをiLOから操作することになったので、
http://h50146.www5.hp.com/lib/products/servers/proliant/manuals/684918-191_ja.pdf
のマニュアルを参考にいろいろと試した。
やってみたところ、リモートコンソールが動かない
手元の環境がMacOS X 10.9なので、まず.NetFrameworkは動かず。。
monoというのがあるらしく、それをまず入れてみる。
$ brew search mono
libmonome  mono       monotone
homebrew/binary/monoberry
homebrewで入れられそうだ。
$ brew install mono
==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/mono-3.4.0.mavericks.bottle.tar.gz
######################################################################## 100.0%
==> Pouring mono-3.4.0.mavericks.bottle.tar.gz
==> Caveats
To use the assemblies from other formulae you need to set:
  export MONO_GAC_PREFIX="/usr/local"
==> Summary
🍺  /usr/local/Cellar/mono/3.4.0: 1128 files, 170M
そして、ブラウザ再起動して表示してみるも、なんら表示が変わらず、
多分だめなんだろうなとあきらめて次を探してみる。

使ってるブラウザがchromeなので、java7だとアプレットが動かない。
java6をインストールしてみる。
http://support.apple.com/ja_JP/downloads/#java
java6が有効になるように設定
$ sudo mkdir -p /Library/Internet\ Plug-Ins/disabled 
$ sudo mv /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin /Library/Internet\ Plug-Ins/disabled
$ sudo ln -sf /System/Library/Java/Support/Deploy.bundle/Contents/Resources/JavaPlugin2_NPAPI.plugin /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin 
$ sudo ln -sf /System/Library/Frameworks/JavaVM.framework/Commands/javaws /usr/bin/javaws
これでリモートコンソール動くようになった。
ちなみに、後から聞いた話ではSafariでやればこんな手順踏まずに済むそうです。。。


参考URL
http://h50146.www5.hp.com/lib/products/servers/proliant/manuals/684918-191_ja.pdf
http://support.apple.com/ja_JP/downloads/#java
http://support.apple.com/kb/HT5559?viewlocale=ja_JP&locale=ja_JP
http://d.hatena.ne.jp/paraches/20131025/1382672759