Cloudera Manager安装streamsets

其他 2020-10-14 18:43:04 阅读次数: 0

文章目录


在这里插入图片描述

1, 下载streamsets的parcel文件

parcel下载地址： https://archives.streamsets.com/datacollector
parcel使用cloudera manager安装文档：https://streamsets.com/documentation/datacollector/3.16.x/help/datacollector/UserGuide/Installation/CMInstall-Overview.html#concept_nb5_c3m_25

  • Install the StreamSets custom service descriptor (CSD). (需要重启cm_server, cm监控服务)
  • (Optional.) Manually install the parcel and checksum files. Typically only needed when the Cloudera Manager Server does not have internet access.
  • Download, distribute, and activate the StreamSets parcel.

2, 使用cloudera manager安装parcel

a, 分发parcel报错

  • 问题1: host_monitor退出,
  • 问题2: parcel下载后解压失败被删除
  • 问题3: 分发parcel卡住
=================第一次：agent, 下载解压报错
[root@test-c6 ~]# find /opt/ -name "STREAM*"
/opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
/opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
/opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.sha
/opt/cloudera/csd/STREAMSETS-3.16.2.jar
/opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
/opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
/opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
/opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
/opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR
/opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR-3.16.2

[root@test-c6 parcel-repo]# tailf /opt/cm-5.12.2/log/cloudera-scm-agent/cloudera-scm-agent.log
[11/Oct/2020 01:36:34 +0000] 28799 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=5611142ize=5712330086 upload=0 state=downloading seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/crcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=982286]
[11/Oct/2020 01:36:36 +0000] 28799 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=5712330ize=5712330086 upload=0 state=seeding seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/clouds/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
[11/Oct/2020 01:36:36 +0000] 28799 Thread-13 downloader   INFO     Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLE2-el6.parcel code=200 state=downloaded
[11/Oct/2020 01:36:36 +0000] 28799 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
[11/Oct/2020 01:36:36 +0000] 28799 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/_DATACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
[11/Oct/2020 01:37:08 +0000] 28799 MainThread agent        INFO     PID '30453' associated with process '124-cloudera-mgmt-SERVICEMONITOR' with paylosname:124-cloudera-mgmt-SERVICEMONITOR groupname:124-cloudera-mgmt-SERVICEMONITOR from_state:RUNNING expected:0 pid:30453' exited unexpectedly
[11/Oct/2020 01:37:34 +0000] 28799 Monitor-GenericMonitor throttling_logger ERROR    Error fetching metrics at 'http://test-c6.com:8091/jmx'
Traceback (most recent call last):
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/generic/metric_collectors.py", line 200ect_and_parse_and_return
    self._adapter.safety_valve))
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/url_util.py", line 207, in urlopen_with_retry_ocation_errors
    return function()
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/generic/metric_collectors.py", line 217_url
    password=self._password_value)
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/url_util.py", line 70, in urlopen_with_timeout
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 104] Connection reset by peer>
[11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR    Failed to read file descriptor max for process 30381: [Errno 2]ile or directory: '/proc/30381/limits'
[11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR    Failed to get file descriptor count for process 30381: [Errno 2file or directory: '/proc/30381/fd/'
[11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR    Failed to get process metrics 30381: no process found with pid
[11/Oct/2020 01:37:36 +0000] 28799 MonitorDaemon-Reporter throttling_logger ERROR    Error sending messages to firehose: mgmt-SERVICEMONITOR-af401409538410569c49e2
Traceback (most recent call last):
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/firehose.py", line 120, in _send
    self._requestor.request('sendAgentMessages', dict(messages=messages))
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 482, in transceive
    self.write_framed_message(request)
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 501, in write_framed_message
    self.conn.request(req_method, self.req_resource, req_body, req_headers)
  File "/usr/lib64/python2.6/httplib.py", line 979, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.6/httplib.py", line 1010, in _send_request
    self.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 967, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 831, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 790, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 771, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
error: [Errno 111] Connection refused
[11/Oct/2020 01:37:36 +0000] 28799 MonitorDaemon-Reporter throttling_logger ERROR    Error sending messages to firehose: mgmt-HOSTMONITOR-af401409b8a410569c49e2
Traceback (most recent call last):
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/firehose.py", line 120, in _send
    self._requestor.request('sendAgentMessages', dict(messages=messages))
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 482, in transceive
    self.write_framed_message(request)
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 501, in write_framed_message
    self.conn.request(req_method, self.req_resource, req_body, req_headers)
  File "/usr/lib64/python2.6/httplib.py", line 979, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.6/httplib.py", line 1010, in _send_request
    self.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 967, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 831, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 790, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 771, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
error: [Errno 111] Connection refused
[11/Oct/2020 01:37:36 +0000] 28799 MainThread agent        INFO     PID '30381' associated with process '121-cloudera-mgmt-HOSTMONITOR' with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30381' exited unexpectedly
[11/Oct/2020 01:38:03 +0000] 28799 MainThread agent        INFO     PID '30243' associated with process '121-cloudera-mgmt-HOSTMONITOR' with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30243' exited unexpectedly
[11/Oct/2020 01:38:07 +0000] 28799 MainThread agent        INFO     PID '30316' associated with process '124-cloudera-mgmt-SERVICEMONITOR' with paylosname:124-cloudera-mgmt-SERVICEMONITOR groupname:124-cloudera-mgmt-SERVICEMONITOR from_state:RUNNING expected:0 pid:30316' exited unexpectedly
[11/Oct/2020 01:38:28 +0000] 28799 MainThread agent        INFO     PID '30416' associated with process '121-cloudera-mgmt-HOSTMONITOR' with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30416' exited unexpectedly

报错后，下载好的parcel被删除

#=================之后, /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel文件被删除, 并开始报下面的错
[root@test-c6 parcel-repo]# find /opt/ -name "STREAM*"
/opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
/opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
/opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.sha
/opt/cloudera/csd/STREAMSETS-3.16.2.jar
/opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
/opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
/opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent

[root@test-c6 parcel-repo]# tailf /opt/cm-5.12.2/log/cloudera-scm-agent/cloudera-scm-agent.log
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Fetching torrent: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-.parcel.torrent
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Starting download of: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLEC-el6.parcel
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Failed adding torrent: file:///opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3parcel.torrent Already present torrent: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=57123300ze=5712330086 upload=0 state=seeding seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/cloude/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLEC-el6.parcel code=200 state=downloaded
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/SDATACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   ERROR    Failed op: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-elTREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel does not exist
Traceback (most recent call last):
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/downloader.py", line 501, in callable
    callback(url, curr_op)
  File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/parcel_cache.py", line 203, in cb
    raise e
Exception: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel does not exist
[11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Finished download [ url: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOL6.2-el6.parcel, state: exception, total_bytes: 5712330086, downloaded_bytes: 5712330086, start_time: 2020-10-11 01:59:47, download_end_time: 2020-10-7, end_time: 2020-10-11 01:59:47, code: 601, exception_msg: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSELECTOR-3.16.2-el6.parcel does not exist, path: /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.el ]

b, 错误解决

  • 第一步: host_monitor,service_monitor 配置修改: jvm内存=1g, 文件描述符数=65536
  • 第二步: 系统参数修改 /etc/security/limits.d/的nofile值=65536, 重启sshd
  • 第三步: 重启监控服务, 重启agent （触发再次下载parcel)
#=================第二次：agent重启(修改host_monitor,service_monitor 的jvm参数1g,
				    Cloudera Management Service的最大进程文件描述符数=65536, 
				    和/etc/security/limits.d/的nofile值=65536, 重启sshd, 
				    重启监控服务, 重启agent) 
			下载解压正常
			
[11/Oct/2020 04:20:03 +0000] 18179 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=56169095712330086 upload=0 state=downloading seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/cloudlood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=983295]
[11/Oct/2020 04:20:05 +0000] 18179 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=57123305712330086 upload=0 state=seeding seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/cloudera//STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
[11/Oct/2020 04:20:05 +0000] 18179 Thread-13 downloader   INFO     Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLE6.parcel code=200 state=downloaded
[11/Oct/2020 04:20:05 +0000] 18179 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
[11/Oct/2020 04:20:05 +0000] 18179 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/ACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
[11/Oct/2020 04:21:43 +0000] 18179 MainThread agent        WARNING  Long HB processing time: 6.20692515373

[11/Oct/2020 04:22:20 +0000] 18179 Thread-13 parcel_cache INFO     Unpack of parcel /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.ETS_DATACOLLECTOR-3.16.2-el6.parcel successful
[11/Oct/2020 04:22:20 +0000] 18179 Thread-13 downloader   INFO     Finished download [ url: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACO-el6.parcel, state: complete, total_bytes: 5712330086, downloaded_bytes: 5712330086, start_time: 2020-10-11 04:15:14, download_end_time: 2020-10-11 0ime: 2020-10-11 04:22:20, code: 200, exception_msg: None, path: /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DA16.2-el6.parcel ]
[11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.12.0-1.cdh5.12.0.p0.29
[11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Loading parcel manifest for: STREAMSETS_DATACOLLECTOR-3.16.2
[11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Ensuring users/groups exist for new parcel STREAMSETS_DATACOLLECTOR-3.16.2.
[11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'sdc']
[11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'sdc']
[11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'sdc', '-K', 'UMASK=022/var/lib/sdc', '--comment', 'StreamSets Data Collector', '--shell', '/sbin/nologin', 'sdc']
[11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Ensuring correct file permissions for new parcel STREAMSETS_DATACOLLECTOR-3.16.2.
[11/Oct/2020 04:25:22 +0000] 18179 MainThread heartbeat_tracker INFO     HB stats (seconds): num:43 LIFE_MIN:0.08 min:0.02 mean:0.22 max:1.12 LIFE_MA
[11/Oct/2020 04:30:52 +0000] 18179 MainThread process      INFO     [120-hbase-MASTER] Updating process.
....

猜你喜欢

转载自blog.csdn.net/eyeofeagle/article/details/109051068
今日推荐
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
【转】spring中对控制反转和依赖注入的理解
tms webcore 安装和使用
java程序员进阶相关书籍
SpringMVC接受请求参数、
如何保存训练好的机器学习模型
MyEclipse、Eclipse设置项目JDK的三个地方
商超行业微信小程序开发定制一般多少钱 （行业技术人员解读）
Markdown编辑器语言——30分钟入门到到精通
Linux系统下MongoDB的简单安装与基本操作
Power Strings
每日归档
更多
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022