华为云云耀云服务器L实例评测｜ClickHouse部署及压测

前言

在上期对文章中，带大家通过华为云云耀云服务器L进行Docker的部署及应用，本次给大家首先介绍在使用华为云云耀云服务器L时，当您需要对帐号的安全信息进行设置时，可以通过"安全设置",进行相关操作，并对ClickHouse部署及压测。

---

往期回顾：
1.华为云云耀云服务器L实例评测｜Ubuntu云服务器申请使用
2.华为云云耀云服务器L实例评测｜Ubuntu系统MySQL 8.1.0 Innovation压测
3.华为云云耀云服务器L实例评测｜Docker部署及应用

📣 1.前言概述

在上期对文章中，带大家通过华为云云耀云服务器L进行Docker的部署及应用，容器是用镜像创建的运行实例。它可以被启动、开始、停止、删除。每个容器都是相互隔离的、保证安全的平台。可以把容器看做是一个简易版的 Linux 环境。本次给大家首先介绍在使用华为云云耀云服务器L时，当您需要对帐号的安全信息进行设置时，可以通过"安全设置",进行相关操作。

📣 2.安全设置

当您需要对帐号的安全信息进行设置时，可以通过“安全设置”进行操作。“安全设置”包括“基本信息”、“敏感操作”、“登录验证策略”、“密码策略”、“访问控制”。
如何进入安全设置?所有用户均可通过控制台入口进入“安全设置”,登录华为云，在右上角单击“控制台”,在“控制台”页面，鼠标移动至右上方的用户名，在下拉列表中选择“安全设置”。

📣 3.ClickHouse安装

clickhouse简介：
ClickHouse 是俄罗斯的 Yandex 于 2016 年开源的用于在线分析处理查询（OLAP :Online Analytical Processing）MPP架构的列式存储数据库（DBMS：Database Management System），能够使用 SQL 查询实时生成分析数据报告。ClickHouse的全称是Click Stream，Data WareHouse。
clickhouse可以做用户行为分析，流批一体
线性扩展和可靠性保障能够原生支持 shard + replication
clickhouse没有走hadoop生态，采用 Local attached storage 作为存储。

? 3.1 申请服务器

##登录华为云云耀云服务器L实例，通过Docker容器拉取镜像，创建容器，拥有自己的云服务器Centos7.8
1)拉取镜像
docker pull daocloud.io/library/centos:centos7.8.2003
root@hcss-ecs-7c99:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest 9c7a54a9a43c 5 months ago 13.3kB
daocloud.io/library/centos centos7.8.2003 afb6fca791e0 3 years ago 203MB
2）创建容器
docker run -d --name centos7.8 -h centos7.8
-p 220:22 -p 3387:3389
–privileged=true
daocloud.io/library/centos:centos7.8.2003 /usr/sbin/init
3)进入容器
root@hcss-ecs-7c99:~# docker exec -it centos7.8 bash
[root@centos7 /]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
[root@centos7 /]# uname -r
5.15.0-60-generic
[root@centos7 /]# uname -a
Linux centos7.8 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

? 3.2 安装前准备

1)指令集检查
ClickHouse可以在任何具有x86_64，AArch64或PowerPC64LE CPU架构的Linux，FreeBSD或Mac OS X上运行。
官方预构建的二进制文件通常针对x86_64进行编译，并利用SSE 4.2指令集。
检查当前CPU是否支持SSE 4.2的命令:
[root@centos7 /]# grep -q sse4_2 /proc/cpuinfo && echo “SSE 4.2 supported” || echo “SSE 4.2 not supported”
SSE 4.2 supported
2）文件数限制修改
调整CentOS系统对打开文件数的限制，在/etc/security/limits.conf和/etc/security/limits.d/20-nproc.conf文件的末尾加上
soft nofile 65536
hard nofile 65536
soft nproc 131072
hard nproc 131072
cat >> /etc/security/limits.conf <<EOF
soft nofile 65536
hard nofile 65536
soft nproc 131072
hard nproc 131072
EOF
cat >> /etc/security/limits.d/20-nproc.conf <<EOF
soft nofile 65536
hard nofile 65536
soft nproc 131072
hard nproc 131072
EOF
3） SELinux 设置
vi /etc/selinux/config
#修改
SELINUX=disabled

? 3.3 RPM安装包

推荐使用CentOS、RedHat和所有其他基于rpm的Linux发行版的官方预编译rpm包。
##安装依赖
yum install -y yum-utils
yum install -y libtool
yum install -y unixODBC
##需要添加官方存储库：
yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo
##安装
yum install -y clickhouse-server clickhouse-client

? 3.4 配置文件

修改配置文件，把 <listen_host>::</listen_host> 的注释打开，这样的话才能让ClickHouse被除本机以外的服务器访问
vi /etc/clickhouse-server/config.xml
cat >> /etc/clickhouse-server/config.xml <<EOF
<listen_host>::</listen_host>
EOF
在这个文件中，有ClickHouse的一些默认路径配置，比较重要的
数据文件路径： /var/lib/clickhouse/
日志文件路径：/var/log/clickhouse-server/clickhouse-server.log

? 3.5 使用ClickHouse

sudo /etc/init.d/clickhouse-server start
clickhouse-client # or "clickhouse-client --password" if you set up a password.

[root@centos7 /]# /etc/init.d/clickhouse-server start
 chown -R clickhouse: '/var/run/clickhouse-server/'
Will run sudo --preserve-env -u 'clickhouse' /usr/bin/clickhouse-server --config-file /etc/clickhouse-server/config.xml --pid-file /var/run/clickhouse-server/clickhouse-server.pid --daemon
/bin/sh: sudo: command not found
Code: 302. DB::Exception: Child process was exited with return code 127. (CHILD_WAS_NOT_EXITED_NORMALLY) (version 23.9.1.1854 (official build))

此处解决的办法是：
[root@centos7 /]# yum insatll sudo

sudo /etc/init.d/clickhouse-server start
/etc/init.d/clickhouse-server status

##客户端登录
[root@centos7 /]# clickhouse-client
ClickHouse client version 23.9.1.1854 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 23.9.1 revision 54466.

Warnings:
 * Linux threads max count is too low. Check /proc/sys/kernel/threads-max
 * Available memory at server startup is too low (2GiB).

centos7.8 :) 

centos7.8 :) show databases;

SHOW DATABASES

Query id: 24cfdcc2-4e5a-46d1-922d-135cf67eb143

┌─name───────────────┐
│ INFORMATION_SCHEMA │
│ default            │
│ information_schema │
│ system             │
└────────────────────┘

4 rows in set. Elapsed: 0.001 sec. 


##比较常用的完整命令
clickhouse-client -u root --password 123456 --port 9001 -h 127.0.0.1

📣 4.ClickHouse压测

? 4.1 下载数据

[root@centos7 /]# yum install -y wget
[root@centos7 /]# wget https://datasets.clickhouse.com/mgbench{1…3}.csv.xz

? 4.2 解压数据

[root@centos7 /]# xz -v -d mgbench{1…3}.csv.xz
mgbench1.csv.xz (1/3)
100 % 543.0 MiB / 7206.5 MiB = 0.075 98 MiB/s 1:13
mgbench2.csv.xz (2/3)
100 % 382.7 MiB / 5808.7 MiB = 0.066 99 MiB/s 0:58
mgbench3.csv.xz (3/3)
100 % 485.0 MiB / 7881.7 MiB = 0.062 100 MiB/s 1:19

? 4.3 创建数据库和表

[root@centos7 /]# clickhouse-client
centos7.8 :) CREATE DATABASE mgbench; 
centos7.8 :) USE mgbench;

CREATE TABLE mgbench.logs1 (
  log_time      DateTime,
  machine_name  LowCardinality(String),
  machine_group LowCardinality(String),
  cpu_idle      Nullable(Float32),
  cpu_nice      Nullable(Float32),
  cpu_system    Nullable(Float32),
  cpu_user      Nullable(Float32),
  cpu_wio       Nullable(Float32),
  disk_free     Nullable(Float32),
  disk_total    Nullable(Float32),
  part_max_used Nullable(Float32),
  load_fifteen  Nullable(Float32),
  load_five     Nullable(Float32),
  load_one      Nullable(Float32),
  mem_buffers   Nullable(Float32),
  mem_cached    Nullable(Float32),
  mem_free      Nullable(Float32),
  mem_shared    Nullable(Float32),
  swap_free     Nullable(Float32),
  bytes_in      Nullable(Float32),
  bytes_out     Nullable(Float32)
)
ENGINE = MergeTree()
ORDER BY (machine_group, machine_name, log_time);


CREATE TABLE mgbench.logs2 (
  log_time    DateTime,
  client_ip   IPv4,
  request     String,
  status_code UInt16,
  object_size UInt64
)
ENGINE = MergeTree()
ORDER BY log_time;

CREATE TABLE mgbench.logs3 (
  log_time     DateTime64,
  device_id    FixedString(15),
  device_name  LowCardinality(String),
  device_type  LowCardinality(String),
  device_floor UInt8,
  event_type   LowCardinality(String),
  event_unit   FixedString(1),
  event_value  Nullable(Float32)
)
ENGINE = MergeTree()
ORDER BY (event_type, log_time);

? 4.4 插入数据

clickhouse-client --query “INSERT INTO mgbench.logs1 FORMAT CSVWithNames” < mgbench1.csv
clickhouse-client --query “INSERT INTO mgbench.logs2 FORMAT CSVWithNames” < mgbench2.csv
clickhouse-client --query “INSERT INTO mgbench.logs3 FORMAT CSVWithNames” < mgbench3.csv

? 4.5 插入数据

Q1.1: 自午夜以来每个 Web 服务器的 CPU/网络利用率是多少？
USE mgbench;

SELECT machine_name,
       MIN(cpu) AS cpu_min,
       MAX(cpu) AS cpu_max,
       AVG(cpu) AS cpu_avg,
       MIN(net_in) AS net_in_min,
       MAX(net_in) AS net_in_max,
       AVG(net_in) AS net_in_avg,
       MIN(net_out) AS net_out_min,
       MAX(net_out) AS net_out_max,
       AVG(net_out) AS net_out_avg
FROM (
  SELECT machine_name,
         COALESCE(cpu_user, 0.0) AS cpu,
         COALESCE(bytes_in, 0.0) AS net_in,
         COALESCE(bytes_out, 0.0) AS net_out
  FROM logs1
  WHERE machine_name IN ('anansi','aragog','urd')
    AND log_time >= TIMESTAMP '2017-01-11 00:00:00'
) AS r
GROUP BY machine_name;

-- Q2：过去一个月顶级请求的平均路径深度是多少？
SELECT top_level,
       AVG(LENGTH(request) - LENGTH(REPLACE(request, '/', ''))) AS depth_avg
FROM (
  SELECT SUBSTRING(request FROM 1 FOR len) AS top_level,
         request
  FROM (
    SELECT POSITION(SUBSTRING(request FROM 2), '/') AS len,
           request
    FROM logs2
    WHERE status_code >= 200
      AND status_code < 300
      AND log_time >= TIMESTAMP '2012-12-01 00:00:00'
  ) AS r
  WHERE len > 0
) AS s
WHERE top_level IN ('/about','/courses','/degrees','/events',
                    '/grad','/industry','/news','/people',
                    '/publications','/research','/teaching','/ugrad')
GROUP BY top_level
ORDER BY top_level;

-- Q3：对于每种类别的设备，每月的功耗指标是什么？

SELECT yr,
       mo,
       SUM(coffee_hourly_avg) AS coffee_monthly_sum,
       AVG(coffee_hourly_avg) AS coffee_monthly_avg,
       SUM(printer_hourly_avg) AS printer_monthly_sum,
       AVG(printer_hourly_avg) AS printer_monthly_avg,
       SUM(projector_hourly_avg) AS projector_monthly_sum,
       AVG(projector_hourly_avg) AS projector_monthly_avg,
       SUM(vending_hourly_avg) AS vending_monthly_sum,
       AVG(vending_hourly_avg) AS vending_monthly_avg
FROM (
  SELECT dt,
         yr,
         mo,
         hr,
         AVG(coffee) AS coffee_hourly_avg,
         AVG(printer) AS printer_hourly_avg,
         AVG(projector) AS projector_hourly_avg,
         AVG(vending) AS vending_hourly_avg
  FROM (
    SELECT CAST(log_time AS DATE) AS dt,
           EXTRACT(YEAR FROM log_time) AS yr,
           EXTRACT(MONTH FROM log_time) AS mo,
           EXTRACT(HOUR FROM log_time) AS hr,
           CASE WHEN device_name LIKE 'coffee%' THEN event_value END AS coffee,
           CASE WHEN device_name LIKE 'printer%' THEN event_value END AS printer,
           CASE WHEN device_name LIKE 'projector%' THEN event_value END AS projector,
           CASE WHEN device_name LIKE 'vending%' THEN event_value END AS vending
    FROM logs3
    WHERE device_type = 'meter'
  ) AS r
  GROUP BY dt,
           yr,
           mo,
           hr
) AS s
GROUP BY yr,
         mo
ORDER BY yr,
         mo;