0. 起因

在我们测试环境k8s上运行了数十个helm release, 昨天helm ls的时候居然报错了

$ helm lsError: trying to send message larger than max (23934705 vs. 20971520)

这个问题搜索下, 发现github issue今年一月就已经有人遇到了 , 今年二月份官方的同学也提pr修复了

为啥我这里还会出现呢

1. 探究

看下修复pr的主要内容, 发现只是把限制从4M提升到20M...

这个值还是一个常量, 目前最近的release和master也一样我们使用的tiller镜像 registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1，已经包了修复内容,看来想解决只能自己修改源码编译了

2. 修复

2.1 修改源码

const maxMsgSize = 1024 * 1024 * 100

这里改成100M,还能超的话只能说要再来一个集群了..

2.2 编译 Helm

拉下代码, 发现helm使用make进行构建, 按照官方尝试初始化和构建

17:03 $ make bootstrap buildglide install --strip-vendor[INFO]  Downloading dependencies. Please wait...[INFO]  --> Found desired version locally github.com/aokoli/goutils 9c37978a95bd5c709a15883b6242714ea6709e64![INFO]  --> Found desired version locally github.com/asaskevich/govalidator 7664702784775e51966f0885f5cd27435916517b![INFO]  --> Found desired version locally github.com/Azure/go-ansiterm d6e3b3328b783f23731bc4d058875b0371ff8109![INFO]  --> Found desired version locally github.com/Azure/go-autorest bca49d5b51a50dc5bb17bbf6204c711c6dbded06![INFO]  --> Found desired version locally github.com/beorn7/perks 3ac7bf7a47d159a033b107610db8a1b6575507a4![INFO]  --> Found desired version locally github.com/BurntSushi/toml 3012a1dbe2e4bd1391d42b32f0577cb7bbc7f005![INFO]  --> Found desired version locally github.com/chai2010/gettext-go c6fed771bfd517099caf0f7a961671fa8ed08723![INFO]  --> Found desired version locally github.com/cpuguy83/go-md2man 71acacd42f85e5e82f70a55327789582a5200a90![INFO]  --> Found desired version locally github.com/cyphar/filepath-securejoin a261ee33d7a517f054effbf451841abaafe3e0fd![INFO]  --> Found desired version locally github.com/davecgh/go-spew 782f4967f2dc4564575ca782fe2d04090b5faca8![INFO]  --> Found desired version locally github.com/dgrijalva/jwt-go 01aeca54ebda6e0fbfafd0a524d234159c05ec20![INFO]  --> Found desired version locally github.com/docker/distribution edc3ab29cdff8694dd6feb85cfeb4b5f1b38ed9c![INFO]  --> Found desired version locally github.com/docker/docker a9fbbdc8dd8794b20af358382ab780559bca589d![INFO]  --> Found desired version locally github.com/docker/go-connections 3ede32e2033de7505e6500d6c868c2b9ed9f169d![INFO]  --> Found desired version locally github.com/docker/go-units 9e638d38cf6977a37a8ea0078f3ee75a7cdb2dd1![INFO]  --> Found desired version locally github.com/docker/spdystream 449fdfce4d962303d702fec724ef0ad181c92528![INFO]  --> Found desired version locally github.com/evanphx/json-patch 36442dbdb585210f8d5a1b45e67aa323c197d5c4![INFO]  --> Found desired version locally github.com/exponent-io/jsonpath d6023ce2651d8eafb5c75bb0c7167536102ec9f5![INFO]  --> Found desired version locally github.com/fatih/camelcase f6a740d52f961c60348ebb109adde9f4635d7540![INFO]  --> Found desired version locally github.com/ghodss/yaml 73d445a93680fa1a78ae23a5839bad48f32ba1ee![INFO]  --> Found desired version locally github.com/go-openapi/jsonpointer 46af16f9f7b149af66e5d1bd010e3574dc06de98![INFO]  --> Found desired version locally github.com/go-openapi/jsonreference 13c6e3589ad90f49bd3e3bbe2c2cb3d7a4142272![INFO]  --> Found desired version locally github.com/go-openapi/spec 1de3e0542de65ad8d75452a595886fdd0befb363![INFO]  --> Found desired version locally github.com/go-openapi/swag f3f9494671f93fcff853e3c6e9e948b3eb71e590![INFO]  --> Found desired version locally github.com/gobwas/glob 5ccd90ef52e1e632236f7326478d4faa74f99438![INFO]  --> Found desired version locally github.com/gogo/protobuf c0656edd0d9eab7c66d1eb0c568f9039345796f7![INFO]  --> Found desired version locally github.com/golang/glog 44145f04b68cf362d9c4df2182967c2275eaefed![INFO]  --> Found desired version locally github.com/golang/groupcache 02826c3e79038b59d737d3b1c0a1d937f71a4433![INFO]  --> Found desired version locally github.com/golang/protobuf 1643683e1b54a9e88ad26d98f81400c8c9d9f4f9![INFO]  --> Found desired version locally github.com/google/btree 7d79101e329e5a3adf994758c578dab82b90c017![INFO]  --> Found desired version locally github.com/google/gofuzz 44d81051d367757e1c7c6a5a86423ece9afcf63c![INFO]  --> Found desired version locally github.com/google/uuid 064e2069ce9c359c118179501254f67d7d37ba24![INFO]  --> Found desired version locally github.com/googleapis/gnostic 0c5108395e2debce0d731cf0287ddf7242066aba![INFO]  --> Found desired version locally github.com/gophercloud/gophercloud 781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d![INFO]  --> Found desired version locally github.com/gosuri/uitable 36ee7e946282a3fb1cfecd476ddc9b35d8847e42![INFO]  --> Found desired version locally github.com/gregjones/httpcache 787624de3eb7bd915c329cba748687a3b22666a6![INFO]  --> Found desired version locally github.com/grpc-ecosystem/go-grpc-prometheus 0c1b191dbfe51efdabe3c14b9f6f3b96429e0722![INFO]  --> Found desired version locally github.com/hashicorp/golang-lru a0d98a5f288019575c6d1f4bb1573fef2d1fcdc4![INFO]  --> Found desired version locally github.com/huandu/xstrings 3959339b333561bf62a38b424fd41517c2c90f40![INFO]  --> Found desired version locally github.com/imdario/mergo 9316a62528ac99aaecb4e47eadd6dc8aa6533d58![INFO]  --> Found desired version locally github.com/inconshreveable/mousetrap 76626ae9c91c4f2a10f34cad8ce83ea42c93bb75![INFO]  --> Found desired version locally github.com/json-iterator/go f2b4162afba35581b6d4a50d3b8f34e33c144682![INFO]  --> Found desired version locally github.com/mailru/easyjson 2f5df55504ebc322e4d52d34df6a1f5b503bf26d![INFO]  --> Found desired version locally github.com/MakeNowJust/heredoc bb23615498cded5e105af4ce27de75b089cbe851![INFO]  --> Found desired version locally github.com/Masterminds/semver 517734cc7d6470c0d07130e40fd40bdeb9bcd3fd![INFO]  --> Found desired version locally github.com/Masterminds/sprig 15f9564e7e9cf0da02a48e0d25f12a7b83559aa6![WARN]  Unable to checkout golang.org/x/text[ERROR] Update failed for golang.org/x/text: Cannot detect VCS[WARN]  Unable to checkout golang.org/x/time[ERROR] Update failed for golang.org/x/time: Cannot detect VCS[WARN]  Unable to checkout google.golang.org/appengine[ERROR] Update failed for google.golang.org/appengine: Cannot detect VCS[WARN]  Unable to checkout google.golang.org/genproto[ERROR] Update failed for google.golang.org/genproto: Cannot detect VCS[WARN]  Unable to checkout google.golang.org/grpc[ERROR] Update failed for google.golang.org/grpc: Cannot detect VCS[ERROR] Failed to install: Cannot detect VCSCannot detect VCSCannot detect VCSCannot detect VCSCannot detect VCSCannot detect VCSCannot detect VCSCannot detect VCSCannot detect VCSCannot detect VCSmake: *** [bootstrap] Error 1

问题1: 国内golang的依赖包各种被墙

.PHONY: bootstrapbootstrap:ifndef HAS_GLIDE        go get -u github.com/Masterminds/glideendififndef HAS_GOX        go get -u github.com/mitchellh/goxendififndef HAS_GIT        $(error You must install Git)endif        glide install --strip-vendor        go build -o bin/protoc-gen-go ./vendor/github.com/golang/protobuf/protoc-gen-go

看一眼Makefile, 这里用的glide管理的依赖包

好在github基本都会有对应的包源码, 有几种解决方式:

clone源码然后按照目录结构放到对应$GOPATH/src下

给git设置proxy,小飞机你懂的~

使用

使用glide的mirror功能设置映射

这里用的方法4, mirrors.yaml, 不想一个一个做映射可以参考这里

问题2: 编译路径

17:36 $ make buildGOBIN=/home/jokimina/golang/k8s.io/helm/bin go install  -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.GitCommit=26c88ec815785708cc19c014366ce4bb000ae89c -X k8s.io/helm/pkg/version.GitTreeState=dirty' k8s.io/helm/cmd/...warning: "k8s.io/helm/cmd/..." matched no packages

如果是直接拉下代码放到随意的目录, build的时候会报错, 需要放到

$GOPATH/src/k8s.io/helm再执行编译

问题3: golang默认不是静态编译, docker alpine镜像没有运行所需要的库

$ make buildGOBIN=/home/xxx/go/src/k8s.io/helm/bin CGO_ENABLED=0 go install  -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.GitCommit=147c8217c7843b53e98528f6df12890d64c696be -X k8s.io/helm/pkg/version.GitTreeState=dirty' k8s.io/helm/cmd/... ~/go/src/k8s.io/helm $ ls bin/helm  rudder  tiller

make bootstrap build执行完后会生成二进制文件到 $GOBIN 目录下 , 直接替换掉阿里云tiller容器内的运行发现报错

# /tiller sh: /tiller: not found

但是本地可以运行, 搜索下

发现是由于默认没有静态编译导致引用了alpine不存在的库链接

# ldd tiller     /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)

重新静态编译下, 看到Makefile默认支持交叉编译这里偷个懒

make TARGETS="linux/amd64" build-cross

节省时间这里就只编译需要的了, 构建好的二进制文件会在 _dist/下

2.3 构建镜像

这里就直接替换tiller然后commit镜像, 不写Dockerfile了

$ docker pull registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1$ docker run -d --name build registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1$ docker cp tiller build:/tiller$ docker commit build registry-vpc.cn-beijing.aliyuncs.com/xxx-service/helm:v2.11.0$ docker push !$

2.4 换掉k8s集群上的tiller

$ helm init --tiller-image registry-vpc.cn-beijing.aliyuncs.com/xxx-service/helm:v2.11.0 --upgrade$ helm versionClient: &version.Version{SemVer:"v2.9.1", GitCommit:"08c1144f5eb3e3b636d9775617287cc26e53dba4", GitTreeState:"clean"}Server: &version.Version{SemVer:"v2.9.1", GitCommit:"e8c5cf08751dfc7be5c4bedcd400ed78355e3468", GitTreeState:"dirty"}

测试下

ok了

3. 后话

在配置参数不变更只是更新镜像的情况下, 其实可以考虑直接update deplayment开销会小很多, helm主要还是首次初始化的时候自动生成了需要的resource.

正常情况下官方限制的20M基本能满足需求, 只是超了一点点的话可以在helm ls的时候用-m或者--deployed限制下返回数量和过滤状态
不

过这里是一个测试集群release会不断增加...所以无法忍受了