Howard Chen's Blog

CODE IS POETRY

Quick Setup Your Local Kafka Env

Project at: https://github.com/howardchn/hello-kafka

This is used for setting up a Kafka environment with a simple producer and continuously sending a number per second. You could just build your own client to consume the predefined topics foobar.

Quick Setup

For quick start, only the following commands will be ready to go.

Note: before running the following commands, make sure the following items are installed.

Prerequisite

  1. docker CE or EE
  2. docker composer
  3. ports 2181, 9092 and 29092 are not in using

Launch The Service

1
2
3
cd [your work directory]
curl https://raw.githubusercontent.com/howardchn/hello-kafka/master/docker-compose.yaml -o docker-compose-kafka.yaml
docker-compose -f docker-compose-kafka.yaml up

Then wait about 15 seconds, the zookeeper, kafka and default simple producer are ready.

Build Your Own Producer

The integrated producer has hard-coded topic name foolbar nad the key is Null (not defined), the content is an incremented number and the sending interval is 1 second. Seems lots of restriction inside. The following steps will help to customize your own producer and docker images.

  1. modify main.py which is the implementation of the producer
  2. build docker image with docker build -t [your producer image name] .
  3. replace the producer image howardch/hello-kafka-producer with [your producer image name] in docker-compose.yaml file
  4. everything is set, execute command docker-compose up -d to run your own environment
kafka
Comments

A Mini App Refactor

Try it first and project in github.

It has been a long time I haven’t write st. Here is one thing I’m caring about these days. I reviewed all my projects in github, and I found one interesting mini app I built few years ago. It helps to get a programming language who is proper to learn. I remember when I initially built it, I’m pretty excited on how they way I built it and it looks like. But few monthes later, I picked this up and review this; I could say it sucks. In another word, I’m kindly improved myself in some aspects, especially consider in user’s standpoint and software structure.

The purpose for this blog

The main purpose that I write this down is that, I want to see my improvement and make it a checkpoint of it. You know, in the developer’s world, if you think your code seems awful and has a lot of space to refactor, that means you already improved your self; and it encourages me to learn more in the near future.

who could read the following content

If you are a starter to build apps, the following content might have some points you could consider. You know that the app you are working on is not for your self, consider as more as possible in user’s standpoint to make it convenient to use and lower the risky users might be careful when using the app.

Round 1 - Overview

I already found there are thousands of users has tried this app (it is pretty low number, but I’m satisify with it); when I reviewed thie records, I could say, many people just interesting in the first half an hour, then never get back. Why this happens? It seems a practise and challenge I will take.

main page

This is a preview of the app. When I enter this page, I found several issues.

  1. The major point is that, why I need user name in the first page? I doesn’t useful for my statistics. I will get rid of it first or move it to the end if someone wants to leave names to encourage me.
  2. What’s this app for? There is no brief introduction about this. So a simple short introduction is necessary.
  3. The title is too lang. Consider to make it shorter.
  4. It is not friendly to web browser. Don’t have too much idea for this. Consider it later.

For the result page, you will only get the name of the programming language. Henn… If I could get more detail about the programming language, that would be better. For example, we could provide following items.

  1. The feature of it.
  2. The difficulty of learning it.
  3. The popularity of this language.
  4. Which big company is using this technology.
    result change

That’s it for the overview.

Round 2 - Implementation

Generally, this is implemented with vuejs + bootstrap + express (node) + mysql.

This is a popular architecture for a normal or even enterprise usage. To be honest, when I design this, my initial imagination is this, so I processed in this way without any doubt.

This time, I found it could much simpler. For the architecture designing, there is not a best architecture, but only has the proper architecture. I could choose a popular architecture for your app, but it might not the proper one; it might make your development and maintainance harder.

The new architecture change is like this.

archiecture change

Get rid of backend

I already found my mini app is not necessary to have express and nodejs involved. The only purpose for add this level is that, I could add a Sqlite support and store the count of how many people prefers a specific language. To fix this, I searched on google, and found some solutions with some free cloud storage service. Leancloud is the one I used. Leancloud provides me an ability to store/fetch simple data object to cloud and easy to backup. It is pretty nice.

After this simplification, I can get rid of the backend part, then I can only focus on the front end part.

Improve frontend

It is hard to describe. Frontend is fansy and lots of technology and framework. Although I choose vuejs as basic framework, but the question is “am I using it the correct way?”. I might have another blog to represent this vuejs refactor experience.

About the pack and deployment, I used to use gulp. With this, I can define several gulp tasks and group the tasks together for building, deployment etc. After I learned webpack, I found it is my favorite one. I don’t need to copy many gulp tasks from one project to another; I don’t need to define many customized tasks; I can merge my css files, styles and many js modules to one file to ship it out… I will try to use webpack unless something it doesn’t support.

Deployment

The deloyment is a headache before. Due to the Sqlite and express level, I need to setup a web server on linux host. It is pretty expensive. Fortunately, I have one, so I can host it in my existing host. While now, I removed the backend support, this app becomes a pure static web app. I can host it on github and ship it with github pages without extra cost.

Follow up and wrap up

At this step, I’m satisfy with current refactor. But there are still st. I can improve more. I will do it once I have time with following improvement.

  1. Add a statistics on represents a ratio of the programming language.
  2. Allow to leave message.
  3. Allow to share the result on wechat or some popular social apps.
  4. Check the result of your history.
  5. Any suggestion from feedback.

See you later. I appreciate any feedback by contact me on Linkedin or email.

refactor
Comments

Space Between Number and Unit

今天在系统发现一个小知识，是之前都没有注意的。比如持续工作1小时30分，你会写成“1h 30m” 还是 “1 h 30 m”? 如果之前，我会毫不犹豫的选择前者。今天迫于强迫症看到系统里面写成后者就想把他改了。介于公司严格的代码审核机制，我还是先调查一下。结果不看不知道，一看吓一跳。居然有个标准可以参考。

注意5.3.3说，数字在单位之前的，都需要加一个空格来分割数字和单位。当然也有例外，那就是度分秒的符号和数字之间不用加空格。

哎！原来一直都没有走国际标准。时刻保持好奇心。

standard
Comments

Time Zone & Daylight Saving Best Practice

每当别人问起怎么存储时间信息的时候，我都会告诉别人，用统一的UTC时间来存储。结果，我一直都错了。这里整理了一份对于时间存储的列表，希望能对你的时间存储行为带来最佳实践。

一定要这样做:

  • 当需要一个确定时刻，保留一个持续化的并和夏令时无关的统一时间。GMT和UTC都适用于这种情况，但UTC会好一些。GMT是一个时区，而UTC是一个标准。
  • 如果选择存储本地时间的话，请把本地时间和UTC时间的时间偏移也记录下来。注意，时间偏移值在一年中的某些月份可能会变。这样后面使用的时候才能被计算成确切时间。
  • 有时，你也需要考虑把UTC和本地时间都存储下来。常常，我们会考虑把他们分布存储到两个字段里面，但有些平台上，他们可以被存储在一个平台里面，如MSSQL 的datetimeoffset字段(https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetimeoffset-transact-sql); 而这个不是sql标准字段。
  • 当需要把日期存储为数字类型，使用Unix Time (https://en.wikipedia.org/wiki/Unix_time) 如需要精确的时间，请考虑替换为毫秒级别。这个数字基于UTC，没有做任何timezone调整。Java: new Date().getTime(); JS: new Date().getTime() 或 +new Date都可以。
  • 如果以后需要修改这个时间戳，请顺带记录上原来时区的ID，用以判断时间偏移和原来存储时间比较是否变化了。
  • 如果是为计划未来的定期任务，存储本地时间会比存储UTC更好，因为通常会夏令时的改动。比如冬令时设置你每天的闹钟为早上8点，转换为UTC时间以后，在夏令时期间，你就会上班迟到一小时了:) https://stackoverflow.com/questions/19626177/how-to-store-repeating-dates-keeping-in-mind-daylight-savings-time/19627330#19627330
  • 当你计划一整天的时间，不要转成UTC或者其他任何时区来记录。
    • 数据库里面这种字段尽量不要包含时刻信息。保留到天即可。
    • 如果数据库或存储平台不允许这样做，那么尽量在读取时忽略时刻的意义。如果确实不确定，那么我们尽量采用正午12：00，而不要用午夜0：00来显示这个时刻。
  • 注意时区差并不是永远都是整数。比如Indian Standard Time is UTC+05:30, and Nepal uses UTC+05:45
  • 相比UTC或者GMT，很多商业规则使用Civil time会更多。安排计划时需要把UTC时间戳转换到本地时区。
  • 时区(Timezone)和时间偏移值(Time Offsets)不一样，时区不会变，而时间偏移值在一年中可能会变可能会变。比如洛杉矶的时区是IANA，在夏令时的时候时间偏移值为UTC-7，冬令时的时间偏移值为UTC-8，和它相邻的亚利桑那州不存在夏令时，时间偏移值一直保持UTC-7。
  • 考虑时间类型（确定时间，海外时间，相对时间，历史时间，计划循环时间），你需要采用不同的存储方式保证时间不会缺失。
  • 让你的操作系统，数据库和应用程序保持一致。
  • 服务器上，设置硬件时钟和操作系统时钟为UTC时间，非本地时间。
  • 服务器端代码，包含网站，永远不要指望服务器本地时间能有意义。
  • 根据你的应用，把时间的设置分成若干种类，对于不同的种类设计适当的存储方案。不要做全局设置。
  • 服务器使用网络时间协议
  • 考虑在FAT32/16的设备上使用本地时间。理由比较奇葩。比如在DOS时代，个人电脑都没有网络功能，时间都是使用Bios时间，存储在3.5寸软盘上。那时人们完全不会关心软盘上面存储的时间，而会使用铅笔在盘面上写下处理时间。因此这些时间不会准确。回到现在，很多移动设备，比如相机等还在使用FAT类型格式； 当你在另一个时区时，这些时间不会被改变，如果存储时间戳，就会导致存储时间的混乱。所以存储本地时间会更好。
  • 当设置周期时间时，比如跟美剧，每周一集更新那种，记住时间会因为夏令时，在不同时区会有不同。
  • 使用>=判断下限，用<判断上限。

千万不要这样做:

  • 不要混淆时区（America/New York）和时区偏移（UTC-5:00），他们是两种概念。https://stackoverflow.com/tags/timezone/info
  • 不要在老浏览器，ES5.1及以前版本使用javascript的Date对象，因为他们在使用夏令时计算的时候会有问题。ES6已经修复。
  • 不要使用客户端时间，至少我不能保证你浏览器上时间时正确的。
  • 不要再告诉别人，都用UTC时间了，虽然这种事我也做过。正确的做法是根据场景，选择对用的存储方式。

最后总结

在涉及到时间存储的问题上，一定要谨慎。把场景列出来以后再考虑适当的存储方式。一个简单的例子，我想过滤出今天所有的日志信息。想想，可能会出现几种不同的场景？所以一般通用的解决方案会把local_time, utc_time, local_tz都存下来。现今最长的timezone是“America/Argentina/ComodRivadavia”， 共32个字符。

timezone
Comments

Learn Python - Module

In the previous topic, it includes some really useful points of Python basics. In this page, we will try to use Modules in Python.

Custom module

There are two ways of using modules. Let’s first create a new Python file names module1.py.

1
2
def say_hello(): 
    print('Hello Howard.')

In another Python file that wants to use the functions in the module1.py, we need the following ways to reference this module.

1
2
3
# method 1. note, no file extension here
import module1 
module1.say_hello()

1
2
3
# method 2. note, we don't have the module name before the function name
from module1 import say_hello
say_hello()

Choose the two methods above wisely and it makes your code easier to read.

Use a build-in module

We are going to download an image from internet and store in disk. Here is the code.

1
2
3
4
5
from urllib import request

url = 'https://avatars2.githubusercontent.com/u/8789008?v=4&s=460'
filename = '1.jpg'
request.urlretrieve(url, filename)

In this code, we import an urllib module. You could use any module like this to support various requirements.

python
Comments

Learn Python - Basics

It is very excited to learn python (I recommend to use python3 directly which will have better support in the future). First, let’s start with an IDE.

Here are three IDEs I like very much.

  • IDLE, it is installed along with the Python installer, which provides a powerful shell of playing with python.
  • PyCharm, JetBrains provides the most powerful IDE for python, it supports many useful functions like refactoring, lint and so on.
  • VSCode is a lightweight IDE for python as well, but you need to install some plugins to support python better. Here are plugins recommended.
    • Code Runner for executing python
    • Python support code intellisence, code formatting, auto indent etc.
    • Pylint is recommended as well.

Here are some points for python, I will make it simple as possible.

  • Number

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    number = 51
    print(number)

    number = 51.3
    print(number)

    number1 = number / 2
    print(number1)

    number2 = number // 2
    print(number2)

  • String

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    str = "Hello World"
    print(str)

    str1 = 'Hello World'
    print(str1)
    print(str1[0:2])
    print(str1[:6])
    print(str1[6:])
    print(str1.upper())
    print(len(str1))

    # if we have some ESC, 'r' means 'raw'
    str1 = r'Hello \r\n Howard!'
    print(str1)

  • List

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    ls = [1, 2, 3, 4, 5]
    print(len(ls))

    ls.append(6)
    print(ls)

    ls1 = ls + [7, 8]
    print(ls1)

    ls2 = ls1[:2]
    print(ls2)

    ls3 = range(0, 10)
    for i in ls3: print(i)

    ls3 = range(2, 10)
    for i in ls3: print(i)

    ls3 = range(0, 16, 4)
    for i in ls3: print(i)

  • if: elif: else (note: is and == are both fine to check the equalvity)

    1
    2
    3
    4
    5
    s = 's'
    if(s == 'a'): print('alpha')
    elif(s is 'b'): print('beta')
    elif(s is 's'): print('superman')
    else: print('unknown')

  • for loop

    1
    2
    3
    4
    5
    6
    items = [1, 4, 2, 6, 9, 7]
    for i in items: 
        print(i)

    for i in items[:4]:
        print(i)

  • while loop (note: while, is, break, continue)

    1
    2
    3
    4
    5
    6
    i = 0
    while i < 10:
        print(i)
        i += 1
        if i is 5: break
        else: continue

  • Comments (single-line and multi-line)

    1
    2
    3
    4
    5
    6
    7
    # num = 20

    """
    str = 'hello'
    str += ' world'
    print(str)
    """

  • Function

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    # void
    def say_hello(name): 
        print('Hello', name)

    say_hello('Howard')

    # with return value
    def get_hello_message(name):
        return 'Hello ' + name

    print(get_hello_message('Howard'))

    # with default value
    def say_hello_with_default_value(name = 'Howard'):
        print('Hello', name)
    say_hello_with_default_value()
    say_hello_with_default_value('Bibioo')

    # with optional parameters
    def someone_do_something(name = 'Howard', action = 'ate', something = 'an apple'):
        print(name, action, something)

    someone_do_something()
    someone_do_something('Bibioo', 'is', 'awesome')
    someone_do_something(something = 'us', action = 'blesses', name = 'God')

    # flexibale variables
    def flexible_var(*args):
        print('it has', len(args), 'arguments')
        for i in args:
            print(i)
    flexible_var('a', 'b', 'c')

    # unpack aruments, watch out the compare of the last two calls
    def unpack_arguments(age, apple_pw, smook_pw):
        health = (100 - age) + (apple_pw * 3.5) - (smook_pw * 4.5)
        print(health)

    person_data = [35, 7, 0]
    unpack_arguments(person_data[0], person_data[1], person_data[2])
    unpack_arguments(*person_data)

  • Set

    1
    2
    3
    4
    my_shopping_cart = {'apple', 'milk', 'beer', 'milk'}
    print(my_shopping_cart)
    print(len(my_shopping_cart))
    print('milk' in my_shopping_cart)

  • Dictionary (watch out the three different loops)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    classmates = { 'Tony': 'cool but smells', 'Emma': 'sits behind me', 'Lucy': 'asks me too much questions' }
    print(len(classmates))
    print(classmates['Tony'])
    for i in classmates:
        print(i)
    for i in classmates.items():
        print(i)
    for k, v in classmates.items():
        print(k, v)

Now here are the basics for Python to remember, get back if need to recall the points, I will do more excited thing in the next part.

python
Comments

Gosh, My Naked Resignation in Sept

一个高龄程序员从离职到迷茫，到转换技术栈，最后进入一个非常满意的公司的一个月。

一个月能做些什么事？看两本书，新学习一门语言，还是旅游一次，再或者是迷茫的过完？而我在这一个月里，经历了离职，迷茫，旅游，学习，最后获得了一个自己满意的新岗位。这一个月，我读了5本书，学习了几个前端框架，巩固了单元测试的不足，捡起了10年没有用的Java，把基础和近几年流行的函数式编程过了一遍。这一个月，我面了6家公司，收到3个offer，最后选择了一家云计算行业的美国公司。回首看来，一个月很短，但是我感觉却是过了一年。

2017年8月16日，是我在前一家公司任职的最后一天。我为这家公司服务有近10年的时间，负责架构和开发了60%公司主流.NET控件产品，职位也是挺高的。离开这家有深厚感情的公司也是迫不得已。年龄大了，需要更宽广的空间和机会。因此，我做回自由人，重新考虑自己以后的职业和发展。

第一周

刚从每天固定的上班，下班的规律中走出来，那是相当的不习惯。早早的起来，泡了一杯茶，静静的坐着。拿出一张白纸，思考我想做什么？

我很清楚自己的能力，因此我用了几分钟绘制出了可能合适的几个option。

思维草图

问题主要分成3类

  • 做技术还是管理？
  • 我的技术栈？
  • 创业还是继续服务于企业？

这3类问题很难回答。

  • 我喜欢技术，但又离不开管理。
  • 技术上，我做了13年.NET产品的开发和架构。大言不惭的认为属于专家级别。另一方面，我自己熟悉web产品，一直在跟踪业界流行的前端框架，并尝试做过一些小型的项目。同时，对于nodejs很感兴趣，用它搭建过过2个网站。我也会是不是学习一下python，它在我之前的工作中开发一些实用的工具非常快捷方便。放弃任何一个都不太忍心。
  • 我有很多想法，有一些产品的雏形，我是把这些做完，还是继续为企业服务。

已经35岁的我，走到这一步，负责任的回答以上任何一个问题都会很难。既然这样，我就用最简单的剃刀原则，砍掉了很难实现的创业。我周围没有希望创业的小伙伴，一个人创业会很辛苦；虽然自己有一些积蓄，但是家庭的负担会比较重，风险也特大。看着那些倒在血泊中的前辈们，我只能在创业的道路上望而却步。基于离开公司的其中一个原因，我希望有机会接触更多更大的范围，不局限对于一个技术或者算法的专一，学习如何处理人与人的关系，人与项目的关系和项目和项目的关系也是一个技术人需要学习的。我认为我可以先不考虑这个问题，因为我觉得我都可以胜任。最后的问题也就是自己的技术栈。做这个决定算是对于市场的妥协。我们打开拉钩，智联等比较大的招聘网站，搜一下.NET相关的职位你就能意识到一些问题。在现在这种流行云，大数据，互联网和跨平台的大环境下，.NET的技术栈稍微显得有点局限（现在奋斗于.NET平台的小伙伴们不要太愤怒，我热爱.NET，从事这个技术栈13年，我的观点是.NET只是一个开发的工具，不要如崩塌的巴别塔，尝试看清眼前，并去接受，拥抱其他的工具）。最后我选出了.NET, JAVA, 前端作为技术栈；管理和技术双行；放弃创业的决定。

中间做了一个小项目，花了2天时间旅游，一周就这样过去了。

第二周

看起来接一点小项目可以维持生计，时间也很充裕；但这种脱离社会，不稳定的职业方向并不是我所赞同的。因此我决定开始找工作。我重新整理了我毕业以后的所有大一点的项目，以最简短语言描述了一遍。信心满满的把简历挂在了大型招聘网站上。按照我第一周思考的结果，以互联网，大数据相关技术为主的公司为主，选择了3家公司（这里就匿名了）投出了自己的简历。毕竟10多年的开发经验应该很快有人回复。可惜2天了，看到简历的进度居然还是在已查看的状态。第3天，其中一家公司标注了不合适，希望我用STAR原则重新修改简历并重新提交。“STAR”???这是什么？我就像一座放置了10年的老钟，瞬间被石化。马上搜索了一下，哦~，原来“现在”的简历需要按照“在什么情境下，用什么方式处理了一件事，最后结果怎样的”的格式来写。好吧，我又把自己的简历重新拿起来修改了一次。重新选择了几家公司提交了。终于收到了面试通知，这已经是第3周了。当然这一周我还做了一些有意义的事，比如把数据结构重新过了一遍，学习了React框架（最近React因为License问题被很多大公司禁止了），把Vue和Angular也重新复习了一次，过了一遍ES6的书，并熟悉了一下不同的前端测试框架等。

第三，四周

为何这两周一起呢？因为这两周基本每周前3天都是面试，后面2天就是等结果。其中见识了什么是好的环境，待遇和福利。才知道自己以前的眼界是多么的窄。最后这家公司也就是我决定去的公司，总共有5-6轮面试，具体我也不记得了，只记得有一天都呆在那里，不停的讲解以前的项目经历细节和现今一些技术的发展状况，也会涉及一些难以调和的人际关系的问题，白板也被刷新了几次，最后还有一段时间是全英语交流。

第3周我收到了一家国内企业的offer，和第4周的同一天收到2家美企的offer。考虑到后面这家做云和数据，现只做自己产品的公司和自己发展规划更加相近，因此我毫不犹豫的接受了这家公司的邀请。

回头看看这一个月，每天都在思考，学习；绝望过，也高兴过；想过自己干，却最终还是回归到企业。这一切都是自己逼出来的。人一生可能会很难有这种经历，以后可能也不会再有，但是确实这些经历让自己更有目标感，更加明确自己人生的规划，也是我这辈子积累的一份财富。因此我把这个记录下来，也许以后看起来只是蝴蝶轻抚一下翅膀，却成为我职业生涯的更上一层楼。

career
Comments

Proudly powered by Hexo and Theme by Hacker
© 2019 Howard Chen