husni.dev rss

Feeling Overwhelmed

Fri, 15 Apr 2022 00:00:00 GMT

You may feel overwhelmed to keep up with the speed of technology changes as a software engineer. But it’s perfectly fine to not know everything. Software development is a complex and broad topic. No one can undertstand everything. Instead of focusing on things you don’t know, focus on things you need to know to do your job.

Most the technology that you need to know to do the job can be learned as you go. You can learn new ideas faster if you understand the underlying concept. A new framework is just another variation of the same concept. I used to think that you have to master the subject first before you can do the actual job. The reality is, you became a master after you do the job.

How to Reuse HTTP Connection in Go

Wed, 07 Apr 2021 00:00:00 GMT

Go’s standard HTTP client supports HTTP keep-alive, which keeps the TCP connection open to be reused for subsequent HTTP request. You have to read the body request to completion before you close it as stated in Response.Body documentation. If you close the request body when it’s not fully read, the TCP connection can’t be reused because there’s still data left on the network to be read. Thus, the TCP connection will be closed.

Let’s take a look at the following program.

package main

import (
	"net/http"
)

func main() {
	for i := 0; i < 10; i++ {
		res, err := http.Get("http://127.0.0.1:8080")
		if err != nil {
			panic(err)
		}

		res.Body.Close()
	}
}

It’s a simple program that tries to connect to a local web server on my local machine. It closes the response body without reading it. We can monitor the network activity with netstat to see how many TCP connections are open while running the program.

netstat -atn | grep 8080
tcp        0      0 127.0.0.1:44390         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44374         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44380         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44392         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44388         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44384         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44378         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44386         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44382         127.0.0.1:8080          TIME_WAIT
tcp        0      0 127.0.0.1:44376         127.0.0.1:8080          TIME_WAIT
tcp6       0      0 :::8080                 :::*                    LISTEN

As expected, there are multiple open TCP connections.

Now, modify the program as follows: the program now reads all of the request body before closing it.

package main

import (
	"io"
	"io/ioutil"
	"net/http"
)

func main() {
	for i := 0; i < 10; i++ {
		res, err := http.Get("http://127.0.0.1:8080")
		if err != nil {
			panic(err)
		}

		io.Copy(ioutil.Discard, res.Body)
		res.Body.Close()
	}
}

Run netstat to see how many TCP connections are open.

netstat -atn | grep 8080
tcp        0      0 127.0.0.1:44332         127.0.0.1:8080          TIME_WAIT
tcp6       0      0 :::8080                 :::*                    LISTEN

You’ll see that there’s only one TCP connections while running the program. That means the program successfully reuses the same TCP connection.

You should fully read the response body before you close it to ensure that the TCP connection is reused. By reusing the same TCP connection, we can avoid the overhead in establishing a new TCP connection, thus providing a significant performance to a program that makes a lot of HTTP requests. If you don’t really need the request body, you can limit the reading with io.CopyN in case the the other end of a connection sends you unwanted large streams of data.

Manage HTTP Cookie in Go with Cookie Jar

Sat, 06 Feb 2021 00:00:00 GMT

Go’s standard HTTP library provides an interface to manage the storage of cookies in HTTP requests. If we provide a jar to http.Client, the jar is used to insert relevant cookies into every request. The jar is also updated with the cookie values of every response. If the jar is nil, cookies are sent only when they are explicitly set on the request.

Consider this example. Even though the server sets a cookie header on the response, the HTTP client will simply ignore it because the cookie jar is not set.

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if cookie, err := r.Cookie("Cookie"); err != nil {
			fmt.Println("You don't have a cookie. Here's one for you. Don't forget to keep it.")
			http.SetCookie(w, &http.Cookie{Name: "Cookie", Value: "Peanut Butter"})
		} else {
			fmt.Printf("You have %s cookie!\n", cookie.Value)
		}
	}))
	defer ts.Close()

	client := &http.Client{}

	if _, err := client.Get(ts.URL); err != nil {
		panic(err)
	}

	if _, err := client.Get(ts.URL); err != nil {
		panic(err)
	}
}

Output:

You don't have a cookie. Here's one for you. Don't forget to keep it.
You don't have a cookie. Here's one for you. Don't forget to keep it.

The net/http/cookiejar package provides in-memory CookieJar implementation. We can use that as the jar on the http.Client.

package main

import (
	"fmt"
	"net/http"
	"net/http/cookiejar"
	"net/http/httptest"

	"golang.org/x/net/publicsuffix"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if cookie, err := r.Cookie("Cookie"); err != nil {
			fmt.Println("You don't have a cookie. Here's one for you. Don't forget to keep it.")
			http.SetCookie(w, &http.Cookie{Name: "Cookie", Value: "Peanut Butter"})
		} else {
			fmt.Printf("You have %s cookie!\n", cookie.Value)
		}
	}))
	defer ts.Close()

	jar, err := cookiejar.New(&cookiejar.Options{PublicSuffixList: publicsuffix.List})
	if err != nil {
		panic(err)
	}

	client := &http.Client{
		Jar: jar,
	}

	if _, err = client.Get(ts.URL); err != nil {
		panic(err)
	}

	if _, err = client.Get(ts.URL); err != nil {
		panic(err)
	}
}

Now, the code produces the following output:

You don't have a cookie. Here's one for you. Don't forget to keep it.
You have Peanut Butter cookie!

Because the net/http/cookiejar package stores cookies in memory, the cookies will be destroyed once the program exits. If you want to persist the cookies, you can build your own cookie jar by implementing the http.CookieJar interface.

Beware of Implicit Memory Aliasing in Go For Loop

Sat, 19 Dec 2020 00:00:00 GMT

A for loop in Go will only use one iterator variable whose value gets updated in each loop operation. Since it’s just a single variable, its address is constant and doesn’t change. If not used carefully, it might lead to unintended behaviour.

Consider the following example.

package main

import "fmt"

func main() {
    var output []*int
    nums := []int{1, 2, 3}

    for _, num := range nums {
        output = append(output, &num)
    }

    fmt.Println("Value: ", *output[0], *output[1], *output[2])
    fmt.Println("Address: ", output[0], output[1], output[2])
}

It will produce the following output:

Value:  3 3 3
Address:  0xc00010c000 0xc00010c000 0xc00010c000

To avoid such problem, we can access the item in the iterable directly instead of using the iterator variable.

package main

import "fmt"

func main() {
    var output []*int
    nums := []int{1, 2, 3}

    for i := range nums {
        output = append(output, &nums[i])
    }

    fmt.Println("Value: ", *output[0], *output[1], *output[2])
    fmt.Println("Address: ", output[0], output[1], output[2])
}

Output:

Value:  1 2 3
Address:  0xc00013a000 0xc00013a008 0xc00013a010

Gosec can detect this problem automatically for you. If you scan the code in the first example with gosec, you’ll get the following output:

[/home/user/project/implicit-memory-aliasing/main.go:10] - G601 (CWE-118): Implicit memory aliasing in for loop. (Confidence: MEDIUM, Severity: MEDIUM)
    9:  for _, num := range nums {
  > 10:                 output = append(output, &num)
    11:         }

Obtain Exponent and Modulus from RSA key in PEM Format

Sun, 09 Aug 2020 00:00:00 GMT

If you understand the math behind RSA algorithm, you should know that RSA consists of several components: $e$ , $d$ , and $n$ such that

m^{ed} \equiv m\, (mod\, n)

Here’s how you can obtain those values from RSA private and public key in PEM format with python.

from Crypto.PublicKey import RSA
private_key_pem='''-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA3fJtY+4/TSuZ07iVqGq6Xa9QMFeor5JZwy7IfyyNOHeVFRSS
dYBo8WDTHTe4flJsRspHa1pI72v1yoJQJ3M7DFAtAxiHzqhAyYZ4wjdxTN8YBVl5
qqbRr7OKTvM8s3Z+2O7aG+ayIyeFKf0DX2ajFkOD3/ql6ne1pQUPCSjLNRSRYdCe
e03eKVw6ORSBUe262gEDf/UWHisg3o2JaVLcmR3ps4ESRcdx0AZ9hYWONGmHn7Fs
NiSp1ry8SVTOnfdcuvjVVfKLVaOui7nntU5OxQijz+/kk4vy58qDedyHxB24wMt6
S+7a5P++ZO20K4/mBIFmdyGIuvOYJY0Gs7X/EQIDAQABAoIBAQDSqNNjUjMLECNF
baG+5O5XFZtLByi43HQOak0rSCISQo5iH9CTbnotAPXgeY7Hd3wGBOCc6GjpttLo
j598kkatfTv8AECGyfQUU2ozDWSgze+CxFZSv1uvJP5VyVEIFaR4St0CNolGDLC8
FcYpusV40ERPRxxL26uxIYgP7YRSr5nReJrZU3xVaBIPxRVmPnJD/7MhaiwkZI53
Trwgp7Za3cXMLxetSXV4mRcyejKUYac+OWrf9/gvvFWegHRHJPz5AfS6jo8qPPy0
HO+yW3w9psSSYv7n+P5hHk6uqfEnPjp80ke+MxPtRTT2+OpjypToZiL4nNZmppA2
p5vQf0NxAoGBAPvLQ2YGZsP4uvbLrRMbFzaVlfiXnwGU4gJ4R2nf6OnKWhWqTI6W
TxuUS5VJetgSgSU721LjGvyBjwJWa7mMmfHnmrZqW7AKNKVlpUqx7nsUy9YHfV7o
zZTXBZxgMEiCWxHYl36DdGbEclQDitjFH+ur8TaVIo7xewZ/cgOudqK/AoGBAOGn
h8lpqoP0U/49BpznMhN1trBct4uLs+EYgK5Dtu2+RQCoApHCQ9QLLFjvVv3plyVm
Hy9+MHovxl9hQ1C6RqIpncWlGiI52gr4tGCiM+VMTQ8h2qOESfSJHqd4hivcHDsZ
/1HDETQa6vF3b/zyXDAQxstBhotHqhdZEHMTn2IvAoGAEI0n/QjmsHMlNxvkYKEj
QX9LwG+XEC+NO8RUTW1ejh+zFEGBjs3qnVF+Nm4CGTn+K0hn/mbXW0nY8Lplkmjf
pbsnvF35cPT6yPSJjQNtPcH8NDGB2D9NZD9OKx1Xjyyid+XY74QMnsTDDyq8cctj
ccZ3sLLU9riA8chm4j8RpVkCgYEA0Xi4N0Nss6OKmPbDfcGsnZ3nYh4fkF22NJ2P
TrWzSU0YNvy/rdkdJ+jbWQ+MvJYr5d6/0lGc5MAWsRHT8EuLKBvjxsZ11RHJE+g9
YXsRG6lR21s84A/8qxDisOnT+0EPv9lpRtgCz1CxikeqYLAG3eOk9qe0IvT6s0PN
XlX33acCgYEAq97EBQqEUusOOnut5Or89+j1e0A7NywSfiWYl0As8hwuxApAYtyV
sHgiPtRCjfD8UcWyD9cbL2Vk/6ZI85u0n7J+OO9ep6pyrMDlCMrx6gEuskyTalsp
ggB5+scSGkfSpCI/Hm2UvQUYf+QzYWpgy0yTmCrFoHibkTKHZQAzsVE=
-----END RSA PRIVATE KEY-----
'''

public_key_pem='''-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3fJtY+4/TSuZ07iVqGq6
Xa9QMFeor5JZwy7IfyyNOHeVFRSSdYBo8WDTHTe4flJsRspHa1pI72v1yoJQJ3M7
DFAtAxiHzqhAyYZ4wjdxTN8YBVl5qqbRr7OKTvM8s3Z+2O7aG+ayIyeFKf0DX2aj
FkOD3/ql6ne1pQUPCSjLNRSRYdCee03eKVw6ORSBUe262gEDf/UWHisg3o2JaVLc
mR3ps4ESRcdx0AZ9hYWONGmHn7FsNiSp1ry8SVTOnfdcuvjVVfKLVaOui7nntU5O
xQijz+/kk4vy58qDedyHxB24wMt6S+7a5P++ZO20K4/mBIFmdyGIuvOYJY0Gs7X/
EQIDAQAB
-----END PUBLIC KEY-----
'''


private_key = RSA.importKey(private_key_pem)
print("private key components:")
print("modulus:", private_key.n)
print("decryption key:", private_key.d, "\n")

public_key = RSA.importKey(public_key_pem)
print("public key components:")
print("modulus:", public_key.n)
print("encryption key:", public_key.e)

The code produces the following output.

private key components:
modulus: 28018210757639036701806576653473108663108237654992710644382580670706356521104503590269747243444881816860960886628707561729506054845669087158286487779668877579111045682687132130967542003640620395709760439291791359332971311569833464916696923708100648936544179482515518284015359758063690936170429993787988992366530178656661760648006433502964162950734482834691341055821131012179885041912952708019133191444259147492689058801662179466923034701521378053315755439283332003332125730956606219922314282161592457216832544141605338942735432555226858379903566758713188147527196578404934586961422561713661928243117076396507324088081
decryption key: 26593295115250601019258987963328215378793426844243199611260418512300199902937036198317581786338181920686317820343502527882298467058608112937639084575835373314784373493536023423023711533858143508166836113427630036711295107571141810758475600688751281969708624754419569088284349884654375863917259370028961766347968136242492409216019897338881947171652949409664617568373420518516510028969311296632860124215508429507388883135792513113188476224349460853934378076812541476273510991540179616816870490909273466785702119283308474487346009112179993007498269623263500977729040200031842282261746661152366200714995020265888356975473

public key components:
modulus: 28018210757639036701806576653473108663108237654992710644382580670706356521104503590269747243444881816860960886628707561729506054845669087158286487779668877579111045682687132130967542003640620395709760439291791359332971311569833464916696923708100648936544179482515518284015359758063690936170429993787988992366530178656661760648006433502964162950734482834691341055821131012179885041912952708019133191444259147492689058801662179466923034701521378053315755439283332003332125730956606219922314282161592457216832544141605338942735432555226858379903566758713188147527196578404934586961422561713661928243117076396507324088081
encryption key: 65537

Source

Amdahl's Law

Sat, 01 Aug 2020 00:00:00 GMT

Let’s say that you want to speed up 75% of your system’s execution time by three times. What is the overall speedup of the whole system? There’s a formula to solve this problem. It’s called Amdahl’s law and can be formulated as follows.

S{\scriptsize max} = \frac{1}{(1-p) + \frac{p}{s}}

Where

$S{\scriptsize max}$ is the maximum speedup of the system
$p$ is part of the system that is improved
$s$ is the speedup of $p$

If you speed up 75% if your system’s execution time by three times, the overall speedup of the whole system will be

S{\scriptsize max} = \frac{1}{(1-0.75) + \frac{0.75}{3}} = 2

Amdahl’s law has few important implications:

As $s$ approaches infinity, $\frac{p}{s}$ approaches zero. This means that no matter how far you optimize a system, the overall speedup is always limited by the part of the system that doesn’t benefit from the improvement ( $\frac{1}{1-p}$ ).
You should make the common case fast.

Uses

Sat, 01 Aug 2020 00:00:00 GMT

In case you’re curious, here’s what I use to get my work done.

Hardware

2018 15” Macbook Pro in space grey.
I also own a PC that runs on Windows 10 for playing games. I use Ubuntu for Windows if I want to do programming stuff on my PC.
Keychron K2.
Logitech MX Master 2S.

Development

VS Code with Dracula theme and FiraCode font.
iTerm2 + Oh My Zsh for Mac and Windows Terminal for PC.

How to Sign Your Git Commit

Mon, 15 Jun 2020 00:00:00 GMT

Today I learned that you can sign your git commit with GPG. Before you can sign your commit, you need to have a GPG keypair configured on your machine.

To generate a GPG keypair, run the following command.

gpg --full-generate-key

Once you have a GPG keypair, you can sign your commit with -S flag.

git commit -S -m "your commit message"

You can sign all commits by default in any local repository by running the following command.

git config --global commit.gpgsign true

You can also add your GPG key to your Github account so that they can verify your commits’ signature.