Gentoo, VPC, Docker and an Ethereum Go node

Blockchain, unlike “cloud computing” is more than a buzz word as it proves to be superior for integral and consistent systems of record in many aspects, such as IT infrastructure footprint and cryptographic security of the data at rest. While there are many projects out there aiming to deliver technology solutions based on blockchain concepts, I believe Ethereum will continue to play a crucial role as an underlying backbone of distributed applications and storage.

Since Ethereum is an open source project, I performed a little exercise of launching a public node. Perhaps I could even try some mining? To make things more difficult, I’ll describe here how I did that on Gentoo running on my VPC (virtual private server hosted by Linode), inside a Docker container. This is in no way an attempt to get rich by mining, since a VPC only operates on a CPU (of which I have 2 cores) and a VGA-compatible stub driver described as:
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02). Quite obviously this cannot run any mining in a serious fashion. All in all, there are some observations gathered throughout the exercise and a few problems solved, which I hope could make someone’s life easier.

I will start with installing the Docker daemon. Surprisingly, there are no software package dependencies. That’s really good, because I want my server to remain minimal.
Packages installed: 389
Packages in system: 43

root@rzski data # equery d docker
* These packages depend on docker:
root@rzski data #

The Ethereum project team now provides an official Docker image. Once the daemon is installed, it is as easy as pulling the image from the official repo by issuing docker pull ethereum/client-go. The image is only 44MB, which again makes my server satisfied as storage space aint cheap these days.

Before creating a new container by running this iamge, here’s a brief comparison of the 3 running modes geth (that’s the name of the Ethereum node software in Go language) can run with:

    –syncmode “full” – geth will download both the block header and data and start validating everything from the genesis block.
    –syncmode “fast” – geth will download only the block header and data, but once it catches up to the current block, it switches to “full” sync mode and starts validating everything on the chosen network. This is the default option.
    –syncmode “light” – geth only participates in validating and doesn’t download anything, but mining is disabled (this can be verified by the list of loaded modules. Even if you try to load the miner module later through the console, geth will print an error stating mining is not possible in this mode).

I went with the default “fast” sync mode, but decided to specify resource limits to prevent my container from slowing down other services I’m running from my server. I did the following:
Create a volume so the blockchain data could remain persistent: docker volume create geth_vol
And then launch the image to create a new running container:
docker run -it -m 2G --cpus 1.5 --storage-opt size=20G --name geth_container -v geth_vol ethereum/client-go --syncmode "fast" console

If you’ve run a node previously, you’ll see how naive I’d been by limiting the size of the container to 20 GB of storage space… Etherscan offers a graph showing how much space is actually needed to operate a public Ethereum node. At the time of writing, it is almost 100 GB, which greatly exceeds what I have available on this VPC, therefore I will have to abandon the mining idea and switch to the light mode. Perhaps in the future, once sharding is enabled, this will not be an issue.

Other settings are quite self-explanatory: I gave the container half of my RAM and one and a half CPU cores, which would result in the typical 150% CPU user time in “top”. The CPU limit can also be specified in microseconds of CPU time for finer granularity and updated on demand with the docker container update directive or via Kubernetes. When it comes to limiting the resources, docker depends on cgroups. In my case, not all cgroup options were compiled into the kernel, therefore upon launching geth in interactive mode, I got the following warning: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted.

This might seem benign but is actually quite tricky. The Linux kernel sends the TERM signal to processes that run out of memory. The OOM state is determined by a mixture of physical and swap memory, therefore once my container consumed all swap, then despite it only allocated half of the physical memory (I had 2 GB left for other services), the container still got killed. The only trace of this killing was a cryptic message in docker ps -a informing that my container stopped with error code 137.

There are two ways to address this: enable cgroups for swap memory in the kernel. The downside of this approach is there’s a performance hit and recompiling the custom kernel is required (in case someone wants to give it a go, steps to achieve are: get the kernel sources, make oldconfig from current settings in config.gz, make menuconfig to enable the missing cgroups swap option, make -j2 to compile the sources and then install grub bootloader, specify the path to the kernel binary, select the boot loader in the VPC dashboard (otherwise it will try to load the default kernel) and that should do the trick).

A slightly faster option is to simply create more swap space and reduce the swappiness. The downside here is naturally the storage space plus, not everyone has the flexibility to move partitions around like in LVM. You can however create a swap file wherever you do have some space at a small performance hit (the calls to pages written to thsi file will go through the FS layer). Here’s how to achieve this:

    First, drop existing caches: echo 3 > /proc/sys/vm/drop_caches
    Second, create a swap file:
    dd if=/dev/zero of=/path/to/swap bs=1024 count=1500000 (for 1.5GB of swap)
    chmod 0600 /path/to/swap
    mkswap /path/to/swap && swapon /path/to/swap
    Finally, reduce swappiness (the level of physical memory use at which pages get written to swap, default is 60): echo 10 > /proc/sys/vm/swappiness Note: use sysctl to make this change persistent if you need to.

With that, the node and the container in which it is running, should remain safe from the OOM killer. There’s also another option here: use docker’s option to disable containers from being OOM-killed, but that’s silly on a production server. The algorithm in a recent kernel will kill the “worst” process. If it can’t, it will kill something else, which could lead to a disaster. In any case, after following the steps above, it is safe to restart the container with docker start geth_container.

Note: it is super comfortable to use zsh with docker, as it auto-completes docker commands and lists help options as well as locally created container and volume names. Despite it is good practice to custom-name things, you don’t have to.

And that’s all – no configuration is required to start participating on the Ethereum network. Peers will be auto-detected within a minute and synchronisation will happen automatically. You might want to reduce verbosity in the console (debug.verbosity(2)) and check which peers you’re connecting to with admin.peers, and obviously the status of your synchronization with eth.syncing.

Mining is only possible after syncing completely, but if you have the disk space for that, then all you need is creating an account:
private.newAccount(“password”) and geth will automatically use this account’s address to store whatever it managed to mine.

Next-gen infrastructure part 2

Around the end of 2016 I wrote a longer article about the state of the IT infrastructure, trying to single out a trend I was observing. I was clearly inspired by Stanislaw Lem’s books as well as my own deep-dive sessions into technology. My conclusions back then were a vision of container or unikernel approach written directly into programmable field arrays by means of combining standard operations in hardware with micro-service architecture, but there was a substantial challenge to be overcome first.

Recently, my friend Matt Dziubinski shared with me an excellent article published under the Electronic Engineering Journal by Kevin Morris that seems to show where things are taking off. Titled “Accelerating Mainstream Services with FPGAs”, the author brings up how Intel acquired Altera, a major player in the FPGA market. It’s been a while since then with many comments suggesting Intel is bringing the hardware infrastructure to the next level. Since then however, the world hasn’t heard about a brain child of the top chip manufacturer fused with hardware accelerators from Altera. How is this merger really driving acceleration in the data center?

The major risk I saw back when I wrote my article was the enterprises’ capability to adopt such technology, given talent scarcity, especially for such low-level tinkering on mass scale. That type of activity is usually reserved for hyperscalers and high frequency trading technology. According to Intel statements and the EEJ article itself, Intel is moving forward in a slightly different direction by launching PACs (news statement), or programmable acceleration cards, which are based on Altera’s Arria 10 GX FPGAs and come with a PCIe interface. That’s right – the smart people at Intel have addressed the challenge by allowing specialized companies tune acceleration cards on per-need basis, which then they can simply insert into their boxes of preference: Dell, HP or Fujitsu. I am guessing integration with blade-type infrastructure is a matter of time as well. This way, enterprises don’t need to hire FPGA programmers with years of Verilog experience anymore. In a consolidating market, that’s a major advantage.

And now, most importantly, a glimpse at the numbers. According to the article on EJJ: In financial risk analysis, there’s an 850% per-symbol algorithm speedup and a greater than 2x simulation time speedup compared with traditional “Spark” implementation. On database acceleration, Intel claims 20X+ faster real-time data analytics, 2x+ traditional data warehousing, and 3x+ storage compression. And that speed-up is without considering the upcoming HBM2 memory and 7nm chip manufacturing process (The FPGAs are on 20nm themselves).

Gmail with own domain

Gmail seems to be everyone’s favorite web frontend for email. Until recently, it has also had the option to allow sending from custom domains, so the recipient would see “from:” for example instead of the not-so-professional These days however Google is promoting their G-suite set of products, which make this modification a bit harder if your domain is purchased from an external vendor. Here’s a brief article explaining how to set up your own domain as the default “from” domain in Gmail.

First of all, to avoid reinventing the wheel, I first googled (heh) for existing approaches and found many cases where an external MTA performs authenticated submission to This is sort of weird, but apparently that’s how Google is fighting spam and email address spoofing. I tried that approach only to find out that in addition to the mandatory authentication (MTA to MTA with passwords?), Google also modifies the header’s “from” field in incoming messages, stamping in the gmail account and moving the previous address to a new header line called “X-Google-Original-From”. As you can imagine, it makes things difficult to manage. In addition to that, Gmail would re-deliver these messages back at where the MX records point, despite their desired configuration was in place, so I had to create a black hole rule to prevent SMTP flooding (discard directive).

For that reason I tried a different approach. Here’s a brief explanation how to set this up using Exim as the MTA (but any other SMTP server would do). In this example, the MX records should point to the external server running the MTA (don’t forget the dot at the end). For outbound mail, Gmail will act as a client (MUA), using Exim as the MTA to authenticate over TLS and send mail out. The Gmail configuration doesn’t change and is explained here. For this to work, authentication data need to be created on the MTA and one more thing: header rewriting at SMTP time, if the domain you’re configuring now isn’t the same as the primary FQDN of the MTA (or, if you allow clients to send with multiple domains from the same server/container). Therefore, to have mail go out with the right FQDN, a rewrite rule like this is required:

begin rewrite
\N^$\N Sh

As for incoming mail, the task is fairly easy. Instead of authenticating to gmail, I redirect/forward the messages to the original gmail account after accepting them as local. This can be achieved by creating an exception for the default redirect route (which normally reads /etc/aliases for redirection paths), by adding a condition to match the new domain in question. Here’s an example:

begin routers

driver = redirect
domains =
data = ${lookup{$local_part}lsearch{/etc/aliases}}
file_transport = address_file
pipe_transport = address_pipe

Any file could be used instead of /etc/aliases, just make sure the UID/GID with which your MTA runs can read it. The format would be, following this example: “my_name:”. And that’s all – it’s SPF-friendly and IMHO cleaner and simpler than the authenticated approach. You might get cursed for rewriting headers by SMTP purists but well, Google does it too.

Artificial Intelligence and Creativity

First, a brief off-topic introduction. A quick search returns the following definition for creativity: “the use of imagination or original ideas to create something; inventiveness”. Not surprisingly, the definition refers to two other words which are similarly awkward if you’re thinking like a machine… Still, we know the process exists, everyone can name creative persons they know directly, so the subject definitely exists. Have we however reached a point where creativity has been distilled down enough to allow coding it into a machine?

Answer to that might depend the understanding of “thinking as a machine”. I wrote that intentionally, because it seems the terms are generally misused. I am used to getting a few sales pitches every month about how their new AI or even machine learning solution will revolutionize my business. I’d usually ask how it works and the answer puts the solutions in one of these pots:
a) simple logic
b) actual AI
c) machine learning
As you can guess, most answers end up being a), sometimes a mixture of a) and b). I am yet to experience someone selling me true c).

What’s the difference? Software allowed conditional responses for a very long time. Conditional clauses can be smartly designed and nested to a point where it seems the software (usually the frontend…) is “very smart”, or even “can predict what the user needs”. That’s not AI though.

AI kicks in where an actual algorithm, based on clues, can make assumptions or scoring while no direct reaction has been designed into the software. We have AI everywhere already: every spam filter, every antivirus engine follows this rule. For anything more complex though, like aiding in decision making, the computing capacity was so high that only now AI for wider use is becoming popular.

And finally there’s machine learning, which brings AI into a new tier: not only the algorithm goes over data to build rules, it can further based on previous runs build further rules, and rules for such rules, and so on. From computation, that’s usually very expensive, which is why there’s a new blooming market for chips and FPGA solutions that optimize this specific type of load to achieve good timing responses in the learning and decision making process.

I hope this becomes a good introduction to what I actually wanted to share. There’s an excellent paper on Arxiv detailing how tasks submitted to AI/ML solutions were “resolved”, sometimes in the most surprising way. The paper covers 27 anegdotes about how AI found the most surprising answers (to the surprise of the researchers).

From this perspective, wouldn’t you call it pure creativity? And if yes, I’d like to propose a new definition of Creativity: “in the huge lake of data(facts and clues), find new connections that are valuable”? I guess it works for Intelligence as well, if you cross out the word “new”.

Talent Management

I had the privilege to write another piece for the IT WIZ magazine, this time on Talent Management. Wide subject and as always, a struggle to write something new. Something that’d be interesting for everyone. So the idea came up to write “with a twist”. The Polish version can be found here.


It’s been a while, so here’s something more social and less technical again. And I get to join the lodge writing about Linus Torvalds! Let me first write this: any dispute on style or approach to problem solving must be second priority if your main product – Linux – remains one of the most stable and secure software ever produced. This is not just any software. We’re talking about millions of lines of code changed between every point release, provided by tens of thousands of developers worldwide. Software working with a broad number of hardware architectures, “talking to” unnumbered amount of peripherial devices. Finally, a piece of software causing discussions like “If Linus is gone, what will happen to the kernel?”.

I believe it was worthwile summarizing in these few sentences what is the true achievement there. I had to start with that brief summary, otherwise I felt I’d be joining the group that might have been well described by Linus in the following words some time in 2014:

“And there’s a classic term for it in the BSD camps: “bikeshed painting”, which is very much about how random people can feel like they have the ability to discuss superficial issues, because everybody feels that they can give an opinion on the color choice. So issues that are superficial get a lot more noise. Then when it comes to actual hard and deep technical decisions, people (sometimes) realise that they just don’t know enough, and they won’t give that the same kind of mouth-time.”

You can say a lot about Linus and his approach to people or dispute. Dispute will always happen if you work with people of strong character, who have devoted parts of their lives to master an area they then have to prove in production. But not everyone you get to work with will be following this principle. And yet, understanding the root cause of some debates and efficiently not wasting any energy on them is a decent treat of a leader – one that doesn’t often appear in the popular coaching memes. Probably because it is also painful and not entirely neutral.

But can you make big achievements and remain neutral?

Improving the VoIP foreign number solution

It has been a while and I’ve noticed a few issues in my solution (more info here) allowing “assigning” a foreign number to your smart phone. Since solving them was a nice little achievement, I’ll share the description here. It was quite an investigation, too!

The first problem was that the connection keeps timing out from time to time. Not always, but I couldn’t determine a pattern (ah, I love these situations!). In such cases there’s no other choice but to get your hands dirty… and in this case it means some deep packet inspection. Asterisk makes this task fairly easy, so tcpdump was not required. Using sip set debug ip/peer on/off allows finding how my asterisk and the SIP client on my mobile are talking. That’s possible even if you don’t have a good understanding of the session initiation protocol – simply googling it’s workflow is enough to see how it should look. Then all you have to do is comparing the expected flow with what you actually get.

Because of my intermittent issue, I had two sets of SIP debug data – when it works, and when it doesn’t. Comparing the two showed me that when the timeout happens, it is actually the phone that stops responding to SIP INVITEs. Having tried a few other SIP clients on the phone, I had a strong feeling this was not going to be a client issue. So maybe the network? Bingo! As soon as I switched off from my wifi and landed on 4G, the connection worked flawlessly and never failed. As this is weird, I went on to research why that migth happen and found some content explaining how most home routers have a faulty implementation of application layer gateway for SIP. Indeed, it was SIP ALG messing my SIP traffic in failed attempts to “secure my traffic” by inspecting source/destination addresses in the SIP packets. In my case, it would filter out my SIP traffic as soon as the router “forgets” the addresses and mappings, which in my case was 30 seconds from registration (TCP handshake from the mobile to my VPS).

Considering the minimum frequency at which SIP clients can re-register is once every 60 seconds and that my router would time out after about 30s, I was only left with half of the register time working. That’s way below my expectations. I can’t reconfigure my home router (thanks to the ISP), so what do? There’s no way of setting the registry time lower in the client itself without dirty hacks. I ended up doign the following: switch from SIP to IAX2 (Asterisk’s preferred protocol for VoIP), and then I wrote a small patch to my ebuild for Asterisk on Gentoo to define the re-register frequency as 25s, using the following:

$ grep sed /usr/portage/net-misc/asterisk/asterisk-11.25.1.ebuild
sed -i 's:EXPIRE.*60:EXPIRE 25:' "${S}"/channels/iax2.h && ewarn patched IAX2 registry timeout

After re-compiling (making such tweaks is very easy thanks to Gentoo) and restarting Asterisk, even though my client kept asking for 60 second sessions, Asterisk would “demand” a new session (from which all it gets is updating the actual IP address of the smart phone) every 25 seconds anyway. And the timeout problem is almost gone!

Almost, because Asterisk has a security setting called “nat”. It has to be set to “no” regardless if it is SIP or IAX2, which means that when the agents register, Asterisk doesn’t inspect the IP addresses in the header – and those do fail since NAT would translate the IP address of my phone from local LAN to the public IP address of my router. Since I control access to my Asterisk using multiple factors and layers (iptables and user/pass), I consider it safe for disabling.

Improving the efficiency of technical teams

Happy New Year to all the regular readers!

Recently, I had a very interesting chat with a fellow IT leader; we discussed the efficiency of tier 1 technical support and how we usually try to achieve the optimal 80% efficiency mark (80% of the cases to be solved within self-service or the first help desk line). While common techniques are indeed… common (by seeking common denominators in root causes), when it comes to tier 2 and above, there were fewer obvious solutions. And that’s natural, given that complexity becomes a major factor and is implied by the higher tier. So what can leaders do to gain improvement in this area?

I did share one recommendation which works for well for me. It caught some attention therefore I’ll cover it briefly here. The method is called obtaining a minimal working example. It is mostly popular among software developers who are more likely to stumble across a block while working on a complex software project. What do? Involve more brains then, right? Well, of course, but who has the time to go through a litany of complex lines of code, while everyone has their own work to do? Especially if you consider sharing the problem with an open source community, you will need to come prepared. You’ll get no input if the question is presented in a way which might resemble “hey, do my work for me, would you”.

A minimal working example is a complete (ergo: working) piece of code which has to be well crafted before can be presented via collaboration tools to narrow down the scope and yet be able to present the issue. That is required to allow a group to investigate quickly and efficiently. Building an MWE is a method, but even more, it is an exercise for the author. The developer has to be able to collect only the required code, leaving the remainder out to avoid obscuring the root cause of the issue and slowing down the investigation. During the process of building the sample, this method requires considerable effort to be put into evaluation of every step and… yes, about 80% of these get resolved during the preparation of the MWE!

While it is certainly a trick software developers and their managers should consider, it is not limited to programming. System administrators can also benefit from this approach, especially when working on complex configuration files, sometimes including vast and compelx scripting. As tier 2 or 3 support teams gain the ability to build a minimalistic setup with the problem bit captured in it, they surely will also be much more competent and empowered to solve and prevent other technical issues. Naturally, they will also be very efficient when working with the vendor (if eventually required). Building an MWE is also encouraging team work through empathy – after all, the MWE is prepared for someone else to understand and gain insight.

So, that’s one method; please share what works best for you!

Enter the commoditized IT, pushing the boundaries

Commoditization of services might sound like a new trend. After all, all those Ubers and Airbnbs seem like fresh ideas, appearing just recently. We still call them “market disruptors”, right? But probably only those working with IT every day realize that the same trend, with aim for the smallest item performing an unique function, has been present in the IT industry for decades. In this article, I will present the trend as I see it and share my observations. For the casual reader – I promise to keep it as user friendly as possible!

With a certain level of simplification, I believe I can make a statement commoditization in IT started when the “distributed environment”, as the IBM people call it, took off with the approach of creating simple, stand alone computing platforms we refer to now as servers. The attempt to offer services installed on multiple servers, dispersed globally for optimal customer experience and resilience is still popular even with the obvious faults of this approach. For example, servers had a set of resources at their disposal, as the vendor designed them to be applicable for majority of use cases. That usually doesn’t match with what the hosted application really needed. These servers were not as resilient as the mainframe, but at a reasonable price, they allow much faster concept-to-software-service process. These inefficiencies naturally lead to costs that normally wouldn’t have to be implied, but also to indirect outcomes such as developers, especially in the high performance computing sector, having to perform low level optimizations of their software.

The second level of infrastructure commoditization I call the concept of defining key resource areas commonly used by software and creating platforms which allow administrators looking after resource pools. Typically, this is referred to as server virtualization. Imagine your developers work on two projects where the product in each case is an application, but with an entirely different behavior. An application doesn’t store much data, but performs difficult computations – no problem, the admin can spawn (yes, spawn!) a new server with some memory but multiple processors (as long as the app can make use of them). The other application does little math but needs to load huge objects into memory? The admin could now use the memory resources saved on the first project and allocate them here. Running out of resources in the pools? Just insert (hot-swap) another blade of CPUs, RAM or expand the SAN for data storage.

Around the same time, the CPU itself has undergone even further commoditization, we could call it level 3. Multiple solutions have been implemented on processors to improve their efficiency without increasing the clock speed or when further miniaturization of the manufacturing process was not yet achievable. One of the key optimization areas was based on the observation that a lot of potential is lost during the processor cycle itself. As a result, various instructions have been implemented, like SMT for physical cores of a CPU, HT for logical cores, but also VT instructions for the actual virtualization. As a result, we get logical cores in physical cores forming a modern CPU.

Level 4 is a more recent tale, when the definition of SDNs, or “software defined networks” appeared. In simple words, the virtualized stack of standard commodities could be expanded to cover the network as well, which in the internet era is a commodity in itself. The idea is to present a complete set of infrastructure items to meet the development design, which naturally speeds up the whole infrastructural design phase of a project and most importantly, offers a great deal of simplicity to everyone involved in the development. Deployment of new applications of services has never been this easy.

With software defined networking and pools of virtual resources at hand, it is not hard to notice this approach still borrows from the good old “application hosted on a server” concept from what I called “level 1” era. This means each virtual server comes with a lot of overhead, including the operating system and its “wealth” of features. An OS is usually prepared with the idea to just work for as many setups as possible, which then by definition makes it “not the most optimal piece”. Level 5 of commoditization comes to the rescue – containers, among them the most popular container technology – Docker. A single server can run multiple containers, which on the outside look like virtual machines themselves, however they contain only the bare minimum of the OS that the applications hosted in them need to operate. Brilliant, isn’t it? Add to that software to operate multiple containers such as Kubernetes.

So what’s next? Commoditized infrastructure pieces, commoditized operating system allow granular choice of elements to operate applications or services. It doesn’t look like much more can be achieved here. Who can further push the boundaries? The answer is based on the fact that, just like the OS, almost all concepts in popular technology are presented in a way to allow 95% of the use cases to succeed. But what happens, when you’re in that remaining 5%? And who usually ends up there? For the last decade, in my opinion, two areas fall in the 5%: high frequency trading and high performance gaming. Add to them extreme data center owners (aka hyperscale), such as Google or Facebook and you’ll notice they all already have the solution. Software will only be software – it gets loaded into standard memory, operations executed using a standard core. But if the same operations applied to a virtual container happen million times per day on a standard piece of commodity, why not move them to hardware?

FPGA, or field programmable gateway arrays are becoming popular, because they allow just that. The software developer’s work doesn’t have to end as soon as the code is sent to the compiler. The compiler allows the code to be loaded and executed by a typical CPU, which by design is universal and can do a lot of things. Most software however perform a set of standard operations and then something unique to their nature, which gets repeated millions of times. FPGAs allow commoditization of that unique, repeatable activity, which when compiled onto the programmable board, can give it a performance boost of 10-100x at only 20x power increase. There is an immense cost reduction, instead of simply scaling the pool of CPU/MEM resources.

Does it mean application programming has to again become very low-level, make a huge turn around from the era of rapid development? Not at all, there are tools available that make it very easy, for example by utilizing Verilog. Certain standard blocks (DSPs) are pre-installed on the boards, to allow the developers to actually implement only the higher level logic. At the same time, programmable boards are available at reasonable prices and they can be re-programmed, unlike the super-expensive boards produced on demand for HFT trading companies.

FPGAs are the next big thing – if the above combination of breakthroughs did not convince you, take into consideration the battle to purchase Lattice, a vendor of FPGAs only, for $1.3 billion. Consider the rise of Xilinx, specializing in programmable SoCs. Last but not least, Intel acquired Altera about a year ago with the goal to join precisely this promising business.

So what the CIO/CTO can do? While it might not be very easy to find Verilog/VHDL specialists who also are capable developers understanding time-to-market and quality programming concepts, it was a matter of time until other vendors try to fill in the gap. Amazon is already offering EC2 instances with programmable hardware. OVH also offers a similar solution called RunAbove, which at the time of writing this article got completely sold out. Last but not least, there’s also, offering a Google Go language compiler optimizing the code developers wrote to be directly installed onto programmable boards.

What’s next? Maybe containers hosting micro services could be moved to FPGAs?